Local Open LLM VRAM and RAM Requirements, Measured

A local LLM's VRAM is set not by model size but by "quantized weights + KV cache + overhead." The measured rule of thumb is simple - about 2GB per billion parameters at FP16, about 0.5GB at INT4, plus 15-20% on top. But the real reason beginners crash their GPU is not the weights, it is the KV cache: grow the context and the KV cache quietly balloons, so at 128K context the KV is about 3.4x the weights. That is why a model that ran fine at 8K dies with OOM on a long prompt.

In plain terms: VRAM is a restaurant bill. The weights are the "cover charge" (paid once, fixed), and the KV cache is a "meter" that climbs while you sit. The more tokens you feed (longer context) the higher the meter, and more guests (concurrent requests) each run their own meter.

First, the terms. VRAM is the memory on the GPU. Weights are the fixed footprint of the model parameters themselves. The KV (key-value) cache is a buffer that stores computed results for already-processed tokens so the next token is fast; it grows with context length. Quantization compresses weights to fewer bits (e.g. 4-bit) to shrink the footprint. Context is the token length the model sees at once.

What sets the VRAM?#

Three terms: weights, KV cache, and runtime overhead. (1) Weights = parameters x bytes per parameter (FP16 2, INT8 1, 4-bit about 0.5). (2) KV cache = 2 x layers x KV heads x head dim x bytes x context length x batch, accumulating per token. (3) Overhead = CUDA, activations, and framework buffers, usually 0.5-1.5GB. Plan a safety margin with a x1.15 multiplier. So it is not "it's 7B, should be fine" but a calculation including precision, context, and concurrency.

See how the three terms stack up to fill VRAM. Weights are fixed, but the KV grows upward with context.

How much does quantization save?#

16 to 8-bit halves it, 4-bit quarters it. FP16 7B is about 14GB, Q8 about 8.5GB, and Q4_K_M (0.57 bytes/weight) about 5GB, which fits comfortably on a 12GB consumer GPU. In many cases 4- and 8-bit feel close to 16-bit in quality, so they work as defaults. One caution: MoE models must load all parameters, not just the active ones (a 120B MoE with 12B active still loads all 120B - only speed follows the active count, memory follows the total). Quantizing the KV cache to 8-bit halves it too (and 2026 brought 3-bit methods with 6x compression).

Why does raising context suddenly blow up?#

Because the KV cache grows linearly with context and concurrency. Measured: the KV cache of Llama 3.1 8B (FP16) is about 4GB at 32K and about 16GB at 128K, exceeding the Q4 weights. Rough figures by context: 0.5-1GB at 4K, 2-4GB at 32K, 8-16GB at 128K. And each concurrent user gets a separate KV, so four users at 32K is 16GB of KV alone. Mitigations are GQA (Llama 3.1 70B cuts to 8 KV heads for an 8x saving - without it, 128K would be about 336GB) and PagedAttention (vLLM, cutting waste to near zero). Also beware the prefill spike: even if the cache fits, the attention matrix balloons momentarily and can OOM, so you want FlashAttention.

Plotting the measured KV of Llama 3.1 8B (FP16) by context shows it grows linearly and passes the Q4 weights at 128K.

See how mitigations press the KV down so 128K becomes tractable. Stacking GQA and PagedAttention brings an explosive requirement into practical range.

So what fits on your GPU?#

The key is calculating before you buy.

• Calculate: total = quantized weights + per-context KV + x1.15. 7B Q4 fits a 12GB (RTX 3060) or even 8GB GPU; 70B Q4 is about 35GB but real-world total can exceed 200GB.
• Context: if you do not need long context, shrink the context window to save KV (the easiest saving). If you do, pick KV 8-bit and GQA models.
• If short: RAM offload is 10-20x slower - hybrid is only practical when 60%+ of layers fit in VRAM. Measure exact values on your own model and context.

Reference links

• llama.cpp (local inference, GGUF quantization)
• Ollama (local runtime)
• vLLM (PagedAttention serving)
• PagedAttention paper (vLLM)
• Hugging Face Accelerate (memory estimation)

Note: GB and multiplier figures are public 2026 measurements and calculations and vary by architecture, precision, context, and framework (not permanent numbers). The KV cache in particular shifts a lot with GQA, KV quantization, and PagedAttention, so measure your exact requirement on your own model, context, and concurrency (these numbers are only a start). Memory prices and capacities move monthly in 2026, so check the vendor page before buying. Reviewed quarterly.