How Much VRAM Do You Need to Run a Local LLM? (2026)

You can size the VRAM a local LLM needs with one rough rule: quantized to 4-bit, a model needs about 0.5-0.6GB of VRAM per billion parameters. So a 7-8B model wants 6-8GB, a 13B about 10GB, a 32B about 20GB, and a 70B about 40GB. At 16-bit (FP16) it is roughly four times that, so even a 7B needs about 16GB. That is why local agents almost always run 4-bit.

In one line: at 4-bit, about 0.5-0.6GB per 1B params. 7-8B=6-8GB, 32B=~20GB, 70B=~40GB, plus a few extra GB for the KV cache on long context.

First, what is VRAM? It is the dedicated memory attached to your graphics card (GPU). To run a model you must load its full weights into this VRAM, and if there isn't room the model either won't launch or falls back to a much slower path. So "will this model run on my GPU?" is really the question "does this model fit in my VRAM?"

What decides VRAM use?#

Required VRAM is the sum of three parts. First, the model weights — parameter count times bytes per parameter, usually the biggest chunk. Second, the KV cache, storage for the intermediate results of every token read so far, which grows with context length. Third, runtime overhead — the odds and ends the execution engine and temporary buffers use. Quantization attacks the first part's "bytes per parameter," shrinking the largest chunk directly: FP16 is 2 bytes per parameter, 4-bit is 0.5 bytes, so weights alone drop to a quarter.

Here is approximate weight-only VRAM by size and precision. (These exclude the KV cache and overhead.)

Does longer context cost more VRAM?#

Yes, and this is where beginners trip. The table numbers are weights only, so trusting them alone and filling VRAM to the brim will overflow in practice, because the KV cache scales roughly with context length. It is negligible for a couple of short exchanges, but feed in a whole long document or run an agent with many tool definitions for a while and you can need several GB beyond the weights. So the practical rule is simple: don't pick a GPU that exactly matches the weight size — leave headroom for the KV cache and overhead (usually 2-4GB or more).

What if you are short on VRAM?#

Three options. First, drop to a smaller model (13B instead of 32B, 7-8B instead of 13B). Second, quantize more aggressively (4-bit instead of 8-bit, sometimes 3-bit). Third, offload some layers to the CPU — the part that doesn't fit on the GPU sits in system RAM and the CPU computes it. Offloading works but is noticeably slower because data shuttles between CPU and GPU. So the rule is single-minded: the agent loop is fastest when the model fits entirely on the GPU. Offloading is a last resort that "makes it run," not a performance tuning.

What about Macs (Apple Silicon)?#

On a typical PC the GPU's VRAM and system RAM are physically separate, but Apple Silicon (M-series) shares one pool of unified memory between CPU and GPU. So a Mac with 32GB or 64GB can run larger 4-bit models than a discrete GPU of the same nominal size — a 64GB Mac, for instance, digests models that would overflow a 24GB RTX 4090, all within unified memory. Runtimes like Ollama and llama.cpp make this work by tapping that memory directly through Apple's Metal API. Bear in mind, though, that unified-memory bandwidth and compute can trail a high-end discrete GPU, so "fits a bigger model" and "runs faster" are two different things.

A quick baseline#

1. On a single consumer GPU, 4-bit 7-8B is a safe default (6-8GB).
2. A 24GB card (RTX 4090) can run a 4-bit 32B.
3. Beyond that, especially 4-bit 70B (about 40GB), you need multiple GPUs or heavier quantization.
4. In every case, pick the card by adding headroom for the KV cache and overhead on top of the table numbers.

Note: these figures are weight-only approximations and vary with runtime, quantization method, and context length. Measure your actual usage before deploying. Hardware and quantization methods move fast, so this table is reviewed quarterly.