4-bit vs 8-bit Quantization VRAM, Measured: A Bigger Model at Q4 Wins

4-bit and 8-bit quantization cut a weight's bit count to shrink memory linearly. Three measured essentials: (1) Q4_K_M (about 4.5 bpw) is the size/quality sweet spot, shrinking a 7B to about 3.9GB with under 2% quality loss. (2) 4-bit is the production floor, so below it (Q3, Q2) reasoning noticeably collapses. (3) So at the same VRAM, "a bigger model at Q4" beats "a smaller model at Q8" - the bigger model's capacity outweighs Q4's 1-2% loss (but only above 4-bit). In short, spend leftover VRAM not on more bits but on a bigger model.

In plain terms: quantization is photo compression. Like turning a raw file (FP16) into a JPEG, Q8 shrinks it invisibly, Q4 nearly invisibly, and Q2 visibly smears it. And "a big source at decent quality" usually beats "a small source at high quality."

How does quantization reduce memory?#

By cutting bits per weight (bpw), lowering size proportionally. FP16 is 16 bits (2 bytes/weight), Q8 about 8.5 bpw, Q4_K_M about 4.5 bpw. The key is the K-quant: grouping into 256-value super-blocks and giving sensitive layers (attention, output) more bits and insensitive ones (FFN) fewer. So Q4_K_M, though named 4-bit, is really about 4.5 bpw and recovers 5-8% quality over the legacy Q4_0 of similar size. Not just how many bits, but where you spend them, sets quality.

How much does each level cost?#

Weights drop to about a quarter of FP16. A 7B goes FP16 13GB -> Q8 6.9GB -> Q4_K_M 3.9GB (about 3.4x compression); a 13B goes 24.2GB -> Q8 12.9GB -> Q4_K_M 7.3GB. But that is weights only, so add 10-20% for KV cache and overhead. And context length is independent of quantization - Q4 or FP16 both support 128K, but long context eats more RAM regardless (a 64K context on a Q4 7B may need 10GB+). So "quantized, therefore context is free" is false.

Which level is worth it for the quality?#

Q4_K_M by default, Q5_K_M or higher for precise work. Q8_0 is effectively lossless (under 0.5% vs FP16, +0.01 ppl) but saves little. Q5_K_M is within 0.08 ppl (25% larger file), and Q4_K_M is under 2% loss, optimal for most tasks. Conversely, below Q3 it breaks on complex, multi-step reasoning and Q2 loses coherence in long outputs. So for coding and math where precision matters, use Q5_K_M+, for maximum fidelity Q8/FP16, and Q4_K_M for the rest. Below 5 bits, choose K-quants over legacy (Q4_0).

At the same VRAM, what should you pick?#

The key is spending leftover memory on parameters, not bits.

• Rule: at the same budget, the biggest model that fits at Q4_K_M/Q5_K_M. The bigger model's capacity beats the 1-2% loss of 4-5 bit.
• Floor: but 4-bit is the line - dropping to Q3/Q2, reasoning decay erases that gain (then go small and precise).
• Headroom: if VRAM is spare, raise bits to Q6/Q8, or squeeze with IQ4_XS to push in a bigger model. Measure exact quality on your own tasks.

Related reading: 로컬 오픈 LLM VRAM·RAM 요구량, 직접 계산·실측, 노트북에서 돌리는 AI 모델 VRAM·RAM 요구량 실측

Note: bpw, GB, and ppl figures are public 2026 measurements and community estimates, many not lab-controlled (not permanent numbers). Quantization quality depends on model, task, and method (especially dynamic quantization), so measure exact loss on your own model and workload (these numbers are only a start). Quantization techniques move fast, so this is reviewed quarterly.

Reference links#

• llama.cpp quantize (quant tool/formats)
• llama.cpp quantization evaluation paper (arXiv)
• Unsloth (dynamic quantization, Dynamic 2.0)
• ggml (GGUF quantization format)
• llama-cpp-python (run quantized models)