4-bit vs 8-bit Quantization: Why 4-bit Runs ~29% Faster (Measured)

Quantization shrinks model weights to fewer bits so the same model runs in less memory. Measure it yourself and the surprise is that 4-bit (Q4_K_M) generates tokens about 29% faster than 8-bit (Q8_0), while losing only about 3-5% on perplexity (a metric of how much the model hesitates on the next word). 8-bit is nearly lossless (under 1%) but slower and heavier on memory.

In one line: for most local use, 4-bit (Q4_K_M) is the balance of speed, memory, and quality; reach for 8-bit only when you have spare VRAM and precision matters.

Why is 4-bit faster than 8-bit?#

It is counterintuitive but real. LLM token generation is bound by memory bandwidth. The GPU spends most of its time reading weights from VRAM per token, not doing multiply-adds, so fewer bits per parameter means fewer bytes to read each token and faster output. 8-bit reads one byte per parameter, 4-bit only about half a byte, so roughly half the data moves.

In llama.cpp's own benchmark (Llama 3.1 8B), Q8_0 generates about 29% slower than Q4_K_M, while Q5_K_M is only about 7% slower. The ratio is hardware-agnostic: whether you run an RTX 4090 or an M3 Max, the bottleneck is bytes read per token.

(Caveat: on GPU serving, 4-bit needs optimized kernels like Marlin or Triton to stay fast; without them, dequantization overhead can make it slower.)

How much quality do you actually lose?#

Less than you would expect. Q8_0 is effectively lossless versus FP16 (under 1%), and Q4_K_M costs about 3-5%. In one direct test, shrinking a 7.2GB model to 4.1GB at Q4_K_M (about 43% smaller) cost just 0.51% perplexity. The perplexity gap between Q4 and Q8 is around 0.05, imperceptible in conversation.

But dropping to 2-bit (Q2_K) spikes the loss to about 8%, a quality cliff. As the chart below shows, loss rises gently down to 4-bit, then falls off sharply at 2-bit, which is why the practical floor is usually 4-bit.

One caveat: perplexity is a blunt instrument. Coding, tool use, and long-context tasks tend to degrade faster than perplexity suggests, so confirm on your real workload.

So is 4-bit always the answer?#

One rule dominates: a bigger model at lower precision almost always beats a smaller model at higher precision when both fit. A 4-bit 70B beats an 8-bit 8B on essentially every benchmark, using less than twice the memory.

Quality loss also scales inversely with size: a 70B at 4-bit feels almost identical to FP16, while a 3B at 4-bit hallucinates noticeably more. Small models are safer at 8-bit; large models make 4-bit nearly free. Code holds up especially well, degrading only 2-3% on HumanEval and MBPP at Q4_K_M.

How do you measure it yourself?#

Do not trust someone else's table; measure on your machine. Run these to measure speed and quality separately.

• Speed: run ollama run model:q4_K_M --verbose and :q8_0 on the same prompt and compare the eval rate (tokens/sec).
• Quality: use llama.cpp's llama-perplexity -m model.gguf -f wiki.test.raw to compare perplexity across precisions on the same text.
• To push 4-bit accuracy higher, use AWQ or GPTQ instead of plain round-to-nearest (about 98.9% accuracy recovery with calibration data).

Quick decision rule: if VRAM >= the FP16 size use FP16; >= half use Q8_0; >= a third use Q4_K_M; below that, use a smaller model at Q4_K_M.

Note: figures are 2025-2026 llama.cpp/community measurements (Llama 3.1 8B, short context) and vary with model, architecture, calibration data, and backend; smaller models lose more at 4-bit. Measure your own with the method above. Reviewed quarterly.

Reference links#

• llama.cpp (GGUF quantization and benchmarks)
• 4-bit vs 8-bit analysis (SitePoint)
• Q4-Q8 quality loss numbers
• AWQ paper (Lin et al., 2023)
• GPTQ paper (Frantar et al., 2022)