4-bit and 8-bit Quantization: Common Pitfalls and Fixes

Two facts dominate 4-bit/8-bit quantization in 2026. First, 4-bit is the "safe floor" and below it is a "cliff" - for most tasks 4-bit costs 1-2% versus FP16, but from 3-bit down, math and multi-step reasoning break. Second, most quantization "failures" are self-inflicted: mismatching the format to the engine (GGUF into GPU serving), measuring without the kernel (skipping Marlin), calibrating on out-of-domain data. And one absolute rule for agents/tool-calling: never Q4 the V-cache - the tail-end quality drop breaks tool-use. This guide walks the quantization-specific traps as symptom -> cause -> fix, with numbers.

In plain terms: quantization is like compressing a photo to JPEG. Moderate compression (4-bit) is invisible, but over-compress (2-bit) and the text smears (math collapse). And if you pick the wrong file format (GGUF into a GPU-only viewer), even a fine photo loads slowly - not because of image quality, but because you didn't match the viewer (engine).

Why are most quantization "failures" self-inflicted?#

Because you mismatch the format to the serving stack, skip the kernel, and measure in isolation. Formats split three ways: GGUF is for llama.cpp/Ollama (CPU+GPU hybrid), AWQ is the GPU-production default (vLLM), GPTQ has broad coverage. Force GGUF into vLLM and it runs at 93 tok/s, 958ms TTFT - slow not because of the algorithm but because GGUF's layout was built for laptops and Macs, so GPU engines never built fast loaders for it. Kernels are the other half: without Marlin you're slower than FP16 (GPTQ -40%, bitsandbytes -64%). With Marlin, GPTQ is 2.6x and AWQ 10.9x faster (Marlin-AWQ is the sweet spot: 51.8% Pass@1, 741 tok/s). Calibration is a trap too - GPTQ/AWQ depend on representative samples, so calibrating on out-of-domain data silently degrades quality. In short, measure the "format-kernel-workload" combination, not the format in isolation.

How far can you shrink the KV cache?#

Lossless to 4-bit, a cliff at 2-bit. In arXiv benchmarks, dropping KV from 16 to 4 bits causes almost no accuracy loss on Qwen3-8B and LLaMA3-8B, but going to 2-bit crashes the average by -15.23 and -10.15. Math falls hardest: Qwen3-8B's MATH goes from ~88 at FP16/4-bit to 47.29 at 2-bit, a roughly 41-point cliff. And degradation starts earlier than you'd think - one 2026 paper reports that instruction-following already wobbles at 8-bit KV, with Pass_strict down ~10 points versus FP16 (about one in four prompts now violates a constraint), collapsing by 6-bit. Practical rules: (1) Q8 KV is the default (halves VRAM, <0.1% quality loss), (2) if VRAM is tight, asymmetric Q4-K + Q8-V (better quality than symmetric Q4+Q4 at the same VRAM - because only Keys have channel outliers), (3) never Q4 the V-cache for agents/tool-calling (Q8 minimum), (4) quantize weights first, KV second (weights are the bigger lever), (5) always pair quantized KV with flash-attention (or decoding crawls).

So what's a safe quantization setup?#

The key is 4-bit as the floor, match the format to the stack, and prefer a bigger model at lower precision.

• Bits: 4-bit is the safe floor (1-2% vs FP16). Math/code/reasoning want Q5_K_M/Q6_K headroom. No 3-bit or below, no re-quantizing (Q8->Q4) - it compounds errors.
• Format: ==NVIDIA GPU reasoning/agents = AWQ (+Marlin); CPU/Mac/offload = GGUF (Q4_K_M)==. Check imatrix for IQ-quants, and calibrate on in-domain data.
• KV/strategy: Q8 KV default, asymmetric Q4-K+Q8-V if tight, no Q4 V-cache for agents. And at equal size prefer the bigger model at lower precision (70B Q4 >> 7B FP16 - larger models absorb loss better). A/B every setting on your data and task, then lock it in.

Related reading: 로컬 LLM, VRAM은 얼마나 필요할까, 로컬 코딩 보조 모델 2026: 직접 돌려보고 고른 현황과 추천

Related reading: 4bit·8bit 양자화 VRAM·RAM 요구량, 직접 실측, 로컬 오픈 LLM VRAM·RAM 요구량, 직접 계산·실측

Reference links

• AWQ (MIT Han Lab, activation-aware quantization)
• GPTQ (IST-DASLab)
• llama.cpp (GGUF, quantize)
• KIVI: 2-bit KV cache quantization (arXiv)
• Qwen llama.cpp quantization official docs

Note: figures like the 4-bit loss (1-2%), 2-bit cliff (MATH 88->47), 8-bit KV degradation (-10 pts), perplexity (6.74, 6.84), and Marlin speedups (2.6-10.9x) are 2026 arXiv, public, and commercial benchmarks that vary by model, architecture (e.g., DeepSeek MLA tolerates Q4 worse), data, and kernel (not permanent; many blog benchmarks are directional). Formats, kernels, and defaults change across stack versions, so validate on your own data before production. Sub-4-bit is a research area that fails without specialized techniques (Kitty, KVarN, SAW-INT4). Quantization practice moves fast, so this is reviewed quarterly.