4-bit and 8-bit Quantization for Beginners: Just Start With Q4_K_M

Quantization is compression that shrinks model weights from 16-bit down to 4 or 8 bits so the same model fits in less memory. The one answer a beginner should memorize: just start with Q4_K_M (Q5_K_M if you have room, Q8_0 for coding and reasoning). A 70B at FP16 (the 16-bit original) is about 140GB and will not fit a gaming PC, but at Q4 it drops under about 40GB. The price is a small quality loss (measured as perplexity, a score of how much the model hesitates on the next word): under 0.5% for Q8 and about 3-5% for Q4_K_M on most benchmarks. The key principle is better to run a small model well than a large one badly.

In one line: quantization is compressing a photo to JPG. 8-bit shrinks it invisibly (high-quality JPG), 4-bit is slightly noticeable but plenty good, and going to 2-bit makes the image fall apart.

A bit is the precision used to store one number. FP16 records each weight finely with 16 bits; Q4 records it coarsely with 4 bits - the file shrinks, but small rounding errors accumulate.

Which level should you pick (a 5-minute decision)?#

Q4_K_M by default, Q5/Q8 if you have room, Q3 and below only for experiments. For the math, reckon roughly 2GB per 1B at FP16, 1GB at Q8, and 0.5GB at Q4. And even at the same 4 bits, the one with a "K" is better: unlike legacy Q4_0, Q4_K_M (a K-quant) keeps the sensitive layers at 6-bit, recovering 5-8% quality at the same bit count. So when Hugging Face shows both Q4_0 and Q4_K_M, always pick Q4_K_M. One more thing: versions with "imat" (importance matrix) in the filename are a measured 2-4% better at the same bits.

How do you get a quantized model?#

Pick one tag or one file and you are done. Ollama's ollama pull model usually pulls a Q4_K_M-class default, while LM Studio and Hugging Face let you choose the level by filename like model.Q4_K_M.gguf. GGUF is the standard that runs everywhere (CPU, consumer GPU, Apple Silicon), so it is the right answer for a beginner. Go deeper and formats split: AWQ for GPU-server quality, GPTQ for NVIDIA throughput, and MXFP4 for models like GPT-OSS (but native only on H100/Blackwell-class GPUs; older GPUs fall back to BF16 and memory rises). A beginner can ignore all three and be fine with GGUF Q4_K_M.

Where do beginners get stuck?#

Three things: too-low bits, the KV cache, and re-quantizing.

• Too low: below Q4 (Q3/Q2) quality drops sharply. A 7B at Q6 usually beats a 13B at Q2 (faster, with more context headroom).
• KV cache: even if the model is 4.4GB, inference needs 1-3GB more for context (the KV cache) - always add it to your VRAM math.
• No re-quantizing: shrinking an already-quantized model (Q8 to Q4) compounds error. Always quantize from the FP16 source.

How do you try it in 5 minutes?#

Start with the safest default.

• Launch an 8B-class model with the ollama pull default (usually Q4_K_M) and judge the felt quality.
• If you have room, compare the same model side by side at Q5_K_M and Q8_0 and see whether you can tell the difference.
• Since quantization hits coding and math harder, prefer one level higher (Q6/Q8) for those tasks.

Reference links#

• llama.cpp (GGUF and quantization)
• GGUF/ggml (format)
• AWQ (activation-aware quantization repo)
• GPTQ (repo)
• AWQ (paper)

Note: size and quality-loss figures are public 2026 measurements and vary by model, task, and quantization implementation (quantization hits coding and STEM harder than general chat). Measure exact quality on your own tasks with the method above. Quantization formats update often, so this is reviewed quarterly.