Local Open LLMs at 4-bit: Speed vs Quality, Measured (8B to 70B)

Running the same 4-bit (Q4_K_M) open LLMs locally, speed is decided by GPU memory bandwidth and quality is decided by model generation, not parameter size alone. On an RTX 4090, an 8B model is fastest at roughly 95-110 tokens/sec, a 14B runs around 60-70 tok/s, and a dense 32B drops to about 20 tok/s. Yet a newer 14B (Qwen3) clearly beats an older 14B on MMLU, so faster is not the same as weaker.

In one line: smaller is faster, but the speed-quality sweet spot is usually a 14B or an MoE (30B-A3B).

What decides token speed?#

Token generation is bound more by memory bandwidth than by raw compute. For every token, the GPU must read the model weights once from VRAM, and the multiply-add units often sit idle waiting for that read. So "tokens per second" is really "how many bytes per second the card can read."

When a model fits entirely in GPU VRAM, that card's bandwidth is the speed ceiling. So a small 8B model clears 100 tok/s on an RTX 4090 (about 1,008 GB/s) but only 45-55 tok/s on an M3 Max (about 400 GB/s). Once a model overflows VRAM, some layers offload to CPU memory, and every pass over those layers stalls on slow system RAM, which is why a 70B falls to 8-15 tok/s on a 24GB card.

Here are community-measured numbers at the same 4-bit precision on one RTX 4090, gathered into a single table.

Does quality follow size?#

No. Generation beats size. By the Qwen3 technical report, Qwen3-14B scores about 81.05 on MMLU, ahead of the previous-generation Qwen2.5-14B (79.66) and Gemma-3-12B (73.87). On the harder MMLU-Pro the gap widens further (Qwen3-14B 61.03 vs Gemma-3-12B 44.91). In other words, a new-generation 14B approaches an old 32B.

The MoE (Mixture of Experts) model Qwen3 30B-A3B does not switch on all 30B; it activates only about 3B parameters per token. Fewer active weights to read make it faster than a dense 8B, while the full 30B of stored knowledge lets it deliver 14B-class quality. Judge by generation and architecture, not parameter count.

How do you measure it yourself?#

Do not trust someone else's numbers; measure on your own machine. The easiest path is Ollama.

• ollama run qwen3:14b --verbose prints the eval rate (tokens/sec) after each answer.
• llama.cpp's llama-bench -m model.gguf reports prompt and generation speed separately.
• Average three runs under identical conditions (same prompt, context length, and quantization) for a fair comparison.

The backend matters too: llama.cpp is usually 3-10% faster than Ollama on NVIDIA, and on Apple Silicon MLX can be 20-30% faster than llama.cpp.

Which one should you pick?#

On a single consumer GPU (12-24GB), a 4-bit Qwen3 14B is the default balance of speed and quality. Step down to an 8B or an MoE (30B-A3B) for snappier replies, or up to a 32B or 70B (or a unified-memory Mac) when top quality matters and slower is acceptable. Narrow the field by your VRAM budget as below, then, before deciding, run it once on your own task (summaries, coding, tool calls) rather than trusting the table.

Note: figures are 2025-2026 community-measured ranges (Ollama/llama.cpp, Q4_K_M, short context) and vary with backend, version, context length, and thermals. Measure your own with the method above. Open-weight models move fast, so this is reviewed quarterly.

References#

• Qwen3 Technical Report (arXiv 2505.09388) — source for the MMLU and MMLU-Pro quality scores
• llama.cpp (ggml-org) — llama-bench for measuring speed
• Ollama — ollama run --verbose prints the eval rate (tok/s)
• NVIDIA GeForce RTX 4090 — ~1,008 GB/s memory bandwidth
• Apple MacBook Pro (M3 Max) tech specs — up to 400 GB/s unified memory bandwidth