Running Local Open LLMs: Common Pitfalls and Fixes

When your local open LLM feels dumb, slow, or keeps crashing, the 2026 truth is that it's usually not the model - it's the defaults, because the parameters you set or ignore when launching a runner (Ollama, llama.cpp) matter more to real-world performance than your model or quantization choice, and the single most-missed thing is that the real culprit blowing up your VRAM is not the "weights" but the "KV cache" - this guide walks the recurring local-LLM traps as symptom -> cause -> fix, with numbers.

In plain terms: tuning a local LLM is like setting the options on a new car. Even with a great engine (model), if the side mirrors are folded in (context 2048) and the trunk is packed full (KV cache pre-allocated), it won't drive well. Before blaming "the car is bad (the model is dumb)," check the dashboard defaults first.

Why does a local LLM suddenly go dumb?#

Usually because the context window is still at its default, or the chat template doesn't match - not because the model is unintelligent. Ollama defaults num_ctx to 2,048 tokens, so on longer chats it silently discards the earlier part - the "ignores what I said 5 messages ago" symptom is exactly this. The fix is to raise the context window to what you actually need in the runner settings. The second common cause is chat-template mismatch. Qwen/Yi/Mistral use ChatML while Llama 3 uses its own header-token format, and if you load a GGUF manually with the wrong format it ignores the system prompt entirely or treats user messages as part of the assistant reply. Set --chat-template (llama.cpp) or the Modelfile TEMPLATE to match the model.

What actually blows up your VRAM?#

The KV cache, not the weights - it's what really limits context length on consumer hardware. The KV cache is pre-allocated for the entire window at startup: with --ctx-size 65536, even a 200-token prompt reserves memory for 65,536 tokens immediately. The diagram below is that pre-allocation trap - even a short prompt reserves the whole window.

So the first fix is matching context to VRAM: 8GB is safe at 32k, 12GB handles 64k for coding, 128k long-RAG needs headroom. The next-highest leverage is KV cache quantization - switching f16 -> q8_0 frees about 2GB at 12GB/64k, letting you offload 1-2 more GPU layers and raising token speed (Ollama OLLAMA_KV_CACHE_TYPE, llama.cpp --cache-type-k/-v). The diagram below is the real "weights vs KV cache" VRAM split.

For weights, Q4_K_M is the default - 75% smaller with near-zero quality loss. But Q2/Q3 collapse, and re-quantizing an already-quantized model (Q8->Q4) compounds errors, so don't.

Slow, but the GPU looks idle?#

GPU at 20-30% is not "idle" - it's a "memory-bandwidth bottleneck" - the most common misdiagnosis. In llama.cpp inference the GPU cores finish a batch fast and then wait for the next weights/cache to arrive, which shows as low utilization. Before trying to squeeze out speed, first check whether the KV cache is quantized and whether VRAM is near the limit. Two things to enable: (1) Flash Attention - it never materializes the full attention matrix, cutting memory O(n²)->O(n) and running faster (--flash-attn, OLLAMA_FLASH_ATTENTION=1; on Vulkan verify by driver). (2) Raise the prefill batch - at the default 512, a 16,000-token prompt is 32 passes, but bumping to 2048 makes it 8. The diagram below is that pass-count reduction.

Caution: KV quantization isn't always free - Gemma 3 drops to 20-30% GPU under Q8_0 KV (CPU at 100%) while a same-size Qwen is fine. Over-quantizing the K-cache causes tool-call and diff errors in coding agents, so test per model at your real serving context length.

So what's a safe default setup?#

The key is don't max everything out - first identify your bottleneck (compute, VRAM, bandwidth, CPU).

• Weights: Q4_K_M by default (Q5_K_M/Q6_K if you have headroom). Math/code want Q5 or higher. No Q2/Q3, no re-quantizing, and mind that GGUF/GPTQ/AWQ are format-incompatible.
• Context/memory: match context to VRAM (8GB=32k), enable KV cache quantization (q8_0) + Flash Attention. The real limiter is the KV cache, not the weights.
• Speed: prefill batch 2048; low GPU utilization is just a bandwidth bottleneck. A/B every setting at your real context length and model, then lock it in.

Related reading: 로컬 LLM, VRAM은 얼마나 필요할까, 로컬 코딩 보조 모델 2026: 직접 돌려보고 고른 현황과 추천

Related reading: 로컬 오픈 LLM VRAM·RAM 요구량, 직접 계산·실측, 4bit·8bit 양자화, 5분 시작 가이드(초보자용)

Reference links

• llama.cpp (local inference, flags)
• Ollama (local runner, env vars)
• FlashAttention (O(n²)->O(n) attention)
• k-quants intro PR (GGUF Q4_K_M etc.)
• Ollama KV cache quantization issue

Note: figures like the num_ctx default (2048), context guidance (8GB 32k / 12GB 64k), KV savings (~2GB), batch effect (2-3x), and quantization reduction (75%) are 2026 public and community-measured numbers that vary by model, runner, driver, and hardware (not permanent). KV-quantization quality impact varies a lot by model (e.g., Gemma 3), and flags/defaults change across runner versions. Always verify at your real context length after configuring. Runners and quantization practice move fast, so this is reviewed quarterly.