Local Coding Assistant Models: Speed and Quality, Measured

For a local coding assistant model, the first questions are not the HumanEval score but whether it runs at a usable speed on your GPU, and its license, and comparing them directly: on 8GB the default is Qwen2.5-Coder 7B (88.4% HumanEval), on 24GB it is the 32B (92.7%, GPT-4o class), for IDE autocomplete (FIM) Codestral is strongest at 95.3% but ships a non-commercial license, and for algorithm-heavy work DeepSeek-Coder-V2-Lite runs in 12GB thanks to its MoE efficiency.

In one line: choosing a coding model is choosing a tool. Chat-style code generation, IDE autocomplete, and algorithm solving each have a different best tool.

Which fits your VRAM?#

It splits cleanly by VRAM tier. At 8GB, Qwen2.5-Coder 7B (about 4.7GB at 4-bit) is genuinely competitive at 88.4%; at 12-16GB, the 14B or DeepSeek-Coder-V2-Lite; at 24GB, Qwen2.5-Coder 32B hits 92.7% and is the strongest open coding model that fits on a single card. The Qwen 14B keeps all parameters active, so its memory-bandwidth use is predictable and it generates tokens faster than the DeepSeek MoE. The diagram below is the VRAM ladder and each tier's default.

Is chat generation the same model as autocomplete?#

No. Different jobs, different best models. For chat-style code generation and refactoring, Qwen2.5-Coder leads across tiers; for autocomplete that fills in at the cursor (FIM, fill-in-the-middle), Codestral tops 2025 at 95.3% pass@1, beating closed models too. Codestral is a 22.2B dense model trained natively for FIM, so it feels instant as you type. DeepSeek-Coder-V2 is strong on LeetCode-style algorithms and math code. Below is the job-to-tool mapping.

Can you trust the benchmark scores as-is?#

Be careful. HumanEval scores vary by evaluation pipeline, and the stricter HumanEval+ (EvalPlus) typically lands 5-10 points lower (rankings mostly hold). The field has also shifted from simple generation (HumanEval) toward agentic benchmarks (SWE-bench) that navigate repos and run tests. So the table above holds for autocomplete and chat, but for multi-step agents look one tier up (the Qwen3-Coder and Devstral generation). Below is that shift in the benchmark's center of gravity.

How do you measure it yourself?#

Measure on your own code.

• Run HumanEval/EvalPlus at the same quantization per model, and read pass@1 alongside token speed (the eval rate from ollama run model --verbose).
• If autocomplete is the goal, build FIM cases in your own language and check cursor-fill accuracy.
• For a commercial product, filter by license before measuring; Codestral needs a separate agreement for paid deployment.

Reference links

• Qwen2.5-Coder (repo)
• DeepSeek-Coder-V2 (repo)
• Codestral (model card)
• EvalPlus / HumanEval+ (repo)
• Qwen2.5-Coder technical report (paper)

Note: HumanEval and FIM figures are public measurements following each model's official methodology (2024-2025) and vary with evaluation pipeline and quantization (HumanEval+ is lower). Measure your own codebase with the method above. Models and licenses change often, so this is reviewed quarterly.