Local Coding Assistant Models: Speed and Quality, Measured

For a local coding assistant model, the first questions are not the HumanEval score but whether it runs at a usable speed on your GPU, and its license, and comparing them directly: on 8GB the default is Qwen2.5-Coder 7B (88.4% HumanEval), on 24GB it is the 32B (92.7%, GPT-4o class), for IDE autocomplete (FIM) Codestral is strongest at 95.3% but ships a non-commercial license, and for algorithm-heavy work DeepSeek-Coder-V2-Lite runs in 12GB thanks to its MoE efficiency.

In one line: choosing a coding model is choosing a tool. Chat-style code generation, IDE autocomplete, and algorithm solving each have a different best tool.

Which fits your VRAM?#

It splits cleanly by VRAM tier. At 8GB, Qwen2.5-Coder 7B (about 4.7GB at 4-bit) is genuinely competitive at 88.4%; at 12-16GB, the 14B or DeepSeek-Coder-V2-Lite; at 24GB, Qwen2.5-Coder 32B hits 92.7% and is the strongest open coding model that fits on a single card. The Qwen 14B keeps all parameters active, so its memory-bandwidth use is predictable and it generates tokens faster than the DeepSeek MoE. The diagram below is the VRAM ladder and each tier's default.

Local coding models compared — HumanEval, VRAM, license (public measurements, 2024-2025) · columns: Model, Size, HumanEval, VRAM (4-bit), Strength / license · 출처 Hax hax.moche.ai/en/p/1028?ref=ai_answer

Model | Size | HumanEval | VRAM (4-bit) | Strength / license
Qwen2.5-Coder 7B | 7B | 88.4% | ~5-8GB | Best value default, 128K, Apache-2.0
Qwen2.5-Coder 32B | 32B | 92.7% | ~24GB | GPT-4o class, single 24GB, Apache-2.0
DeepSeek-Coder-V2-Lite | 16B (2.4B active) | 81.1% | ~12GB | Algorithms, MoE efficiency, MIT
Codestral | 22.2B | 86.6% (FIM 95.3%) | ~14GB | Best autocomplete (FIM), non-commercial

Contents

Is chat generation the same model as autocomplete?

Can you trust the benchmark scores as-is?

How do you measure it yourself?

Is chat generation the same model as autocomplete?#

No. Different jobs, different best models. For chat-style code generation and refactoring, Qwen2.5-Coder leads across tiers; for autocomplete that fills in at the cursor (FIM, fill-in-the-middle), Codestral tops 2025 at 95.3% pass@1, beating closed models too. Codestral is a 22.2B dense model trained natively for FIM, so it feels instant as you type. DeepSeek-Coder-V2 is strong on LeetCode-style algorithms and math code. Below is the job-to-tool mapping.

Can you trust the benchmark scores as-is?#

Be careful. HumanEval scores vary by evaluation pipeline, and the stricter HumanEval+ (EvalPlus) typically lands 5-10 points lower (rankings mostly hold). The field has also shifted from simple generation (HumanEval) toward agentic benchmarks (SWE-bench) that navigate repos and run tests. So the table above holds for autocomplete and chat, but for multi-step agents look one tier up (the Qwen3-Coder and Devstral generation). Below is that shift in the benchmark's center of gravity.

How do you measure it yourself?#

Measure on your own code.

Run HumanEval/EvalPlus at the same quantization per model, and read pass@1 alongside token speed (the eval rate from ollama run model --verbose).

If autocomplete is the goal, build FIM cases in your own language and check cursor-fill accuracy.

For a commercial product, filter by license before measuring; Codestral needs a separate agreement for paid deployment.

Reference links

Qwen2.5-Coder (repo)

DeepSeek-Coder-V2 (repo)

Codestral (model card)

EvalPlus / HumanEval+ (repo)

Qwen2.5-Coder technical report (paper)

Note: HumanEval and FIM figures are public measurements following each model's official methodology (2024-2025) and vary with evaluation pipeline and quantization (HumanEval+ is lower). Measure your own codebase with the method above. Models and licenses change often, so this is reviewed quarterly.