Local Multimodal (Image+Text) Models: Common Pitfalls and Fixes

The most common misjudgment with local multimodal (image+text) models in 2026 is concluding "the model is dumb" when it reads an image wrong. The real cause is usually not the model but the "vision pipeline". A VLM doesn't understand a photo directly - a vision encoder turns the image into "visual tokens," and a projector (mmproj) maps those into a form the LLM can read before it processes them like words. So three things break silently: the projector file is missing or mismatched, resolution explodes the visual-token count until VRAM overflows, and tiling trades detail for speed. This guide walks the local-VLM-specific traps as symptom -> cause -> fix, with numbers.

In plain terms: a VLM is a conversation through an interpreter. The LLM can't see the image directly - the vision encoder + projector, the interpreter, must translate the picture into "words." Drop the interpreter (mmproj) and there's no conversation at all; make the picture too big (high resolution) and the interpreter pours out too many words, overflowing the desk (VRAM). Shrink the picture hard to summarize it (fixed resize) and it misses the small print.

Why does it fail to read the image, or hallucinate it?

Because the projector (mmproj) file is missing or architecturally mismatched. In llama.cpp a VLM is a pair - base model + mmproj - and loading the base alone cannot process images at all. A mismatched pair (another model's projector) fails or hallucinates - always use the matching projector for that exact model. Second, image tokens overflow the micro-batch and die on an assertion - one photo becomes hundreds of tokens, so raise --ubatch-size to 512 or more. Third, current builds include projector memory in the --fit calculation (per a recent PR), so on OOM suspect an old build, another process, or a batch built without the projector. In short: most "it won't read the image" problems are about wiring (files, batch), not weight (model intelligence).

Why are resolution and tiling so dangerous?

Because high-resolution images are the biggest VRAM killer for local VLMs. The vision encoder's self-attention builds a score tensor (KQ) that is O(N^2) in the number of visual tokens N - at high resolution KQ alone reaches several GB, making single-GPU inference impractical. Mitigations: apply FlashAttention in the vision path too, tile the Q tensor (breaks cache-friendliness so a small speed hit, but lets you shrink vision VRAM arbitrarily), or offload vision weights to CPU (streamed CPU->GPU at execution). And tiling itself is double-edged: the old way, resizing to a fixed resolution, causes geometric distortion and loses fine detail (small text, tables, chart OCR collapse), while dynamic tiling splits the image and encodes each region to keep quality but generates thousands of visual tokens, exploding compute and latency. So the practical answer is to explicitly control token count via resolution and tile caps.

Which local VLM goes on which VRAM?

How a model handles tokens decides the choice. The Gemma 3 family resizes images to 896x896 and encodes a fixed 256 tokens - VRAM is predictable but it can be weak on non-standard ratios and dense documents (though document understanding is strong - DocVQA 27B at 85.6). By contrast Qwen2.5/3-VL uses native resolution and variable tokens, so it's strong on OCR and fine detail across aspect ratios but its memory swings with image size. VRAM tiers (from a deployment guide): edge (Jetson Orin Nano) = moondream 1.8b, qwen2.5vl 3b, gemma3 4b; 8-16GB single GPU = qwen2.5vl 7b, gemma3 4b, llava 7b; 24GB+ = gemma3 12b/27b, qwen3-vl 32b. When memory is tight, Gemma 3 4B (~4.2GB) fits well, and the QAT variant gemma3:<size>-it-qat, trained at low precision, beats ordinary post-training quantization at the same bit width. The core lesson: a VLM is not "an LLM with eyes" but a pipeline of two models (vision + language) joined by a projector, and most failures happen at that joint, not in the language.

So what's a safe local-VLM setup?

The key is match the projector pair, control tokens/VRAM via resolution, and pick the model to fit your token strategy.

• Wiring: base + matching mmproj (two files), --ubatch-size 512+ to avoid image-token assertions, a current build so projector memory is in --fit.
• Resolution/VRAM: for high resolution use vision FlashAttn, Q-tiling, CPU offload, and cap resolution/tiles to control visual tokens (prevent dynamic-tile blowup).
• Model choice: for documents and predictable VRAM, Gemma 3 (fixed 256); for OCR, fine detail, odd ratios, Qwen-VL (native variable); when tight, a QAT variant. A/B every choice on your own image set, then lock it in.

Related reading: 로컬 코딩 보조 모델 2026: 직접 돌려보고 고른 현황과 추천, Ollama·LM Studio·llama.cpp 실행기, 2026년에는 무엇을 고를까?

Related reading: 로컬 멀티모달(VLM) 모델 VRAM·RAM 요구량 실측, 로컬 멀티모달(VLM) 모델, 5분 시작 가이드(초보자용)

Reference links

• llama.cpp multimodal docs (mmproj, vision)
• llama.cpp repository
• Qwen2.5-VL (native resolution, variable tokens)
• InternVL (open VLM family)
• Open-source VLM guide (BentoML)

Note: figures and behaviors like the mmproj pairing, --ubatch 512, the O(N^2) vision KQ, Gemma 3's 896->256, Qwen-VL's native/variable tokens, DocVQA 85.6, the VRAM tiers, and QAT are based on 2026 llama.cpp official docs/PRs, model cards, and public benchmarks, and vary by build, model, version, and image (not permanent; the model card and llama.cpp docs are the final authority). Generations move on - Gemma 3 -> Gemma 4, Qwen2.5-VL -> Qwen3-VL - so check the latest generation for new builds. Always validate on your own images and documents. The local-VLM stack moves fast, so this is reviewed quarterly.