Local Multimodal (VLM) for Beginners: Suspect the mmproj Projector

A local multimodal model (VLM) is an LLM with eyes - it "looks at" images, tables, and documents and answers in text. It starts in five minutes, and the one thing every beginner gets stuck on is the "mmproj projector": without it the model loads fine and answers text, but it cannot see your image (it silently ignores it). Luckily Ollama wires the projector automatically for supported models, so ollama run qwen3-vl:8b lets you send an image and get an answer in one line. The trap is not "it does not run" but "you thought it worked, but its eyes never opened."

In one line: a VLM is an assistant with eyes. A text LLM has only ears (text); a VLM has eyes plus ears. It is the opposite of image generators (SDXL, Flux): those go text to image, this goes image to text.

First, the terms. VLM is a Vision-Language Model, mmproj is the "projector" file that maps the image features from the vision encoder into tokens the LLM understands, and OCR is reading the text inside an image out as text.

What does a local VLM do?#

Document OCR, screenshot Q&A, chart reading, and image description, all locally and 100% offline with no API key. The architecture is three stages: a vision encoder (CLIP/SigLIP) sees the image, a projector maps those features into the token space the LLM understands, and the LLM answers in text. That middle "projector" is the mmproj file, the heart of vision. Because a vision encoder is stacked on, expect it to be about 30-60% slower than a same-size text model in measured community reports. The "eyes" are not free - you pay in speed.

What do beginners get stuck on most?#

The mmproj projector not attaching, so the model's "eyes never open." Ollama auto-bundles the projector for supported models like qwen3-vl, MiniCPM-V, and LLaVA, so it just works. But the Qwen 3.5 and 3.6 family, which reworked vision, ship the projector as a separate GGUF, and Ollama's default flow does not wire it up, so text works but image input fails (as of May 2026). Then you point llama.cpp at the projector with --mmproj or use LM Studio (MLX-VLM on Apple Silicon). So a beginner's rule #1 is: "if the image is ignored, suspect the projector, not the model."

Which model for your GPU?#

VRAM decides the model. With 8GB+, Qwen3-VL 8B is a comfortable all-rounder (OCR, charts, 256K context). At 6-8GB, MiniCPM-V 4.5 or LLaVA 1.6 7B (~6GB) is the standard start; with only 2GB, Moondream2 is practically the only choice. Watch out: a high-resolution image makes image tokens explode, so VRAM is not just the model size (resize smaller). And heavily quantized small vision models are weak at Korean text OCR, so if Korean documents are central, verify on a bigger model yourself.

How do you do it in 5 minutes?#

Start with the easiest path.

• Install Ollama and run ollama run qwen3-vl:8b (8GB+), or MiniCPM-V at 6-8GB, then send an image.
• If the image is ignored, suspect the projector, not the model - switch to a supported model or attach it via llama.cpp --mmproj.
• Resize high-resolution images, and measure Korean OCR on your own documents (leaderboards skew English and are only a start).

Reference links#

• Ollama (local VLM runtime)
• llama.cpp (mmproj/libmtmd vision)
• Qwen3-VL (all-rounder VLM)
• MiniCPM-V (light OCR/documents)
• moondream (ultra-light VLM)

Note: model, VRAM, and speed figures are public 2026 guides and model cards and vary by quantization, resolution, and language. Ollama not wiring new-model mmproj reflects a May 2026 situation and may change, so if an image is ignored, check your tool version first. Measure exact Korean OCR quality on your own documents (leaderboards are only a start). The VLM ecosystem moves fast, so this is reviewed quarterly.