Run a Local Open LLM: A 5-Minute Beginner Guide

A local open LLM takes about 5 minutes to start, and only one thing matters — pick a model that fits your memory (RAM/VRAM): the steps are (1) install Ollama with one line, (2) pull a model sized to your memory, (3) chat right away, and the rule of thumb is everything, since FP16 is about 2GB per 1B params and the most common Q4 quantization is a quarter of that, so a 7B fits in about 4-5GB — meaning 8GB runs a 7B (Q4), 16GB a 13B, 24GB a 32B — and the beginner pick is the well-balanced Qwen3 8B (Q4_K_M, about 5.5GB) at a measured 25-40 tok/s on an RTX 4060 or 3070.

In one line: picking a model is the amount of food for your pot. Pour a 70B into a small pot (8GB) and it overflows (swap) and burns; put in the right amount (7B) and it simmers nicely.

What do you install first (Ollama vs LM Studio)?#

Ollama if you like the command line, LM Studio if you like clicking. Both run the same models on the same engines (llama.cpp/MLX) and expose an OpenAI-compatible API. After a one-line install, ollama run qwen3:8b starts chatting in 30 seconds, which is great for dev and automation. LM Studio puts model search, download, and chat in a GUI, easier for a first look. Many people use both: choose in LM Studio, integrate with Ollama. For a beginner, install just one and get a single small model running end to end. Below is the 5-minute, 3-step flow.

Which model fits your computer?#

GPU VRAM comes first; Apple Silicon's unified memory is the weapon. The practical minimum is 16GB RAM plus 6GB+ VRAM (or Apple Silicon), enough for a 3-7B at Q4. Macs use the M3/M4/M5 unified memory as VRAM, so 64-128GB can run the largest open models (though measured 20-30% slower than equivalent VRAM). CPU-only runs Qwen3 4B (about 3GB) at 5-10 tok/s, fine for chat but sluggish for real-time conversation. And context (the KV cache) eats more memory: as chats grow, budget an extra 10-20% on top of the model size (Ollama's default context is 4,096 tokens). The rule of thumb below is the whole sizing calculation.

How fast is it, and where do beginners get stuck?#

Set expectations with numbers, and change only one thing at a time. A mid GPU (RTX 4060 12GB) does a 7B Q4_K_M at a measured 40-60 tok/s, while a CPU does a 3B at 5-10 tok/s (a GPU is effectively required for interactive 7B+). The three beginner traps: (1) choosing a model bigger than your memory crawls via swap (go one size smaller); (2) for quantization, Q4_K_M is the quality/speed sweet spot (preferred over Ollama's default Q4_0); (3) "local equals fully private" is only half true: inference is on your device, but the app may check for updates, send telemetry, search online, or expose a local server, so check offline mode, server settings, and logs before sensitive files. Below is a summary of the three traps.

How do you do it in 5 minutes?#

Start small and scale one step at a time.

• Install Ollama in one line and launch your first chat with ollama run qwen3:8b (for 8GB).
• Ask the same question to feel the tok/s, and if it is slow, drop one model size.
• Change only one of model size, quantization, context, or GPU at a time and watch the difference. Check offline settings before sensitive data.

Reference links

• Ollama (local runtime, one-line install)
• llama.cpp (inference engine, quantization)
• Qwen3 (beginner pick model)
• Llama models (Meta)
• Gemma (Google, HF)

Note: the memory-to-model-to-speed figures are public 2026 measurements and guides and vary by GPU, quantization, context, and cooling. Measure exact speed on your own device with the method above. Models and tools update often, so this is reviewed quarterly.