Mini PC and Raspberry Pi AI: A 5-Minute Beginner Guide

AI runs on a mini PC or Raspberry Pi in five minutes, and the key is picking a "small model." On a Raspberry Pi 5, Gemma 3 1B runs a comfortable measured ~18-22 tok/s, while putting a 7B on the same board gives ~1-2 tok/s - unusable for chat, which breeds the myth that "edge AI does not work." And the second trap: bolting on an AI HAT (NPU) does not make Ollama faster by itself - most NPUs are for vision and pre-compiled models, so you cannot load just any LLM. So success is not an expensive accelerator but matching the model to the box first.

In one line: success on a mini PC or Raspberry Pi is not an expensive accelerator but "fitting a small quantized model to the box" - on a Pi 5, Gemma 3 1B is a comfortable ~18-22 tok/s while a 7B is an unusable ~1-2 tok/s, and an NPU HAT is usually vision-only and will not speed up an LLM by itself.

In plain terms: a small box is a compact car. It does the grocery run fine, but load a truck's cargo (7B) and it will not move. Fit the cargo to the box (1-4B) and it becomes a reliable "always-on assistant" that runs every day.

Can a small box really run AI?#

Yes, but only a small, quantized model. In 2026, aggressive quantization (Q4_K_M by default) and compact models make chatbots, home automation, and offline assistants practical on a Pi 5 (8GB). Measured, Gemma 3 1B is ~18-22 tok/s (about 0.8GB), Gemma 3 4B ~8-11, and Llama 3.2 3B ~5 tok/s. But the 8GB Pi is the minimum (4GB is too little), and since it pegs all 4 cores, active cooling and NVMe (models load 2-5GB) are practically required. Use ollama run --verbose to watch tok/s live.

The bars below are measured Pi 5 speeds by model size: smaller gets sharply faster, and 7B drops into the no-chat zone.

What do beginners get wrong most?#

Two things: sizing the model bigger than the box, and mistaking the NPU for a cure-all. First, a 7B "runs" on a Pi but swaps at 1-2 tok/s and is useless - the Pi is the entry point, not the endgame. The second is the real trap: the common AI HAT+ (Hailo-8L, 13 TOPS) is a "vision" accelerator and cannot run LLMs ("you can't load Llama onto it"). LLMs need only the newer AI HAT+ 2 (Hailo-10H, 40 TOPS), and even then only compiled versions from Hailo's model zoo, limited to 1-1.5B. Bottom line: if LLMs are the goal, CPU Ollama is usually easier, and the NPU shines for real-time vision.

The same "AI HAT" does different things by generation. The split below is what runs and what does not.

Which board for your goal?#

Match budget, goal, and speed. (1) To learn cheapest and most reliably, Pi 5 8GB + Ollama + Gemma 3 1B. (2) To run 3-7B comfortably, the Jetson Orin Nano Super ($249) is CUDA, so llama.cpp, Ollama, and vLLM "just work" - 1-3B ~28-55 tok/s, and even 7B ~14-15 tok/s (about 15W). (3) Justify a Hailo HAT only when real-time vision (like object detection) is the real goal (a light LLM is a bonus). A mini PC (built-in NPU/iGPU) wins on always-on power, but LLM speed still comes down to quantization and model size.

How do you do it in 5 minutes?#

Start with the easiest path.

• On a Pi 5 (8GB) with active cooler and NVMe, install the official Ollama script (ARM64, systemd automatic) and start with ollama run gemma3:1b.
• If it is slow, shrink the model to 1-4B, not 7B, and use --verbose to watch tok/s and fit your goal (always-on assistant, automation).
• Buy an accelerator only when vision is the goal, and for LLM-only first measure whether CPU suffices (on your own task, not a leaderboard).

Reference links

• Ollama (edge LLM runtime)
• llama.cpp (light inference engine)
• Hailo Model Zoo (NPU-only compiled models)
• RPI-Hailo-Hat-Ollama (Pi + NPU worked example)
• Gemma 3 1B (light model, Hugging Face)

Note: tok/s, power, and size figures are public 2026 measurements (community and vendor benchmarks) and vary by quantization, cooling, power, and version. NPU-supported models in particular are tied to a vendor model zoo, so check current threads before buying. Measure exact speed on your own board and model (leaderboards are only a start). Edge hardware and models move fast, so this is reviewed quarterly.