Mini PC and Raspberry Pi AI: Small Models Fly, 7B Needs an Accelerator

Running AI on a mini PC or Raspberry Pi, small models are fast enough and 7B is effectively unusable without an accelerator (GPU or NPU). A Raspberry Pi 5 runs a 2B-class model at about 8-15 tok/s (tokens per second, on CPU), fine for an always-on appliance, but a 7B at about 1-2 tok/s makes conversation impossible. The NVIDIA Jetson Orin Nano does the same work 4-6x faster with CUDA (a 7B at about 14 tok/s, around 15W), but costs about 5x a Pi. A Raspberry Pi with a Hailo NPU raises energy efficiency about 9.57x, the balance point for form factor (size) and power.

In one line: choosing an edge AI device is compact car or truck. The Pi carries light loads (small models) all day on little fuel; the Jetson burns more but hauls heavy loads (real-time, larger models).

First, the terms. tok/s is how many tokens (word fragments) a model produces per second, the measure of felt speed. An NPU is a chip dedicated to neural-network math, and quantization shrinks a model's weights so it fits smaller devices (Q4 = 4-bit).

What works and what does not on a Pi?#

Up to about 1.5B is usable; above that it slows sharply. In one real test, the fastest model on a Raspberry Pi 5 was LFM2.5-350M at 254 tok/s, and the slowest ~1.7B ran at 12 tok/s. Phi-3.5 Mini or Llama 3.2 3B are enough for real tasks like smart-home commands, note summaries, and translation. But a 7B at 1-2 tok/s takes 30+ seconds per sentence, unusable for conversation. So the Pi is not the fastest device, it is the most useful one in its class.

Why is the Jetson faster and pricier?#

Because of the CUDA GPU and its ecosystem. The Jetson Orin Nano pairs an Ampere GPU with CUDA for a structural edge in neural inference, 4-6x a Pi on a 2B. A 7B (Q4_K_M) generates at 14.2 tok/s at 14.8W. And CUDA is effectively the standard: llama.cpp, vLLM, ExLlamaV2, and quantization formats like GGUF/AWQ/GPTQ all run on it, while non-CUDA hardware often needs model conversion. The price is the catch: one Jetson costs about five Raspberry Pi 5 16GB boards.

How do power and efficiency differ?#

At the edge, the key is not absolute speed but power per token. Adding a Hailo NPU to a Pi raises energy efficiency about 9.57x over CPU and lifts throughput too. The Jetson has a strong CPU baseline, so GPU offloading adds a gentler ~1.24x. To sum up: for tight spaces an NPU module has the highest throughput density, absolute speed goes to the Jetson, and the efficiency-plus-form-factor balance goes to a Pi with an NPU.

How do you measure it yourself (with edge gotchas)?#

Measure on your device, avoiding the traps.

• Read per-model tok/s from the eval rate of ollama run model --verbose, and measure watts with a plug-in power meter.
• A Pi browns out (a momentary stall from low voltage) under inference below 27W USB-C, so use a 27W+ adapter, and set OLLAMA_NUM_THREADS=4 explicitly on ARM.
• Running a 7B on an 8GB Pi wears the SD card via swap, so move swap to NVMe or disable it.

Reference links#

• llama.cpp (edge inference)
• Ollama (local runtime)
• jetson-containers (Jetson stack)
• Evaluation of LLM inference on single-board computers (paper)
• Benchmarking LLM inference on accelerated SBCs (paper)

Note: tok/s, power, and price figures are public 2025-2026 measurements and real-device tests and vary with model, quantization, cooling, and power supply. Measure your own device with the method above. Boards and accelerators ship often, so this is reviewed quarterly.