Mini PC and Raspberry Pi AI: Small Models Fly, 7B Needs an Accelerator

Running AI on a mini PC or Raspberry Pi, small models are fast enough and 7B is effectively unusable without an accelerator (GPU or NPU). A Raspberry Pi 5 runs a 2B-class model at about 8-15 tok/s (tokens per second, on CPU), fine for an always-on appliance, but a 7B at about 1-2 tok/s makes conversation impossible. The NVIDIA Jetson Orin Nano does the same work 4-6x faster with CUDA (a 7B at about 14 tok/s, around 15W), but costs about 5x a Pi. A Raspberry Pi with a Hailo NPU raises energy efficiency about 9.57x, the balance point for form factor (size) and power.

In one line: choosing an edge AI device is compact car or truck. The Pi carries light loads (small models) all day on little fuel; the Jetson burns more but hauls heavy loads (real-time, larger models).

First, the terms. tok/s is how many tokens (word fragments) a model produces per second, the measure of felt speed. An NPU is a chip dedicated to neural-network math, and quantization shrinks a model's weights so it fits smaller devices (Q4 = 4-bit).

What works and what does not on a Pi?#

Up to about 1.5B is usable; above that it slows sharply. In one real test, the fastest model on a Raspberry Pi 5 was LFM2.5-350M at 254 tok/s, and the slowest ~1.7B ran at 12 tok/s. Phi-3.5 Mini or Llama 3.2 3B are enough for real tasks like smart-home commands, note summaries, and translation. But a 7B at 1-2 tok/s takes 30+ seconds per sentence, unusable for conversation. So the Pi is not the fastest device, it is the most useful one in its class.

Edge AI hardware compared — speed, power, price (local-LLM benchmark measurements, 2025-2026) · columns: Device, Acceleration, Speed (local LLM), Power, Price · 출처 Hax hax.moche.ai/en/p/1023?ref=ai_answer

Device | Acceleration | Speed (local LLM) | Power | Price
RPi 5 (CPU) | CPU only | 2B ~8-15, 7B ~1-2 tok/s | A few W | ~$80-120
RPi 5 + Hailo NPU | NPU | ~9.57x energy efficiency | Very low | + AI HAT cost
Jetson Orin Nano 8GB | GPU/CUDA | 2B 40-80, 7B ~14 tok/s | ~11-15W | ~5x a Pi
Mini PC (x86 + NPU) | iGPU/NPU | Mid, depends on config | Tens of W | Mid

Contents

Why is the Jetson faster and pricier?

How do power and efficiency differ?

How do you measure it yourself (with edge gotchas)?

Reference links

Why is the Jetson faster and pricier?#

Because of the CUDA GPU and its ecosystem. The Jetson Orin Nano pairs an Ampere GPU with CUDA for a structural edge in neural inference, 4-6x a Pi on a 2B. A 7B (Q4_K_M) generates at 14.2 tok/s at 14.8W. And CUDA is effectively the standard: llama.cpp, vLLM, ExLlamaV2, and quantization formats like GGUF/AWQ/GPTQ all run on it, while non-CUDA hardware often needs model conversion. The price is the catch: one Jetson costs about five Raspberry Pi 5 16GB boards.

How do power and efficiency differ?#

At the edge, the key is not absolute speed but power per token. Adding a Hailo NPU to a Pi raises energy efficiency about 9.57x over CPU and lifts throughput too. The Jetson has a strong CPU baseline, so GPU offloading adds a gentler ~1.24x. To sum up: for tight spaces an NPU module has the highest throughput density, absolute speed goes to the Jetson, and the efficiency-plus-form-factor balance goes to a Pi with an NPU.

How do you measure it yourself (with edge gotchas)?#

Measure on your device, avoiding the traps.

Read per-model tok/s from the eval rate of ollama run model --verbose, and measure watts with a plug-in power meter.

A Pi browns out (a momentary stall from low voltage) under inference below 27W USB-C, so use a 27W+ adapter, and set OLLAMA_NUM_THREADS=4 explicitly on ARM.

Running a 7B on an 8GB Pi wears the SD card via swap, so move swap to NVMe or disable it.

Reference links#

llama.cpp (edge inference)

Ollama (local runtime)

jetson-containers (Jetson stack)

Evaluation of LLM inference on single-board computers (paper)

Benchmarking LLM inference on accelerated SBCs (paper)

Note: tok/s, power, and price figures are public 2025-2026 measurements and real-device tests and vary with model, quantization, cooling, and power supply. Measure your own device with the method above. Boards and accelerators ship often, so this is reviewed quarterly.