Running AI on a Laptop: What Belongs on the NPU vs the GPU?

AI on a laptop starts in 5 minutes, but the rules differ from a desktop - thermals, battery, and the NPU decide everything, so the key is small, always-on tasks on the NPU all day, and heavy tasks on the GPU or unified memory. Transcription (Whisper base/small), live captions, embeddings, and 1-4B small LLMs run quietly all day on an NPU at a measured 5-10W, while 7B+ chat LLMs, image generation (SDXL), and Whisper Large need a discrete GPU or unified memory. And RAM and memory bandwidth, not the marketed TOPS number, are the real bottleneck.

In one line: run small, always-on AI on the NPU at low power all day, and heavy models on the GPU or unified memory in short bursts. RAM, bandwidth, and thermals set real speed, not the TOPS on the spec sheet.

In plain terms: laptop AI is fuel-efficient city driving. The NPU carries small loads on little fuel all day (battery), while big loads (large models) burn a lot and need the charger (wall power) and a big engine (GPU).

What can your laptop do (by class)?#

NPU laptops do small AI all day, GPU laptops do big models in short bursts, and a MacBook does as much as its unified memory. The Copilot+ PC bar is a 40+ TOPS NPU and 16GB RAM, running transcription, captions, Windows Studio Effects, and small LLMs locally. Gaming/discrete-GPU laptops run 7-13B LLMs and image generation fast in measurements but draw 30-40W, so battery is short. A MacBook (Apple Silicon) uses unified memory wholesale: 16GB handles around 7B, 128GB up to 70B-class - but a fanless design thermally throttles on sustained inference past 10-15 minutes. Here an NPU (neural processing unit) is a small chip dedicated to AI matrix math at low power, while unified memory is an architecture where CPU and GPU share the same RAM so a large model can load whole.

How do an NPU and a GPU differ?#

An NPU is "steady" on little power; a GPU is "hard" on lots of power. The NPU specializes in low-precision matrix math like INT8, doing the same background blur at 5-10W versus a GPU's 30-40W, and it holds about 92% of peak on battery in measurements (an external dongle drops about 63% from heat). But big LLMs do not route through the NPU yet: as of 2026 Ollama and llama.cpp do not target NPUs and use GPU acceleration like Metal or CUDA instead. The NPU's sweet spot is Whisper variants, Phi-class small models, embeddings, and always-on agents. In short, think of it as NPU for "small AI that runs quietly all day" and GPU for "big AI you run hard for a short burst."

Where do beginners get stuck?#

Three things: thermals, RAM, and NPU expectations.

• Thermals: a benchmark is not 10 minutes, it is real use. Thin laptops throttle on sustained inference, dropping from a measured 24 to 18 t/s (plug in and use a stand for big jobs).
• RAM: above all, do not skimp on RAM - the real bottleneck for loading models is RAM and bandwidth, not TOPS.
• NPU expectations: "NPU equals big-LLM acceleration" is a misconception. It is for small, always-on models, not for a 70B.

How do you do it in 5 minutes?#

Start with the small task, respecting the laptop's limits.

• For transcription, launch Whisper base/small; for chat, a 1-4B small LLM first (both are laptop-friendly).
• Run the same task on battery and on the charger and compare the speed gap (throttling).
• Run heavy models (7B+, image gen) on the charger, and use a stand to shed heat on long runs.

Reference links

• Whisper (transcription model)
• ONNX Runtime (NPU runtime)
• Core ML Tools (Apple on-device)
• llama.cpp (GPU-accelerated inference)
• Copilot+ PC NPU developer docs

Note: TOPS, power, tok/s, and bandwidth figures are public 2026 measurements and docs and vary by chip, cooling, power, and software. Measure exact speed on your own laptop on both battery and charger with the method above. Laptop AI chips and runtimes change fast, so this is reviewed quarterly.