Running AI Agents Locally: VRAM and RAM Requirements, Measured

When you run an AI agent locally, the VRAM killer is not the model but "the context the loop accumulates." An agent is not one question and answer but a repeated decide, run a tool, observe, and each turn appends the tool result to the context and feeds the whole thing back to the model. Three measured essentials: (1) a 2K starting prompt balloons to 20K+ tokens on a 10-step task (tool definitions alone are 1-3K). (2) That context piles into the KV cache, and on long loops the KV cache grows larger than the model weights. (3) So budget for the "whole loop," not the first reply - otherwise the agent OOMs mid-task. It is not "the model is 8GB, so it fits" but model + accumulated context + concurrency together.

In plain terms: agent memory is a snowball. Each step (tool call) sticks on more observation like snow, and before it reaches the bottom (task done) the ball fills the room (VRAM). Pick the room by the first step's size and you jam midway.

Why is agent memory different?#

Because the loop keeps re-feeding the context. A normal chat is one in, one out, but an agent re-runs the system prompt + tool list + every prior step each turn. So context grows and the KV cache keeps eating VRAM. Production agents send 50K-500K input tokens per request (only hundreds of output tokens). The sizing question is not "what model fits?" but "what model + context window + retrieval + how many running at once fit?"

How much does context accumulate?#

Linearly with token count, and frightfully on long loops. A 70B's KV is about 14GB at 32K and 43GB at 128K, encroaching on a large share of its fp16 140GB weights. Even small models add 2-6GB at default context and far more at long context - a 128K conversation on a 30B adds 8-12GB of RAM beyond the model. When tool results are big documents or logs, several GB attach at once. So "context engineering" - trimming tool output to only what is needed before it enters context - matters as much as raw VRAM.

Why does it crash mid-task?#

Because the KV cache grows uncontrolled. Locally the signs are clear - after a long conversation generation slows sharply (KV pushed into swap), OOM appears during CPU offload, and Ollama or llama.cpp dies mid-generation to the kernel OOM killer. Watch the prefill spike too: the attention matrix grows quadratically with sequence length, so even if the KV fits, that moment can OOM (you want FlashAttention). That is, the infrastructure collapses exactly when the agent reads a long document or enters a long reasoning loop.

How do you budget and shrink it?#

The key is budgeting for the whole loop and pressing the KV down to avoid OOM.

• Budget: add 15-20% headroom over weights-plus-context to pick a GPU tier (a tight fit crawls via swap).
• Compress: FP8 KV cache (e.g. OLLAMA_KV_CACHE_TYPE=q8_0) halves it, cap context with num_ctx or max-model-len, and reuse the system prompt with prefix caching.
• Discipline: longer context is not always better - very long contexts suffer "lost in the middle," so summarize and prune. Measure exact values on your own task loop.

Related reading: 에이전트 브라우저 제어, 직접 써보고 느낀 점과 한계, ComfyUI로 이미지·영상 만들기: 우리가 직접 굴리며 잰 운영 회고

Related reading: 로컬 오픈 LLM VRAM·RAM 요구량, 직접 계산·실측, 로컬 음성합성(TTS) 오픈모델 — VRAM·RAM 요구량 실측

Reference links

• vLLM (PagedAttention, prefix caching)
• Ollama (KV cache type, num_ctx)
• llama.cpp (KV quantization, context)
• PagedAttention paper (vLLM)
• FlashAttention (mitigates the prefill spike)

Note: GB and token figures are public 2026 measurements and estimates and vary by model, context, concurrency, and attention backend (not permanent numbers). KV depends heavily on tool output and document size, so measure exact memory on your own agent loop (these numbers are only a start). Longer context is not always a gain (lost in the middle). The agent and serving stack moves fast, so this is reviewed quarterly.