Local RAG Document Q&A: How Much VRAM and RAM You Need
In short: Running a local RAG (retrieval-augmented generation) document Q&A stack means budgeting memory for three parts at once — the embedding model, the vector index, and the generator LLM — and for a standard setup with an 8B generator at Q4 you add roughly 5GB of weights to the KV cache, the embedder, and overhead, so real VRAM lands around 12GB
Running a local RAG (retrieval-augmented generation) document Q&A stack means budgeting memory for three parts at once — the embedding model, the vector index, and the generator LLM — and for a standard setup with an 8B generator at Q4 you add roughly 5GB of weights to the KV cache, the embedder, and overhead, so real VRAM lands around 12GB tight, 16GB comfortable, and 24GB when the context passes 32K tokens.
In short: RAG memory is not "model size." It is weights + KV cache + embedder + overhead, and because RAG stuffs retrieved documents into the prompt, the KV cache is the real bottleneck. Choosing a context length is choosing a GPU.
What we actually measured on the Hax box today is the hardware baseline. Via nvidia-smi, free, and nproc: two NVIDIA RTX PRO 6000 Blackwell GPUs (96GB each), an AMD Ryzen Threadripper PRO 9995WX (96 cores / 192 threads), and 1.0TiB RAM (873GiB free). This session had no torch, faiss, or ollama runtime, so we could not run the pipeline itself; the per-component VRAM figures below are computed from public model cards and 2026 benchmarks and labelled as estimates.
Why does RAG use memory in three places?#
RAG uses memory in three places because the embedding model that turns a question into a vector, the vector index that finds documents, and the generator LLM that writes the answer each hold their own memory. Think of a librarian (embedder) pulling books off shelves (the index) and handing them to a writer (the LLM). The embedder and index are light; the weight sits on the generator and the KV cache it builds to remember context.
How much VRAM does the embedding model use?#
The embedding model is far lighter than the generator, and in 2026 most options fit inside 0.3 to 5GB. BGE-M3 (568M parameters, MIT) is the self-hosted default: one model doing dense, sparse, and multi-vector retrieval across 100+ languages, about 1.1GB in FP16, usually paired with BGE-reranker-v2. nomic-embed-text (~137M, 8192 tokens) is the most-pulled lightweight option in the Ollama ecosystem and runs on a laptop CPU with no GPU. Qwen3-Embedding-8B tops MTEB multilingual at 70.58 but uses about 5GB even at Q4 and runs roughly 25x slower than nomic, so reserve it for offline corpus indexing rather than live queries.
How much do the generator and KV cache need?#
The generator's weights follow a simple rule at Q4_K_M — about 0.5GB per billion parameters — plus a separate KV cache. So 8B is about 5GB, 32B about 18-20GB, and 70B about 40-46GB of weights alone (Q4_K_M loses under 2% versus FP16; below Q4, reasoning and code degrade). The real variable in RAG is the KV cache: because retrieved documents lengthen context, on Llama 3.1 8B at Q4 the KV cache reaches about 3.4x the weights at 128K context — roughly 0.5-1GB at 4K, 2-4GB at 32K, and 8-16GB at 128K. Quantizing the KV cache to q8_0 or q4_0 cuts that roughly in half, and Flash Attention or vLLM's PagedAttention reduces waste further.
| Setup | Generator weights | Embedder + reranker | KV cache (context) | Real VRAM total | GPU fit |
|---|---|---|---|---|---|
| Light demo | 8B Q4 ~5GB | nomic ~0.3GB | 4K ~0.7GB | ~7-8GB | 8-12GB |
| Standard doc QA | 8B Q4 ~5GB | bge-m3 1.1GB + reranker 1.1GB | 32K ~2-4GB | ~10-12GB | 16GB |
| Long / high quality | 32B Q4 ~19GB | bge-m3 ~1.1GB | 32K KV q8 ~2GB | ~23-25GB | 24-32GB |
Can a 12GB GPU get started?#
A 12GB GPU can experiment with 8B standard doc QA at short context, but it is tight for stable long-context serving. Weights of 5GB plus 2.2GB of embedder and reranker already exceed 7GB, and growing the context lets the KV cache eat the rest quickly. Practically: 12GB is for short context with KV quantization, 16GB is the stable baseline for 32K doc QA, and 24-32GB comfortably holds a 32B generator or long-context, multi-user serving. CPU RAM is secondary to VRAM but still matters — index loading, model loading, and document parsing all use host memory, so keep at least 16GB, or 32GB+ for large corpora.
What should you measure before adopting?#
Before adopting, measure per-component memory peaks and a context budget rather than demo accuracy. A safe order: compute weights for the generator, embedder, and reranker at your quantization; budget the KV cache separately at your target context (4K/32K/128K); run index-search-generate on ten real documents and record peak VRAM and RAM; measure time-to-first-token and throughput per context length; and re-check model and reranker licenses before commercial use.
The conclusion is not "8B is small." An 8B RAG becomes a 16GB-class operational target the moment it puts documents into context. Fix the context length first, and the GPU choice follows.
Note: figures are computed from public model cards and benchmarks as of 2026-07-01 (estimates); only the hardware baseline was measured on this box. They shift with runtime, quantization, context, and concurrency, so measure your own before committing. The local model ecosystem moves fast and we revisit this quarterly.
Related reading: 로컬 음성합성(TTS) 오픈모델 — VRAM·RAM 요구량 실측, Ollama·LM Studio·llama.cpp 실행기 VRAM·RAM 실측
Reference links
Responses
No responses yet. Be the first to respond.