Local RAG Document Q&A: Speed and Quality, Measured (Embedding, Vector DB, Chunking)
In short: For local RAG document Q&A, speed and quality are decided not by the LLM but by the embedding model, the vector database, and chunking. Compare them directly and the self-hosted quality default is BGE-M3 (568M, dense + sparse hybrid, MTEB about 63), while on a CPU laptop nomic-embed-text (137M, 274MB, MTEB 62.4) is the fit.
For local RAG document Q&A, speed and quality are decided not by the LLM but by the embedding model, the vector database, and chunking. Compare them directly and the self-hosted quality default is BGE-M3 (568M, dense + sparse hybrid, MTEB about 63), while on a CPU laptop nomic-embed-text (137M, 274MB, MTEB 62.4) is the fit. For storage, use FAISS for fast search or Chroma for metadata filtering and persistence.
In plain terms: RAG is a library with a librarian. The embedding shelves books by topic (retrieval quality), the vector DB fetches them fast (speed), and chunking decides how finely the books are cut.
First, tracing how data flows from a question to an answer makes the role of each part clear.
Which embedding model should you pick?#
It matters most because it is the foundation of retrieval quality. BGE-M3 does dense (semantic) and sparse (keyword) search from one model, giving hybrid retrieval without a separate keyword index, covers 100+ languages, and is the de facto 2026 self-hosted default (MIT). nomic-embed-text runs on a laptop CPU with no GPU at 137M / 274MB and an 8,192-token context, and is the most downloaded on Ollama. For ultra-light edge, all-MiniLM-L6-v2 is the fastest.
| Model | Size | MTEB | Context | Strength / license |
|---|---|---|---|---|
| BGE-M3 | 568M | ~63.0 | 8192 | Dense + sparse hybrid, 100+ languages, MIT |
| nomic-embed-text | 137M | 62.4 | 8192 | Laptop CPU (274MB), most pulled on Ollama, Apache |
| all-MiniLM-L6-v2 | 22M | ~56 | 512 | Ultra-light edge, fastest, MIT |
FAISS or Chroma for the vector DB?#
They serve different jobs. FAISS is a fast search library; Chroma is an application database. FAISS searches billions of vectors in milliseconds with IVF/HNSW and even GPU acceleration, but has no DB features like persistence or metadata. Chroma is embedded-first: across 500K chunks it filters by metadata like department or date and returns results in about 20ms, and your data survives a restart. But it slows at scale; one practitioner found it 50-100x slower when iterating on experiments. Use Chroma for fast prototyping and metadata filters, FAISS (or Qdrant) for large scale and lowest latency.
Why is chunking the hidden variable?#
Because quality collapses well before the advertised context limit. In one benchmark the top models are perfect within context, but even BGE-M3 slips to 0.920 at 8K, and lightweight models (mxbai, nomic) drop to 0.4-0.6 retrieval accuracy at just 4K characters (about 1,000 tokens). A 512-token model truncates most of a document outright. So the lighter the model, the smaller you must cut the chunks.
How do you measure it yourself?#
Measure on your own data; an MTEB score does not represent your documents (and MTEB v2 is not directly comparable to v1).
- Use 20-50 of your own questions to compare each model's top-k answer recall and embedding time.
- Benchmark vector DBs on the same data with a tool like VectorDBBench for fair latency and throughput.
- Sweep chunk size (256, 512, 1024 tokens) to find where recall breaks down.
Note: MTEB and latency figures are public 2025-2026 measurements and vary with data, version, and hardware (MTEB may not match your domain). Measure your own documents with the method above. Models and tools change often, so this is reviewed quarterly.
Responses
No responses yet. Be the first to respond.