Embedding Search VRAM: The Killer Is the Index, Not the Model

Embedding and semantic-search VRAM has the opposite twist from other models - the model is not the memory killer. Three measured essentials: (1) embedding models are small - bge-m3 is 568M parameters at about 2GB VRAM, and small ones are hundreds of MB, so they run even on CPU. (2) The real killer is the vector index: roughly index size ≈ N_docs × dimensions × bytes_per_value, so 10M docs × 1024 dims is about 41GB in float32. (3) So the lever is not "model quantization" but dimension reduction (Matryoshka) plus vector quantization (int8, binary) - int8 cuts 4x, binary 32x. In short, embedding memory is set not by the model but by corpus size and vector representation.

In one line: an embedding is a library's index cards. One librarian (the model) is enough, but with 10M books the card box (vector index) fills the room. Make cards smaller (fewer dimensions) or summarized (quantized) and you index far more books in the same room.

First, the terms. A vector index is the data structure that stores document embeddings (number vectors) so they can be searched; Matryoshka (MRL) trains a vector so that using only its front slice still preserves meaning; and vector quantization stores each number in fewer bits (int8, 1-bit).

Why is embedding VRAM a twist?#

Because the model is small and the index grows with the corpus. An LLM's weights eat memory, but an embedding model encodes once and you must store every result vector to search. That storage is the real cost - one vector per document, as many numbers as the dimension. So it is not "bge-m3 is 2GB, so 2GB is fine" but how many documents, at what dimension, at what precision.

How much does the vector index cost?#

The formula N_docs × dimensions × bytes_per_value is the whole story. For 10M docs × 1024 dims, float32 is 41GB, int8 10GB, binary 1.3GB (plus HNSW graph overhead). In a Sentence-Transformers measurement, a binary index + int8 rescoring used 5.2GB memory and 52GB disk, versus 200GB memory and 200GB disk for normal retrieval. So as the corpus grows, the model stays the same while only the index explodes, so decide dimension and precision up front.

How do you shrink it?#

Dimension reduction and vector quantization are orthogonal, so combine them. First, Matryoshka (MRL): bge-m3 can be truncated from 2048 to 256 dims, so 128 dims cuts storage by over half at about 2% quality loss. Second, vector quantization: int8 (scalar) is the safe default, a 4x cut at 1-3% loss (Recall drops only 1.46% at 384 dims). Binary is 32x but loses 7-11%, so reranking is mandatory (it collapses at low dims). Combine them (MRL 128 + int8) for about 78% savings, and binary + int8 rescoring reaches up to 32x at about 3% loss.

So how do you design it?#

The key is designing the index, not the model.

• Default: bge-m3 (about 2GB) is enough for the model, and make the index int8 by default (4x, 1-3% loss).
• At scale: a binary search → int8 rescoring → cross-encoder rerank three-stage pipeline is the standard for low memory and high quality.
• Savings: for a big corpus, cut dimensions with MRL first, then add quantization. Measure exact loss and size on your own corpus.

Related reading: 에이전트 브라우저 제어, 직접 써보고 느낀 점과 한계, 로컬 LLM, VRAM은 얼마나 필요할까

Related reading: 4bit·8bit 양자화 VRAM·RAM 요구량, 직접 실측, 로컬 오픈 LLM VRAM·RAM 요구량, 직접 계산·실측

Reference links#

• BGE-M3 / FlagEmbedding (multilingual embeddings)
• sentence-transformers (embeddings, quantization)
• Embedding quantization (Hugging Face blog)
• faiss (vector index, HNSW)
• Embedding storage optimization paper (arXiv)

Note: GB and loss figures are public 2026 measurements and estimates, and vary by corpus, dimension, index (HNSW/IVF), and whether reranking is used (not permanent numbers). Binary collapses at low dimensions, so use it only with reranking. Measure exact memory and accuracy on your own corpus and queries (these numbers are only a start). The embedding and vector-search stack moves fast, so this is reviewed quarterly.