Embedding Search: Which Model and Method Are Fast and Accurate?

For semantic search, speed and quality are decided as much by which retrieval method you use as by which embedding model, where a dense bi-encoder is fastest with one vector per document, ColBERT (late interaction) is more accurate at the token level but has a larger index, and a cross-encoder reranker is the most accurate but slow, so you apply it only to the top 20-200. On models, Qwen3-Embedding 8B leads at MTEB Multilingual 70.58, while its 0.6B variant is about 639MB and runs on a laptop CPU.

In one line: the embedding model sets accuracy, and the retrieval method (dense, ColBERT, reranker) sets the speed-accuracy trade-off. A two-stage flow that retrieves fast with dense search and re-sorts with a reranker almost always wins.

In plain terms: an embedding is a ruler that turns sentences into coordinates. Nearby coordinates mean similar meaning. How precisely and how fast you apply that ruler decides search quality.

How do the retrieval methods differ?#

With the same embedding, the matching method changes accuracy and cost. A dense bi-encoder reduces query and document each to one vector and finds matches fast by cosine similarity, with a small index. ColBERT keeps a vector per token so each query token "votes" (sum-of-max) for its best-matching document token, approaching cross-encoder accuracy while still indexing documents offline. A cross-encoder reranker feeds query and document together for top accuracy, but recomputes each pair and is slow. Put simply, dense "stores coordinates ahead of time," ColBERT "stores every token's coordinates," and the reranker "reads from scratch on every query."

Which model should you pick?#

Choose by footprint and quality. Qwen3-Embedding ships in 0.6B, 4B, and 8B, all Apache-2.0. The 8B reached MTEB Multilingual 70.58, ranking No.1 as of June 2025 (ahead of the proprietary Gemini-Embedding), while the 0.6B is about 639MB at 4-bit, runs on a laptop CPU, and is competitive with BGE-M3 at about 1/10 the size. gte-Qwen2-7B scores 70.24 on MTEB English (60.25 retrieval), and the 1.5B 67.16. Dimensions matter too: a Matryoshka-trained model lets one model serve multiple output sizes (for example 64-768), trading a little accuracy to shrink the index to as low as 25KB per document. In plain terms it is trained so that only the front slice of a big vector still preserves meaning, so when storage or speed is tight you just cut the dimension.

Why is a reranker almost always worth it?#

Because two stages beat one. Retrieve the top 100 with dense search, then re-sort with a cross-encoder reranker, and recall clearly improves; the Qwen3-Reranker family raised performance over the embedding alone on every task. You contain the cost by reranking only the top candidates — you recompute 100 items, not the whole corpus, so even a slow model is affordable. Latency is manageable too: one ModernBERT+ColBERTv2 pipeline reported 31.4ms query encoding plus 26.3ms reranking for 57.7ms per query, under the 100ms that interactive use needs.

How do you measure it yourself?#

Measure on your own data; MTEB is a starting point, not your domain.

• Use the same question set to measure each model's recall@k and embedding throughput (sentences per second).
• Compare dense-only vs dense+reranker vs hybrid (BM25+dense) on the same data.
• If storage is the constraint, use Matryoshka to cut dimensions and plot the accuracy-vs-index-size curve.

Reference links

• Qwen3-Embedding (repo)
• Qwen3-Embedding (paper)
• MTEB leaderboard
• ColBERT (repo)
• Matryoshka Representation Learning (paper)

Note: MTEB and latency figures are public 2025-2026 measurements and vary with data, dimension, and hardware (MTEB may not match your domain). Measure your own data with the method above. Models and leaderboards change often, so this is reviewed quarterly.