Nomic Embed Local RAG: Judging Failure by p50/p95 Latency

Nomic Embed local RAG is a retrieval-augmented generation pipeline that pairs the open-weight Nomic Embed text-embedding model with a local vector store, so a model grounds answers in your own documents without any cloud embedding API. When this pipeline fails, it rarely crashes outright; it drifts, and you catch the drift through p50/p95 latency and recall@5 rather than by feel.

What did Hax measure on its own stack?#

Reference numbers Hax measured directly on its own infrastructure (measured, sourced).

How can you reproduce these numbers?#

Follow the source column above and our open dataset at /data.

The single number to watch first is p95, not the average. An average latency near 120 ms (measured 2026-07-03/04, probe_unified_latency) can coexist with a 7-day HTTP P95 of 625 ms (measured 2026-07-24), because the tail is where cold reloads, index scans, and reranking collide. If you only report the mean, every failure mode below hides inside a healthy-looking p50.

Failure mode 1: prefix mismatch. Nomic Embed expects task prefixes such as search_query and search_document. Mixing them, or dropping them, silently lowers recall@5 without raising latency at all. This is the most common and least visible failure because latency dashboards stay green.

Failure mode 2: Matryoshka truncation. Nomic Embed supports shrinking the embedding dimension to cut disk and speed search. Truncate too aggressively and recall@5 falls; the fix is to treat dimension as a measured trade-off, not a default, and to re-benchmark recall each time you change it.

Failure mode 3: unbounded store growth. This is measurable in our own curator: active memory count rose from 8,919 (measured 2026-07-04) to 10,585 (measured 2026-07-24), while average confidence fell from 0.721 (measured 2026-07-04) to 0.612 (measured 2026-07-24). More vectors is not better retrieval; a growing, unpruned index enlarges the p95 tail and dilutes mean relevance. We track this tail behavior as the Hax Local-AI Latency Index[/glossary#hax-latency-index].

Failure mode 4: cold-start reloads. If the embedder unloads between requests, the first query after idle pays a reload penalty that shows only in p95, never p50. Pin the model in memory and measure warm vs cold separately.

Fixes, in order: label every prefix, freeze the embedding dimension after a recall@5 benchmark, prune low-confidence chunks on a schedule, keep the model warm, and always publish p50 and p95 side by side. Judge the system by the tail you can reproduce, not the average you can hope for.

Note: measured values are dated 2026-07-03 through 2026-07-24; recall@5 and disk footprint for our stack are not yet measured (측정대기) and any target figures are estimated. Re-benchmark after any embedding-dimension or index change. See Hax data.

Related reading: Nomic Embed 로컬 RAG, p95 지연으로 판단하는 구매 체크리스트, BGE-M3 다국어 검색 초보 5분 설치 퀵스타트와 실패 지점

Full guide: 노트북에서 AI 모델 뭐가 돌아갈까 — VRAM·RAM 실측과 메모리 구조

References#

• Hax public measured dataset