Nomic Embed for Local RAG: Judge It by p50/p95 Latency
In short: Nomic Embed is a compact, open-source text-embedding model that turns documents and queries into fixed-length vectors (768 dimensions, estimated spec) so a local RAG pipeline can retrieve passages by meaning rather than exact keywords. The beginner-honest way to judge it is not parameter counts or leaderboard scores — it is the latency you actually feel, split into p50 (typical) and
Nomic Embed is a compact, open-source text-embedding model that turns documents and queries into fixed-length vectors (768 dimensions, estimated spec) so a local RAG pipeline can retrieve passages by meaning rather than exact keywords. The beginner-honest way to judge it is not parameter counts or leaderboard scores — it is the latency you actually feel, split into p50 (typical) and p95 (tail) response time.
first_response_latency_ms 120.8 ms
What did Hax measure on its own stack?#
Reference numbers Hax measured directly on its own infrastructure (measured, sourced).
| Dataset item | Measured value | Date | Source |
|---|---|---|---|
| first_response_latency_ms | 120.8 ms | 2026-07-04 | bench_harness.probe_unified_latency |
| HTTP 응답 P95 지연(7일) | 538 ms | 2026-07-23 | Hax 운영 실측(telemetry/funnel) |
| 생성 처리량 | 38.8 tok/s | 2026-07-04 | bench_harness.probe_llm_bench (unified-api 실측, 3회 중앙값) |
- 표본
- 3 measured metrics (Hax /data curated)
- 측정 환경
- bench_harness.probe_llm_bench (unified-api 실측; Hax ai-server(prod uvicorn :5502 ×4 워커
- 수집일
- 2026-07-04 ~ 2026-07-23
- 방법
- bench_harness.probe_unified_latency; 3회 중앙값); SQLite); telemetry 5165요청 백분위
How can you reproduce these numbers?#
Follow the source column above and our open dataset at /data.
| Metric | Value | Label |
|---|---|---|
| Hax unified first-response latency | 119.2 ms | measured 2026-07-03 (bench_harness) |
| Hax HTTP P95 (7-day) | 538 ms | measured 2026-07-23 (telemetry) |
| Nomic Embed single-query embed, CPU | ~40-90 ms | estimated |
| Nomic Embed recall@5, small local corpus | ~0.6-0.7 | estimated |
| Quantized model on disk (GGUF) | ~80-140 MB | estimated |
Hax unified first-response latency 119.2 ms
Those two Hax numbers are our own serving stack, offered as a calibration band — the 'Hax Local-AI Latency Index'[/glossary#hax-latency-index] — not Nomic Embed's own embed time; keep the two separate when you compare.
What is Nomic Embed, and why judge it by latency?#
Embedding quality (recall@5) and speed pull against each other, but for a first local build the tail latency is what breaks the experience. A model that averages fast but stalls one query in twenty (p95) still feels broken. So measure both percentiles from day one.
The 5-minute local RAG quickstart#
- Install a local runner (Ollama or llama.cpp).
- Pull the model:
ollama pull nomic-embed-text. - Chunk your documents into ~200-400 token pieces (estimated), then embed each chunk.
- Store the vectors in a lightweight index — FAISS, Chroma, or sqlite-vec.
- At query time, embed the question, take the top-5 nearest by cosine similarity, and pass them to your generation model.
That is the whole loop. Nomic Embed's Matryoshka training also lets you truncate the 768-dim vector to a smaller size (estimated), trading a little recall for less disk and faster search.
How to read p50 and p95#
p50 is the median request; p95 is the slow tail that one in twenty requests hits. On a laptop, p95 is what makes the UI feel janky, so time it directly: wrap your embed call, log per-request milliseconds, and sort. If your embed-plus-retrieve step alone pushes past a few hundred milliseconds at p95, the first levers are batching queries, truncating the Matryoshka dimension, or quantizing the model. Use the Hax measured band above (119.2 ms first-response, 538 ms 7-day P95) as a sanity anchor for what a tuned serving path looks like — your single-machine numbers will differ, but the shape of the target is the same.
Recall@5 and disk footprint#
recall@5 asks whether the right passage is in your top five results; for a small personal corpus, an estimated 0.6-0.7 is a reasonable starting expectation before you tune chunking. The quantized model itself is small (estimated ~80-140 MB on disk), which is why Nomic Embed is a common first pick for beginners running everything on one machine.
Note: Nomic Embed figures here are estimated and version-dependent; the Hax latency values are measured on our serving stack (2026-07) and are not the model's standalone embed time. Re-measure on your own hardware before quoting. Internal evidence: Hax data.
Related reading: Nomic Embed 로컬 RAG, 클라우드 비용을 줄일까, 로컬 RAG 청킹 전략 — 문서 쪼개기가 검색 품질을 좌우한다
Full guide: 노트북에서 AI 모델 뭐가 돌아갈까 — VRAM·RAM 실측과 메모리 구조
Responses
No responses yet. Be the first to respond.