Our LLM serving: throughput vs latency cross-check

Our LLM serving: throughput vs latency cross-check reports operational numbers measured directly on our ai-server (Hax) stack, and — instead of dumping figures — explains what each number actually means for a real decision. Even if you are new to running local AI, this single post should let you grasp 'what do I decide when I see this number' in about five minutes.

What these numbers mean#

200토큰 전체 생성 지연 5153ms를 처리량으로 환산하면 38.8 tok/s로, 실측 처리량 38.8 tok/s와 정확히 일치한다(교차검증 통과). 이는 큐 대기나 네트워크 지연 오염 없이 순수 생성 속도가 그대로 나온다는 뜻이라, 이 수치를 용량 산정의 기준선으로 신뢰할 수 있다.

How we measured it (reproducible conditions)#

These are not vendor specs or marketing figures; they are values we measured ourselves under the conditions below. We list the conditions because when the conditions change, the numbers change too — cold start versus warmed up, batch size, and the exact hardware all shift the result for the same model. So we state reproducible conditions (measured 2026-07-04):

• bench_harness.probe_llm_bench (unified-api 실측, 3회 중앙값)

Rather than memorizing a single number, understand it together with these conditions, so you can diagnose for yourself why your own environment produces a different value.

How to use this in practice#

The derived judgment above translates straight into an operating decision. The point is not to memorize raw figures but to read the relationships between them — a ratio of two values, a utilization rate, a cross-check — because those relationships tell you what to scale up and what to conserve. We use this to check existing headroom before buying new hardware, and to split workflows into a fast path and a quality path. The same logic applies directly to your own local AI setup.

Why this beats vendor specs#

These are values measured in our own operating environment, not vendor sheets or someone else's benchmark. Every number above is measured (not estimated), with date and source (Hax /data). Unlike generic AI-written prose, this derived judgment cannot be produced without the measurement. Only our own measured values are used; no private tokens or internal paths are exposed.

Note: the values above are our own stack measurements as of 2026-07-04 and are refreshed when conditions change (measured values only, no estimates).

Related reading: 오픈웨이트 vs 클로즈드 LLM, 직접 본 속도·품질·비용, Mistral Small 문서 요약: 지연 실측과 데이터 잔류 정책

References#

• Hax data