Our LLM serving: throughput vs latency cross-check reports operational numbers measured directly on our ai-server (Hax) stack, and — instead of dumping figures — explains what each number actually means for a real decision. Even if you are new to running local AI, this single post should let you grasp 'what do I decide when I see this number' in about five minutes.

Hax /data measured — our ai-server ops (own stack, measured) · columns: Metric, Measured value, Date, Source · 출처 Hax hax.moche.ai/en/p/1232?ref=ai_answer

Hax /data measured — our ai-server ops (own stack, measured) · columns: Metric, Measured value, Date, Source · 출처 Hax hax.moche.ai/en/p/1232?ref=ai_answer

Metric | Measured value | Date | Source
생성 처리량 | 38.8tok/s | 2026-07-04 | bench_harness.probe_llm_bench (unified-api 실측, 3회 중앙값)
전체 생성 지연(200토큰) | 5153ms | 2026-07-04 | bench_harness.probe_llm_bench (unified-api 실측, 3회 중앙값)

목차

What these numbers mean

How we measured it (reproducible conditions)

How to use this in practice

Why this beats vendor specs

References

측정 방법론 · bench_harness.probe_llm_bench (unified-api 실측, 3회 중앙값)

표본

2 measured metrics (Hax /data curated)

측정 환경

bench_harness.probe_llm_bench (unified-api 실측

수집일

2026-07-04

방법

3회 중앙값)

What these numbers mean#

200토큰 전체 생성 지연 5153ms를 처리량으로 환산하면 38.8 tok/s로, 실측 처리량 38.8 tok/s와 정확히 일치한다(교차검증 통과). 이는 큐 대기나 네트워크 지연 오염 없이 순수 생성 속도가 그대로 나온다는 뜻이라, 이 수치를 용량 산정의 기준선으로 신뢰할 수 있다.

How we measured it (reproducible conditions)#

These are not vendor specs or marketing figures; they are values we measured ourselves under the conditions below. We list the conditions because when the conditions change, the numbers change too — cold start versus warmed up, batch size, and the exact hardware all shift the result for the same model. So we state reproducible conditions (measured 2026-07-04):

bench_harness.probe_llm_bench (unified-api 실측, 3회 중앙값)

Rather than memorizing a single number, understand it together with these conditions, so you can diagnose for yourself why your own environment produces a different value.

How to use this in practice#

The derived judgment above translates straight into an operating decision. The point is not to memorize raw figures but to read the relationships between them — a ratio of two values, a utilization rate, a cross-check — because those relationships tell you what to scale up and what to conserve. We use this to check existing headroom before buying new hardware, and to split workflows into a fast path and a quality path. The same logic applies directly to your own local AI setup.

Why this beats vendor specs#

These are values measured in our own operating environment, not vendor sheets or someone else's benchmark. Every number above is measured (not estimated), with date and source (Hax /data). Unlike generic AI-written prose, this derived judgment cannot be produced without the measurement. Only our own measured values are used; no private tokens or internal paths are exposed.

Note: the values above are our own stack measurements as of 2026-07-04 and are refreshed when conditions change (measured values only, no estimates).