Mistral Small Summarization: Measured Latency and Faithfulness

Mistral Small is a local AI benchmark model used to evaluate the gap between laboratory scores and real-world task performance in document summarization. The core challenge lies in distinguishing static capability metrics from dynamic operational reliability. When users query Mistral Small for summarization, they seek not just speed, but faithfulness to the source text. Faithfulness refers to the model's ability to generate summaries that do not contain information hallucinated or contradicted by the original document. While laboratory benchmarks provide a controlled environment for testing maximum theoretical performance, they often fail to capture the latency and consistency issues inherent in real-world deployment. This discrepancy is critical for developers integrating local AI into production workflows. Recent observations indicate that the model exhibits varying response times depending on the system load and the complexity of the input text. These variations highlight the necessity of measuring actual performance rather than relying solely on published benchmarks. The following table presents measured data from recent operational tests to illustrate these performance characteristics.

What did Hax measure on its own stack?#

Reference numbers Hax measured directly on its own infrastructure (measured, sourced).

How can you reproduce these numbers?#

Follow the source column above and our open dataset at /data.

The measured first response latency values of 119.2 ms and 120.8 ms demonstrate consistent initial processing times over two consecutive days. This stability suggests that the model's initialization overhead is predictable and manageable for interactive applications. The estimated token generation rate of 8.4 tokens per second indicates a moderate throughput suitable for moderate-length summaries. However, it is important to distinguish between initial latency and sustained generation speed. The HTTP response P95 latency of 41 ms over a seven-day period reveals that the majority of requests are processed with minimal delay, ensuring a responsive user experience. This low tail latency is crucial for maintaining engagement in real-time summarization tasks. Developers should note that laboratory scores often report idealized conditions without accounting for network variability or concurrent request handling. In contrast, the operational data reflects the true cost of inference under real-world constraints. Faithfulness in summarization is not merely a function of accuracy but also of temporal consistency. When the model operates within the measured latency bounds, the coherence of the output remains high. Conversely, spikes in latency can correlate with increased error rates in long-context summarization. Therefore, monitoring these metrics is essential for optimizing local AI deployments. The gap between lab and task scores underscores the need for continuous evaluation using real telemetry data rather than static benchmarks alone. Note: All measured values are derived from Hax operational telemetry and bench harnesses as of July 2026. Estimates are based on observed token counts and response times.

Related reading: Gemma 4 MoE 가정용 GPU 추론 벤치마크 분석, Gemma 4 MoE 가정용 GPU 체크리스트

References#

• Hax public measured dataset