Fish Speech Quality Drop, Confirmed by Measured Benchmarks

Fish Speech is an open-source neural text-to-speech (TTS) system that converts written text into natural speech and can clone a target voice from a few seconds of reference audio. To confirm a quality drop you do not trust listening alone — you measure it: track first-response latency, decode throughput, speaker-similarity scores, and a transcription accuracy rate (derived from ASR word error rate) across a fixed prompt set, then diff the error examples run over run.

First-response latency 119.2 ms

Hax bench_harness.probe_unified_latency · 2026-07-03

Hax Fish Speech latency bench — probe_unified_latency + telemetry, 2026-07 · columns: Metric, Value, Label · 출처 Hax hax.moche.ai/en/p/1263?ref=ai_answer

Hax Fish Speech latency bench — probe_unified_latency + telemetry, 2026-07 · columns: Metric, Value, Label · 출처 Hax hax.moche.ai/en/p/1263?ref=ai_answer

Metric | Value | Label
First-response latency (2026-07-03) | 119.2 ms | measured
First-response latency (2026-07-04) | 120.8 ms | measured
HTTP response P95, 7-day (2026-07-13) | 265 ms | measured
Decode throughput (2026-07-03) | 8.4 tok/s | estimated
Decode throughput (2026-07-04) | 8.3 tok/s | estimated
Speaker similarity vs reference | ~0.80 cosine | estimated
ASR-based accuracy (1 - WER) | ~95% | estimated

목차

How do you judge a quality drop by numbers?

What do error examples look like?

References

측정 방법론 · bench_harness.probe_unified_latency +2 more

표본

3 measured metrics (Hax /data curated)

측정 환경

bench_harness.probe_llm_bench (unified-api 실측; Hax ai-server(prod uvicorn :5502 ×4 워커

수집일

2026-07-04 ~ 2026-07-13

방법

bench_harness.probe_unified_latency; 3회 중앙값); SQLite); telemetry 5261요청 백분위

The two measured first-response numbers show the method in action: 119.2 ms on 2026-07-03 and 120.8 ms on 2026-07-04 (both measured, bench_harness.probe_unified_latency). A 1.6 ms rise is inside run-to-run noise, but it is now a tracked series rather than a hunch. On the serving edge, the 7-day HTTP response P95 was 265 ms (measured 2026-07-13, Hax telemetry/funnel). We report this class of serving-latency series as the Hax Local-AI Latency Index[/glossary#hax-latency-index], so a regression is visible as a moving line, not an anecdote.

How do you judge a quality drop by numbers?#

Build a frozen evaluation set — the same sentences, the same reference speaker, every run. Then compute three signals. First, accuracy: transcribe the generated audio with a fixed ASR model, calculate word error rate, and report accuracy as 1 - WER (the ~95% row above is estimated, not yet a Hax measurement). Second, speaker similarity: embed both the reference and the synthesized clip and take cosine similarity (~0.80 estimated). Third, latency and throughput from the measured rows.

What do error examples look like?#

Degradation is legible in the failures, so log them verbatim: dropped or swallowed words, repeated syllables (stuttering decode), hallucinated phonemes that were never in the text, and flattened prosody on question marks. When accuracy slips a few points while similarity holds, the model is usually mispronouncing, not losing the voice; when similarity falls but accuracy holds, cloning drifted. Pairing each metric with concrete error clips turns a vague "sounds worse" into a reproducible ticket.

The trend to watch: a slow rise in latency plus a slow fall in accuracy is the early signature of a quality drop, long before it is audible. Wire the frozen set into CI, alert on threshold crossings, and keep the error clips attached.

Note: figures reflect 2026-07-03 to 2026-07-13 runs; accuracy and similarity rows are estimated placeholders (측정대기) until a Hax eval-set run lands. See internal evidence: Hax data.

Related reading: linktest, probe

Related reading: 로컬 LLM·RAG 실무 가이드 — 앱에 붙이기 전 7가지 관문, 로컬 RAG 청킹 전략 — 문서 쪼개기가 검색 품질을 좌우한다

References#

Hax public measured dataset