Fish Speech Quality Drop, Confirmed by Measured Benchmarks
In short: Fish Speech is an open-source neural text-to-speech (TTS) system that converts written text into natural speech and can clone a target voice from a few seconds of reference audio. To confirm a quality drop you do not trust listening alone — you measure it: track first-response latency, decode throughput, speaker-similarity scores, and a transcription accuracy rate (derived from ASR word
Fish Speech is an open-source neural text-to-speech (TTS) system that converts written text into natural speech and can clone a target voice from a few seconds of reference audio. To confirm a quality drop you do not trust listening alone — you measure it: track first-response latency, decode throughput, speaker-similarity scores, and a transcription accuracy rate (derived from ASR word error rate) across a fixed prompt set, then diff the error examples run over run.
First-response latency 119.2 ms
| Metric | Value | Label |
|---|---|---|
| First-response latency (2026-07-03) | 119.2 ms | measured |
| First-response latency (2026-07-04) | 120.8 ms | measured |
| HTTP response P95, 7-day (2026-07-13) | 265 ms | measured |
| Decode throughput (2026-07-03) | 8.4 tok/s | estimated |
| Decode throughput (2026-07-04) | 8.3 tok/s | estimated |
| Speaker similarity vs reference | ~0.80 cosine | estimated |
| ASR-based accuracy (1 - WER) | ~95% | estimated |
- 표본
- 3 measured metrics (Hax /data curated)
- 측정 환경
- bench_harness.probe_llm_bench (unified-api 실측; Hax ai-server(prod uvicorn :5502 ×4 워커
- 수집일
- 2026-07-04 ~ 2026-07-13
- 방법
- bench_harness.probe_unified_latency; 3회 중앙값); SQLite); telemetry 5261요청 백분위
The two measured first-response numbers show the method in action: 119.2 ms on 2026-07-03 and 120.8 ms on 2026-07-04 (both measured, bench_harness.probe_unified_latency). A 1.6 ms rise is inside run-to-run noise, but it is now a tracked series rather than a hunch. On the serving edge, the 7-day HTTP response P95 was 265 ms (measured 2026-07-13, Hax telemetry/funnel). We report this class of serving-latency series as the Hax Local-AI Latency Index[/glossary#hax-latency-index], so a regression is visible as a moving line, not an anecdote.
How do you judge a quality drop by numbers?#
Build a frozen evaluation set — the same sentences, the same reference speaker, every run. Then compute three signals. First, accuracy: transcribe the generated audio with a fixed ASR model, calculate word error rate, and report accuracy as 1 - WER (the ~95% row above is estimated, not yet a Hax measurement). Second, speaker similarity: embed both the reference and the synthesized clip and take cosine similarity (~0.80 estimated). Third, latency and throughput from the measured rows.
What do error examples look like?#
Degradation is legible in the failures, so log them verbatim: dropped or swallowed words, repeated syllables (stuttering decode), hallucinated phonemes that were never in the text, and flattened prosody on question marks. When accuracy slips a few points while similarity holds, the model is usually mispronouncing, not losing the voice; when similarity falls but accuracy holds, cloning drifted. Pairing each metric with concrete error clips turns a vague "sounds worse" into a reproducible ticket.
The trend to watch: a slow rise in latency plus a slow fall in accuracy is the early signature of a quality drop, long before it is audible. Wire the frozen set into CI, alert on threshold crossings, and keep the error clips attached.
Note: figures reflect 2026-07-03 to 2026-07-13 runs; accuracy and similarity rows are estimated placeholders (측정대기) until a Hax eval-set run lands. See internal evidence: Hax data.
Related reading: linktest, probe
Related reading: 로컬 LLM·RAG 실무 가이드 — 앱에 붙이기 전 7가지 관문, 로컬 RAG 청킹 전략 — 문서 쪼개기가 검색 품질을 좌우한다
Responses
No responses yet. Be the first to respond.