Methodology
The page that decides whether the numbers are worth anything. Every metric, its formula, and the exact normalization applied before scoring — each one shown beside what it produced in edition 2026-1.
| Pillar | Weight | Required |
|---|---|---|
| ASR — CER 70% + MER 30% | 20% | Baseline |
| LLM — judge 50% + extraction 50% | 25% | Baseline |
| TTS — MOS, CoMOS, SIM, loop-back, stability | 30% | Optional |
| E2E — TTFB 40% + naturalness 60% | 25% | Optional |
Core Score is ASR + LLM rescaled to 100, so every entrant has one. Full Score covers all entered pillars and is withheld until every one of them is final — ranking a complete stack against a partially-evaluated one would be meaningless.
CER = (S + D + I) / |reference|
Computed corpus-level, weighted by reference length — not as a mean of per-clip rates. The two give different numbers: a mean over clips lets one short utterance weigh as heavily as a ninety-second call. Per-clip values are stored for diagnostics and never published as the headline.
Cantonese has no orthographic word boundaries, which makes Word Error Rate meaningless, so CER is the standard for this language. Mix Error Rate tokenises Han characters individually and Latin runs as whole words, so a misheard English word costs one unit rather than four characters — without it, code-switched speech is systematically over-penalised.
Undocumented normalization is the single largest source of ASR benchmark disputes. The exact code below runs in our scorer and is published for participants to run locally, so both sides compute the same number from the same bytes.
| Rule | Canonical form |
|---|---|
| Script | Traditional. Simplified is not folded — it is different output. |
| Width | NFKC; full-width folds to half-width |
| Case | Latin lowercased |
| Punctuation | Removed, CJK and Latin |
| Whitespace | Removed entirely |
| Numerals | Chinese folded to Arabic — 三百 = 300 |
| Currency | Symbol and 蚊 / 元 folded away |
| Fillers | Retained. They are speech. |
Every number is accompanied by a 10,000-sample percentile bootstrap, resampled over clips. Entries whose 95% intervals overlap are placed in a shared rank band and reported as tied. Publishing a strict order on differences inside the noise floor is false precision, and it is the failure every serious benchmark is eventually criticised for.
Participants implement a WebSocket server; our orchestrator is the client. Protocol v1.1, audio as pcm_s16le_16k. The per-clip deadline scales with clip length — a 60-second clip allows 30000ms from end-of-utterance to first response byte, measured on our monotonic clock only.
That deadline is the real integrity control. We do not claim audio is undownloadable — it is streamed to your process and that claim would be unenforceable. What we claim is narrower and true: the response window is far too short for a human to transcribe the audio, every copy of a dataset pack is issued to a named recipient and recorded in the audit log, the clip subset rotates between windows, and final standing is verified against a held-out set nobody has seen.
We do not watermark the audio. A per-entrant signal in the waveform would either be audible enough to change what is being measured or weak enough to survive nothing, and a control that cannot be relied on should not be described as one. Attribution rests on the issue record instead, which is enforceable because it names a person.
The other pillars measure an answer. This one measures behaviour between turns, so it runs a different protocol: we hold three scripted conversations against the entrant’s endpoint and time the gaps.
Latency is the interval from our end-of-utterance to the first frame announcing that the system has begun speaking — not to a completed reply, so buffering the whole answer before announcing it cannot look fast. We score the 95th percentilerather than the median: a system that is usually quick and occasionally takes three seconds is experienced as a slow one, and a median hides exactly that. Below 300ms the score is flat, because a caller cannot tell 120ms from 280ms and rewarding that difference optimises something nobody experiences. It reaches zero at 2000ms.
Yielding the floor. Every scenario ends on a turn where the caller talks over the reply. We measure how long the system keeps speaking afterwards and count it as yielding within 500ms. A system that is fast and will not stop talking is worse than a slow one, and a benchmark that measured only latency would rank it first.
The pillar is 40% latency and 60% human-rated naturalness. Latency publishes as soon as it is measured; the pillar total is withheld until raters have been through the transcripts, so a fast system is never credited with being a good conversationalist before anyone has checked.
Each scored run draws a fresh random subset from the rotation pool, stratified across difficulty tiers so a lucky draw of short clips cannot flatter a system. The draw is seeded per window and recorded, so any run can be reconstructed exactly.
Repeated runs still accumulate exposure, and the arithmetic is not favourable: with a pool of 55 and a draw of 30, an entrant who uses every run in their quota has seen most of the pool. That is why the pool grows between editions and why final standing is confirmed against a held-out set of 12 clips that no entrant ever receives.