E2E · 25% of the score · optional
Audio in, audio out, on the clock. Not whether the parts work, but whether the whole thing holds a conversation someone would stay on.
Everyone agrees that under 300ms feels instant, and nobody knows what that is like. Play a row: the caller finishes, the wait happens at full speed, then the system starts. Same sentence every time — only the silence changes.
Scored on the 95th percentile, not the median: a system that is usually quick and occasionally takes 2 seconds is experienced as a slow one.
From the last frame of the caller’s audio leaving our socket to the first byte of the reply arriving. Our monotonic clock only — participant timestamps are never trusted. Reported with the measured network round trip alongside, so hosting location is visible rather than silently rewarded.
Human evaluators rate a whole exchange. Did the turn-taking work. Would you trust this with a claim.
Every scenario ends on a turn where the caller talks over the reply. We measure how long the system keeps speaking after that, and count it as yielding if it stops within 500ms. Nothing in the protocol asks it to stop — a request would measure compliance with our wire format rather than whether it behaves like a person on a phone.
A sixty-second clip allows 30000ms from end of utterance to first response byte. That deadline is also the integrity control: it is far too short for a person to transcribe the audio by hand.