Multilingual voice agents: benchmark task completion

TL;DR: Evaluate multilingual voice agents on verified task completion, critical-field correctness, unsafe actions, and recovery behavior for each language cohort. Word error rate is useful recognition evidence, but it cannot establish that the agent preserved the right name, amount, date, permission, or final business outcome.

A transcript can contain many harmless wording differences and still preserve the requested task. It can also look almost perfect while changing a single negation, account digit, or appointment date that determines whether the final action is correct.

That is why a multilingual voice-agent benchmark needs more than one recognition score. It needs consented test audio, explicit task expectations, controlled business resources, language-aware annotation, and an outcome check against the system that owns the action.

The scope matters. FLEURS introduced speech evaluation across 102 languages, with about 12 hours of speech supervision per language, while the Whisper paper describes training with 680,000 hours of multilingual and multitask supervision; neither number is a result for your agent's appointment or support workflow.

This article specifies a reproducible evaluation protocol and explains the limits of available aggregate evidence. It complements the call regression harness and outcome metric definitions; no multilingual task-completion experiment was run for this article.

Published: November 9, 2026. Last updated: November 9, 2026. Reported product recognition results are attributed to their public report and kept separate from the proposed agent evaluation.

Table of contents

1. Define success before collecting speech
2. Keep recognition and task metrics separate
3. Build a consented multilingual corpus
4. Treat names, numbers, and dates as typed fields
5. Evaluate language switches and repair
6. Run controlled end-to-end trials
7. Score outcomes with language-aware review
8. Report cohorts and uncertainty honestly
9. Interpret public 60db evidence at its boundary
10. Frequently asked questions
11. Publish a protocol before a leaderboard

Define success before collecting speech

A multilingual task succeeds when the authorized business result matches the caller's intended, confirmed request. The language of the conversation is a condition of the test, while the owning service's resulting state provides the strongest evidence that the action happened correctly.

Choose a bounded workflow first. An appointment change, structured information request, or support escalation can have a clear expected result, whereas a broad instruction to "have a good conversation" leaves the benchmark dependent on subjective impressions.

For an appointment change, success can require the correct synthetic customer, original appointment, requested slot, timezone, and confirmed operation receipt. A friendly exchange that changes the wrong appointment should fail even if the transcript is readable and the caller sentiment appears positive.

Separate task success from safe fallback. An agent that recognizes uncertainty and requests a human may avoid harm, but that is a different outcome from completing the task without assistance.

Likewise, distinguish an unavailable business option from a language failure. If the requested slot does not exist and the agent explains that correctly, the appropriate outcome may be correct refusal or successful alternative selection rather than failed recognition.

Write these rules before running any model. Otherwise the team can reinterpret ambiguous cases after seeing favorable or unfavorable results, which weakens the comparison.

Define critical errors explicitly. A wrong amount, unsupported identity assumption, omitted negation, or unauthorized action can warrant a blocking failure even when the overall completion rate looks acceptable.

Your policy owner should define actions that require confirmation or human review. The workflow enforcement guide describes how to enforce that contract outside the prompt, which keeps benchmark success from depending only on conversational compliance.

Create an expected-state record for each scenario. Include permitted outcomes, forbidden writes, allowed clarification paths, and whether handoff counts as success, safe fallback, or unresolved task under the published protocol.

Use synthetic resources in the business system. A test appointment or account should be isolated from production data, and every trial needs a reset rule so previous runs cannot make later trials easier or harder.

Keep task semantics comparable across languages without assuming literal translation is equivalent. A scheduling question can require different natural phrasing, politeness, or date expression while still targeting the same canonical business state.

When translation changes ambiguity, record it. A benchmark that makes one language's prompts unusually formal and another language's prompts conversational may measure prompt construction as much as system behavior.

The smallest useful benchmark covers one task and a few well-defined language cohorts. Expand after the labels, resets, and outcome checks are reliable rather than collecting a large audio archive that cannot be scored consistently.

Keep recognition and task metrics separate

Word error rate measures edit differences between a reference transcript and recognition output under a chosen tokenization and normalization policy. Its usual form is substitutions plus deletions plus insertions divided by the number of reference words.

Those edit counts give every reference token the same unit weight. A missing filler word and a missing negation can each contribute one deletion even though their consequences for the task are very different.

NIST's Speech Recognition Scoring Toolkit is an established source for speech-recognition scoring tools. Whatever implementation you choose, pin its version and publish the text preparation rules so a reported WER can be reproduced rather than inferred from a dashboard label.

WER is not always directly comparable across languages. Token boundaries, writing systems, spelling conventions, and the handling of code-switched words can change the denominator and the edit alignment.

Character error rate can add useful recognition context for some writing systems. It still does not tell you whether the final amount, date, or permission was correct, so it should remain a diagnostic measure alongside task-level scoring.

Use critical-field accuracy for the semantic values that drive the workflow. Check the canonical amount, currency, date, resource identity, or destination rather than requiring an arbitrary transcript spelling when several spellings represent the same permitted value.

Also measure repair. An agent may initially recognize a digit incorrectly, ask for confirmation, and correct the field before execution; the final task can succeed while the initial recognition still contributes diagnostic error.

Keep first-pass accuracy and post-repair accuracy separate. That distinction shows whether the system understands in ordinary speech or depends on repeated clarification that may exhaust callers.

The table compares evidence types rather than products. No single metric proves the entire interaction, and a checkmark applies only to the property explicitly observed.

Measure latency too. Correct responses that repeatedly arrive after the caller begins speaking again can produce failures that a transcript-only evaluation misses, so use the speech-to-playback timing protocol for each language cohort.

Do not average stage timings into a fictional end-to-end value. Endpointing, recognition, reasoning, tool work, synthesis, transport, and playback need an actual shared measurement boundary before their effects can be summarized as caller response delay.

Recognition, critical fields, and business receipts answer different questions.

Build a consented multilingual corpus

A useful corpus reflects the language conditions your workflow intends to support. It should include natural speech, appropriate speakers, clear permission for the planned processing, and enough metadata to interpret results without exposing identities.

Recruit speakers under a documented consent process. Explain what will be recorded, which systems will process it, who can review it, how long it will be retained, and how permitted withdrawal or deletion will be handled.

Synthetic speech is useful for deterministic regressions, but it is not a complete substitute for human speech. It can omit hesitation, accent variation, self-correction, timing, microphone behavior, and the conversational habits that challenge a live system.

Public datasets can help test recognition. The Google FLEURS dataset card documents its multilingual data and license; inspect its task, splits, and terms before reuse, and do not assume read speech represents telephone dialogue.

Keep public-corpus recognition trials separate from end-to-end task trials. A pre-recorded sentence without an interactive caller cannot demonstrate that the agent asked an effective clarification or recovered after a language switch.

Define cohorts by meaningful conditions. Language, speaker variety, channel, noise, code-switch pattern, task type, and critical-field class can all matter, but collecting every possible combination at once makes sample sizes too thin to interpret.

Choose the cohorts from product scope and observed risk. If Hindi-English switching and spoken dates are expected, prioritize those cells before adding languages that the public product contract does not support.

Balance scenario difficulty across cohorts. Equivalent tasks should contain comparable ambiguity and required actions, while language-specific expression remains natural rather than mechanically translated.

Keep speaker splits separate across development and final evaluation. Reusing the same voices and phrasings while tuning can overstate performance on unseen callers even when the test text changes slightly.

Version the corpus and labels. Changing a reference transcript, accepted spelling, or expected date should create a new documented dataset revision rather than silently changing historical scores.

Store minimal speaker metadata. Use consented, purpose-relevant cohort labels and opaque references, and avoid publishing combinations that can identify a speaker through a rare accent, locale, or scenario detail.

The data boundary guide helps map the recording path. Audio collection, transcription, human annotation, model judging, and artifact storage can each create a separate disclosure and retention boundary.

Reject unusable recordings under a written rule before inspecting product outcomes. Keep the exclusion counts and reasons, because quietly dropping difficult clips after a failed run biases the result.

Treat names, numbers, and dates as typed fields

Critical fields need semantic labels that preserve their business meaning. The benchmark should ask whether the right value reached the authorized operation, not whether the transcript resembles one preferred spelling.

Names are a difficult example. W3C's guidance on personal names describes variation in name structure and ordering, which warns against assuming that every caller has the same given-name and family-name pattern.

Use synthetic names in published fixtures. In a real authorized test, preserve the accepted display form and matching policy without converting every name into a Western form or stripping meaningful characters for convenience.

Unicode normalization can reduce representation differences. Unicode Standard Annex 15 defines normalization forms, but normalization does not translate, transliterate, infer identity, or establish that two similar-looking names belong to the same person.

Avoid overly permissive matching. Removing every mark or collapsing all transliterations can hide a wrong entity, especially when two test records differ only in a task-critical detail.

Numbers need units. An amount without currency, a time without timezone context, and a measurement without its unit are incomplete semantic values even if every digit was recognized correctly.

Represent financial amounts using the owning system's canonical form, such as integer minor units plus currency, rather than comparing floating-point strings generated from speech. The exact representation should follow the application's existing contract.

Dates need a declared reference time and locale assumptions. "Next Friday" cannot be scored reproducibly if the benchmark does not fix the current date, timezone, and interpretation rule used in that scenario.

Unicode CLDR's date specification documents locale-sensitive date and time representation. Formatting support does not resolve conversational ambiguity, so the agent should ask a clarification when the task cannot be safely grounded.

Phone numbers, account references, and postal codes should preserve meaningful leading zeros and country context. Treating every digit sequence as an integer can destroy the value before the model or scorer even sees it.

Negation deserves a dedicated label. "Do not cancel" and "cancel" can differ by one short word while demanding opposite actions, so the expected-state record should include forbidden effects as well as the desired positive result.

Corrections also need explicit semantics. A caller who says an initial digit sequence and then replaces one digit should produce the corrected canonical value, while the transcript can retain both versions for diagnostic review.

The following runnable fixture demonstrates exact semantic matching for a synthetic task receipt. Save it as outcome_check.py and run python3 outcome_check.py; these fields are application examples, not a provider API.

from dataclasses import dataclass

@dataclass(frozen=True)
class Expected:
    resource: str
    date: str
    amount_minor: int
    currency: str

@dataclass(frozen=True)
class Receipt:
    resource: str
    date: str
    amount_minor: int
    currency: str
    committed: bool

def task_succeeded(expected, receipt):
    return receipt.committed and (
        expected.resource,
        expected.date,
        expected.amount_minor,
        expected.currency,
    ) == (
        receipt.resource,
        receipt.date,
        receipt.amount_minor,
        receipt.currency,
    )

expected = Expected("synthetic-a", "2030-06-14", 12500, "INR")
assert task_succeeded(expected, Receipt("synthetic-a", "2030-06-14", 12500, "INR", True))
assert not task_succeeded(expected, Receipt("synthetic-a", "2030-06-14", 1250, "INR", True))
assert not task_succeeded(expected, Receipt("synthetic-a", "2030-06-14", 12500, "INR", False))
assert not task_succeeded(expected, Receipt("synthetic-b", "2030-06-14", 12500, "INR", True))
print("semantic outcome checks passed")

This check deliberately does not score transcript similarity. Production scoring should also check authorization, prohibited writes, and scenario-specific alternatives, while keeping the expected values independent of the model being evaluated.

Evaluate language switches and repair

Language switching is a conversational behavior, not merely a list of enabled language codes. A caller can use one language for the request, another for a product name, and a different script convention for a number or address.

Define switch types in the corpus. Between-turn switching, within-turn switching, borrowed words, quoted names, and a request to change the reply language create different recognition and response obligations.

Keep the desired output language explicit where it matters. A caller who asks in a mixed utterance may prefer a consistent response language, and the benchmark should not infer preference only from whichever language has more tokens.

Evaluate preservation of critical fields across a switch. The agent should not translate a proper name into a different entity, change a currency assumption, or lose a correction because it appeared after an English phrase in an otherwise Hindi turn.

Repair behavior is often more informative than first-pass success. Test whether the agent asks a specific clarification, reads back the uncertain value, and updates the canonical proposal before requesting confirmation.

Do not reward repeated generic apologies. A useful repair reduces uncertainty, while an agent that keeps asking the caller to repeat the entire sentence can sound polite without making progress.

Include controlled recognition errors in deterministic tests. Inject an incorrect interim digit, then a corrected final value, and assert that the final business action uses the corrected value rather than the first available hypothesis.

For audio trials, preserve real timing. The VAD versus turn detection guide explains how pauses, short acknowledgments, and resumed speech can affect endpointing; multilingual phrasing should not be forced through a single untested silence assumption.

Test interruptions during readback. A caller can reject the wrong date before the agent finishes speaking, and the interruption safety model should prevent the old proposal or stale audio from continuing as if the correction never happened.

Also test explicit uncertainty. An agent that cannot establish a critical value should take the allowed clarification or handoff path instead of inventing a plausible spelling or number to complete the task.

The warm handoff protocol helps evaluate that fallback. Score whether the human received the language preference, unresolved field, and uncertainty status, rather than counting any transfer attempt as successful multilingual support.

Keep code-switch scoring human-readable. A reviewer should be able to see which segment changed language, what semantic value was expected, and whether the final operation preserved it, without reverse-engineering a single aggregate score.

Critical errors need their own fixtures even when the transcript looks readable.

Run controlled end-to-end trials

End-to-end trials should use the same business scenario and observation contract across language cohorts. Otherwise a faster model, easier prompt, different carrier route, or warmer connection can be mistaken for a language effect.

Pin the agent configuration, model versions where available, prompt revision, voice, allowed languages, tool schema, and backend behavior. Record fields that a managed service does not expose rather than guessing a private version.

Use the same authorized transport conditions for matched comparisons. A browser call, narrowband telephone call, and custom stream can differ in audio processing and timing, so report them as separate cohorts unless the comparison intentionally studies transport.

The transport guide explains those ownership boundaries. Do not claim the managed browser route uses a particular protocol without a documented contract, and do not treat standalone speech streaming as the full agent stream interface.

Control business state between trials. Reset synthetic appointments, clear only test-owned operation records, and verify the reset before the next run so a duplicate-action protection rule does not skew later results.

Randomize trial order within a declared plan. Running every language sequentially can confound language with provider load, cache state, deployment changes, or network conditions across the test window.

Repeat stochastic interactions. A single pass says little about reliability, while repeated trials with the same scenario and controlled variations reveal whether success depends on an unusually favorable model response.

Record every attempted trial. Failed connection, unsupported language configuration, caller simulation failure, timeout, and unavailable backend evidence need explicit labels rather than disappearing from the denominator.

Use a scripted caller for deterministic branching where possible, but preserve a consented human evaluation layer for natural repair. A caller script that instantly supplies perfectly normalized answers can conceal the very multilingual ambiguity you intended to test.

Define who stops the trial. The shutdown semantics guide covers cleanup of media, tasks, and post-call jobs; a benchmark must release its own resources even when the agent fails or the harness times out.

Capture a trace with minimal content. The observability guide connects turn, tool, and business-operation identities, which lets you explain a failure without copying raw customer-like speech into every report.

Keep outcome checks independent of the agent's narration. Query the synthetic business system for the receipt, verify critical fields, and assert the absence of prohibited writes before calling the trial successful.

flowchart TD
    A[Consented scenario corpus] -->|Version inputs| B[Controlled call trial]
    B -->|Capture stage evidence| C[Recognition and turn trace]
    B -->|Read owning service| D[Business receipt]
    C -->|Language-aware review| E[Diagnostic labels]
    D -->|Match critical fields| F[Task outcome]
    E -->|Join by trial identity| G[Cohort report]
    F -->|Join by trial identity| G
    G -->|Publish limits and counts| H[Reproducible evaluation]

Score outcomes with language-aware review

Use deterministic checks where the task has canonical values. A committed receipt with the wrong amount should fail without requiring a model judge, while open-ended explanation quality may need a human rubric.

Reviewers need competence in the evaluated language and relevant code-switch pattern. A translation-only reviewer can miss politeness, negation, register, pronunciation, or ambiguity that the original speech makes apparent.

Define annotation categories with examples from synthetic fixtures. Useful categories include recognition error, field normalization error, wrong-language response, premature turn ending, ineffective clarification, unauthorized action, backend failure, and successful recovery.

Allow multiple contributing labels. A wrong date can begin with recognition, survive a weak readback, and finally reach an insufficient backend gate; forcing one root-cause label can erase the chain that developers need to repair.

Separate observation from interpretation in annotations. "The receipt contains the wrong date" is observable; "the model ignored the correction" requires evidence about the correction event and the decision path.

Use independent review on a meaningful sample, including both successes and failures. Review only flagged failures and you can miss false successes, while review only a convenient language and you can overstate coverage for the rest.

Resolve disagreements through a documented adjudication rule. Keep the initial labels so agreement can be inspected, and do not silently change a difficult reference until all systems appear correct.

Automated judges can assist with bounded semantic review after calibration. Compare them with language-qualified human labels, preserve an uncertain outcome, and inspect performance separately by language and task-critical field.

Do not use the same model's self-report as the only success label. It may confidently explain that it completed an operation whose receipt is missing or whose arguments are wrong.

Calibrate acceptance thresholds against consequences. A benchmark for informational assistance can tolerate different ambiguity than one that changes appointments or financial records, so do not copy a single confidence cutoff across every action.

Keep the scorecard compact enough to diagnose. Task success, critical-field errors, unsafe actions, safe fallback, clarification burden, and response timing usually tell a more useful story than an opaque blended quality score.

The outcome metrics guide provides the broader reporting model. Its central discipline applies across languages: a positive conversational label does not establish that the caller's intended business result happened.

Report cohorts and uncertainty honestly

A multilingual report should expose where evidence is strong, weak, and absent. Show sample counts, exclusions, failures, and uncertainty for each meaningful cohort before presenting an overall summary.

A micro-average weights trials or tokens according to the chosen denominator. A macro-average gives each cohort equal weight, which can reveal poor performance in a smaller language group that a large dominant cohort would otherwise conceal.

Neither average is universally correct. Publish the aggregation rule and the cohort values so readers can see whether the result fits their deployment mix.

Do not average percentiles across languages. A combined latency percentile must be computed from the pooled observations under the declared weighting, while per-language percentiles remain useful for diagnosing distinct user experiences.

Separate unsupported from failed. A language outside the configured product scope is not evidence of a supported-language regression, but it is still a product-fit limit that a buyer or developer needs to know.

Show missing instrumentation explicitly. If the final business system cannot expose a receipt, report the outcome as unverified rather than converting a natural-sounding closing statement into success.

Include confidence intervals or a suitable uncertainty analysis when estimating rates. Repeated trials from the same speaker or scenario are correlated, so an analysis that treats every turn as an independent caller can overstate certainty.

For a small pilot, simple counts plus transparent limitations may be more honest than a precise-looking statistical claim. Choose an analysis that matches the sampling design instead of using a formula because it produces a narrow interval.

Disclose tuning exposure. If the team adjusted prompts using the evaluation set, call it a development result and reserve a separate held-out set for the final comparison.

Publish the failure taxonomy and a few synthetic reconstructions. Readers can learn from an incorrect date correction or an ineffective language switch without seeing identifiable voices, names, or raw transcripts.

Keep a machine-readable manifest for reproducibility. It should contain scenario identities, corpus version, configuration revision, trial status, expected outcome reference, and aggregate derivation rules without embedding secret keys or protected audio links.

A benchmark publication should also state what was not tested. Regions, accents, noise types, task classes, call routes, and language combinations outside the corpus remain unknown, even when the headline number is favorable.

Compare changes with paired scenarios

When evaluating a new prompt or model configuration, preserve the scenario pairing. Run the old and new configurations against equivalent corpus items and compare which specific tasks changed outcome, rather than attributing a difference between unrelated trial sets to the configuration alone.

Pairing does not eliminate randomness. Repeat the planned trials, record execution order, and retain the configuration identity so a temporary service incident or unusual caller response can be investigated before a release decision.

Inspect asymmetric changes. A new configuration can improve ordinary information requests while introducing a wrong-date failure in a smaller code-switch cohort, and a single net completion score can conceal that trade-off.

Keep prohibited actions outside a compensating average. If the release contract forbids unauthorized cancellation, several additional successful routine calls do not erase a new unauthorized cancellation in the test set.

Use a cohort change table with counts of improved, regressed, unchanged, and unverified outcomes. Include the critical-field classes involved, then inspect representative synthetic reconstructions to decide whether the difference belongs to recognition, turn handling, policy, or backend execution.

A change in the scorer requires its own comparison. Rescore a fixed set with both scoring versions, document disagreements, and avoid presenting a scoring-policy change as a product improvement.

The same applies to normalization. If a new text rule stops counting a transliteration difference as an error, the recognition metric can improve without any change to what the caller experienced or what the agent did.

Archive the aggregate derivation with the corpus manifest. A reader should be able to reproduce the published counts from the permitted trial records, while protected audio and speaker identity remain outside the public artifact.

Release decisions should name the evidence gap that remains. An improvement on one supported language pair is a bounded result, and the next test should target an unmeasured condition or a known failure mechanism rather than expanding the headline beyond the data.

Report unresolved and excluded trials alongside completed and safe-fallback outcomes.

Interpret public 60db evidence at its boundary

Public recognition data can help choose diagnostic cohorts, but it cannot replace an agent-level benchmark. The result needs to stay attached to the endpoint, corpus, aggregation rule, and date that produced it.

60db's Hindi ASR report, generated May 31, 2026, reports primary five-subset micro-average WER of 12.95% for batch and 15.65% for streaming. The requests used the hi,en language hint. That primary result excludes FLEURS; the full six-subset corpus contains 9,997 clips and 122,747 reference words, with all-six WER of 12.96% and 15.49%, respectively.

The primary five-subset clip count is 9,580, calculated by subtracting the 417-clip FLEURS subset from the full corpus. The report excludes that subset from the headline comparison because historical vendor columns contain data-quality artifacts, and it did not rerun all competitors under a matched current protocol.

These are reported Hindi recognition results, not multilingual 60db Agent task completion or independent measurements run for this article. Model revision, hardware, and the separate primary reference-word denominator are not fully disclosed, so the figures should not acquire more precision through retelling.

The current 60db Agent language contract distinguishes listen_languages from speak_languages and describes Hindi-English code-switching. Treat enabled languages as configuration scope and qualify actual workspace availability; language support does not establish equal measured performance across languages or tasks.

An evaluation can use the public recognition report as motivation to inspect cohort sensitivity. It still needs interactive trials, confirmed critical fields, backend receipts, and language-qualified review before publishing a voice-agent completion claim.

Keep synthesis-stage evidence separate too. A TTS onset measurement cannot establish that a caller understood a multilingual readback, just as a transcript score cannot establish that the final business operation was correct.

The latency benchmark guide explains how to measure the complete caller experience. Apply that boundary alongside the semantic outcome checks rather than replacing either with a convenient stage-level headline.

Frequently asked questions

What should a multilingual voice-agent benchmark measure first?

Start with verified task completion and prohibited actions for a bounded workflow. Add critical-field correctness, repair behavior, and latency so the report can explain why a task succeeded or failed.

Why is word error rate insufficient?

WER counts transcript edit differences under a scoring policy. It does not distinguish a harmless wording change from a missing negation or wrong digit that changes the authorized business action.

Can an agent succeed with an imperfect transcript?

Yes. It can preserve the relevant meaning or clarify an uncertain field before acting, so score initial recognition and final task outcome separately rather than requiring every transcript token to match.

Can a nearly perfect transcript still produce a failed task?

Yes. A single critical value can be wrong, or the action can fail after recognition because of authorization, stale state, tool execution, or missing confirmation.

Does support for many languages imply equal quality?

No. A supported-language list defines configuration scope, while quality requires measurements for the relevant language, task, speaker, and channel conditions.

Should I use public speech datasets?

They are useful for recognition diagnostics when their license, task, and splits fit the evaluation. They do not automatically represent interactive telephone dialogue or demonstrate successful business-task completion.

Is synthetic speech enough for the full benchmark?

No. It supports reproducible regressions, but natural hesitation, accents, self-correction, channel effects, and conversational repair require consented human speech or appropriately qualified evidence.

How should names be scored?

Use the workflow's accepted semantic identity and display-form rules. Avoid assumptions about universal name order, and do not normalize so aggressively that different entities collapse into the same value.

How should spoken money be represented?

Compare canonical amount and currency fields using the owning application's contract. Integer minor units are often suitable, while floating-point text or a number without currency can hide a consequential mismatch.

What makes relative dates reproducible?

Fix the scenario's reference date, timezone, locale assumptions, and accepted interpretation. If the utterance remains ambiguous, the expected behavior should require clarification rather than guessing.

How should code-switching be tested?

Include within-turn and between-turn switches, borrowed terms, proper names, corrections, and explicit reply-language preferences. Check that critical fields and task intent survive the switch, not just that both languages appear in the transcript.

Should clarification count as failure?

Not automatically. Record clarification burden and final outcome separately, because a targeted clarification can prevent harm while repeated ineffective questions can still make the experience unusable.

Can a human handoff count as success?

Only under the benchmark's published outcome definition. Report safe fallback separately from autonomous completion, and verify that the human received usable context and accepted the task.

How many samples does each language need?

That depends on the desired uncertainty, failure rarity, and sampling design. Publish counts and limitations, and avoid precise reliability claims from a small or heavily repeated set of speakers and scenarios.

Should I report micro- or macro-averages?

Report the aggregation rule and per-cohort results, using both views when helpful. Micro-averages reflect the chosen population weighting, while macro-averages can expose weak performance in smaller language cohorts.

Can language latency percentiles be averaged?

No. Compute a combined percentile from the appropriate pooled observations, or report each language's percentile separately with its sample count and measurement boundary.

Can an automated judge replace bilingual review?

It can assist after calibration against language-qualified labels, but it should not be the only evidence for critical business outcomes. Preserve uncertainty and inspect judge errors by language and task type.

What do the public Hindi ASR numbers prove?

They report recognition performance under the corpus and endpoint conditions described in that report. They do not establish multilingual agent task completion, live-call reliability, or a current controlled comparison against every competitor.

How should unsupported languages appear in a report?

Label them as outside the tested or supported scope rather than silently excluding them from product-fit discussion. Do not mix unsupported configurations with supported-language regression rates without explaining the denominator.

What should a first 60db Agent evaluation look like?

Use documented, workspace-enabled language settings, synthetic business resources, consented speakers, and the same task expectations across cohorts. Verify final receipts and publish observed limits before making any completion or superiority claim.

Publish a protocol before a leaderboard

A useful multilingual benchmark connects speech to the action it was meant to support. Recognition quality, critical-field preservation, confirmation, authorization, repair, and the final business receipt each explain a different part of that path.

Start with one bounded task, a consented corpus, and explicit expected outcomes. Keep natural language and code-switching realistic, isolate synthetic business resources, and score outcomes independently of the agent's own narration.

Report language cohorts, failures, exclusions, and uncertainty before an overall average. Use public recognition results as scoped evidence, and keep unmeasured task completion or unavailable language coverage visible rather than filling the gaps with confidence.

For 60db Agent, begin with a documented language configuration your workspace can use and a small controlled trial set. Publish the protocol and the observed failure mechanisms first; a defensible leaderboard can follow only after comparable measurements exist.