Language hints versus autodetection for Hindi-English calls

TL;DR: Compare language hints and autodetection on the same reviewed tasks, preserving the exact input policy and recording critical fields separately from transcript quality. Language selection, recognition of a switch inside an utterance, output script, and verified task completion require different evidence.

---

A Hindi-English scheduling call produces a readable transcript, yet the booked time differs from the caller's corrected request. Changing the language hint may change the transcript without fixing the booking. You need a comparison that can identify which boundary changed.

Start with a paired task ledger. Give every input recording a stable identity, run each declared policy against the same recording, and preserve the raw output alongside reviewed fields and an independent outcome receipt. The multilingual benchmark method supplies the wider task boundary; this guide makes the language-policy comparison reproducible.

As of November 2026, Microsoft's language-identification documentation distinguishes at-start and continuous identification and explicitly excludes within-sentence language changes from continuous identification. That documented limit is enough to rule out treating a returned language label as proof that every switched word was recognized.

The standard-library artifact below validates synthetic paired records. It does not call an ASR service, process recordings, or measure Hindi-English recognition accuracy for 60db Agent. Its first useful result is a trustworthy answer to a narrower question: which comparisons have enough compatible evidence to inspect?

Table of contents

1. Define the task before the language policy
2. Separate hints, candidates, and output language
3. Build a paired corpus without reference leakage
4. Score fields and outcomes independently
5. Validate an immutable comparison ledger
6. Diagnose differences before choosing a policy
7. Report what the comparison can support
8. Frequently asked questions
9. Run one defensible comparison

Define the task before the language policy

A useful comparison begins with the action a caller wants to complete. For scheduling, that might be selecting an available interval and confirming it before a write. For identifier capture, it might be preserving an exact sequence and asking for clarification when a symbol remains uncertain.

These tasks have different failure costs. A transcript spelling difference can be harmless when a reviewed alias maps unambiguously to the intended synthetic entity, while a changed digit can invalidate the entire task. Define those acceptance rules before looking at either policy's output.

A task record should describe the initial request, the information available to the agent, permitted clarification, and the evidence required for completion. Keep the business operation separate from the recognizer's output. The outcome-metrics boundary explains why a successful recognition request cannot substitute for an action receipt.

Code switching also needs an operational definition. A recording can contain utterances in different languages, a switch inside one utterance, an English product token inside another language, or a name whose spelling does not reveal its pronunciation. Annotate the event you intend to test instead of calling all mixed content the same phenomenon.

Do not infer a person's language identity from these labels. A task annotation says what a reviewer marked in this recording under a declared rubric. It does not establish nationality, proficiency, first language, or how every future caller will speak.

For an early pilot, choose one task family with a small set of material fields. Write down what makes the task unsuccessful, what should trigger clarification, and what remains unknown after a timeout. This gives you a useful diagnostic even before there are enough observations to compare policies statistically.

The early acceptance checklist can be applied manually. Confirm that both policies saw the same input artifact, used the same external task context, and were scored against the same reference revision. A changed contact list or availability snapshot would create a different task even if the audio bytes were identical.

Include repair behavior in the task definition. A caller can replace a date or reject a proposed interpretation after the recognizer has produced a fluent partial transcript. Treat partial transcripts as revisions, and score the final approved intent rather than whichever intermediate text happened to look complete.

The figure's shared input is the controlled unit. Its join represents comparison eligibility, not agreement or success. Both branches can produce eligible records while differing on every critical field.

An evaluation unit also needs a stopping rule. Decide whether one clarification is permitted, whether a caller correction opens a new intent revision, and when the agent should abandon an uncertain action. Otherwise two policies can receive different opportunities to recover and still be reported as if they completed identical tasks.

Keep a small task-state map with the reference. For example, a synthetic slot request can begin unresolved, become proposed, require confirmation, and finish with either a matched receipt or an explicit unresolved disposition. A fluent response is an observation along that path; it cannot jump over the required confirmation state.

Annotate background context that changes the interpretation. A relative date depends on a reference instant and calendar. A name depends on the scoped registry available at that point in the task. Those values must remain consistent across paired trials, even when they are stored outside the recording manifest.

This is also where you decide whether the experiment is recognition-only or conversational. A recognition-only replay can compare transcription and extracted fields from fixed audio. A conversational experiment introduces generated prompts, caller repair behavior, and timing dependencies, so the later turns may no longer be identical inputs across policies.

For that reason, do not describe a branching dialog as a strict audio-paired experiment after the branches diverge. You can pair the initial scenario and compare end-to-end outcomes under a separate protocol. Preserve the narrower fixed-input comparison as its own result so readers can tell which question each observation answers.

Separate hints, candidates, and output language

A language hint is a configuration value with provider-specific semantics. It may constrain recognition, influence decoding, select among candidates, or apply to a different part of the speech pipeline. Record the documented meaning for the exact endpoint and model you invoke.

Google Cloud's automatic language-detection guide describes recognition using supplied language codes and documents model and location restrictions. A candidate list is therefore part of the tested policy, not an invisible convenience that can change between trials.

A fixed-language request and a request containing candidate languages are different experiments. A continuous language label and a transcript containing words from multiple languages are also different outputs. Decide which property your application needs before turning either output into a pass condition.

The public Whisper language-tagging discussion illustrates a real developer question about Hindi-English input and language-selection behavior. The existence of that question supports investigating the boundary; comments in a discussion do not establish the current behavior of another provider or model version.

A separate ADK issue about speech language configuration asks whether an output speech setting constrains input transcription. Keep those configurations separate in the manifest. A response spoken in the requested language says nothing by itself about the policy applied to incoming audio.

Use explicit names such as input_policy, candidate_languages, output_language, and transcript_script_policy. Avoid a single ambiguous language field that adapters reinterpret. BCP 47 language tags provide a standard tagging vocabulary; tags do not guarantee recognition quality or make a script policy unnecessary.

An input policy should also specify what it does when a supported configuration is unavailable. Do not silently fall back from candidates to a fixed language and keep the original policy ID. Record the fallback as another policy revision or mark the attempt unsupported.

The following table describes evidence requirements, not vendor capability rankings. A check means the evidence can support the stated conclusion when the configuration is documented; a cross means that evidence alone is insufficient.

A language label can still be useful for diagnostics or routing. The mistake is asking it to prove content it never described. Preserve it beside the transcript and configuration rather than replacing the transcript's evidence with a label.

This distinction matters when a spoken name appears in an unexpected script. Evaluate whether the application can identify and confirm the intended entity separately from whether its display follows the chosen script convention. The Tamil-English alias method develops that separation without equating transliteration and identity.

A configuration capture should include both the request you intended to send and the configuration the adapter actually submitted. Retain a sanitized endpoint identity, request schema version, candidate list, relevant feature flags, and any documented server response about acceptance. A friendly wrapper name is not enough to reproduce the policy.

Record unsupported combinations before launching large collections. Try one authorized request that exercises the exact endpoint, region, model, audio format, and candidate-language combination. This is an interface check rather than an accuracy test, and it prevents an entire corpus from being run under an unnoticed fallback.

Use a separate policy identity when candidate order, context prompts, or transcript normalization changes. Even when documentation does not say those details affect results, preserving them avoids an unexplained hidden variable. You can later establish which values are immaterial through a controlled test rather than discarding them by assumption.

Do not silently replace a language code with a region variant because the original was rejected. The replacement might be appropriate, but it is a configuration decision that belongs in the manifest. A rejected intended policy and an accepted replacement policy should remain distinguishable in coverage reporting.

The same discipline applies to script postprocessing. If an adapter rewrites provider output into another script, keep the original output and the transform identity. Score the raw recognition layer and the transformed display layer under their own rules so an improvement in display consistency does not conceal a change in recognized content.

Build a paired corpus without reference leakage

The pairing unit should be an immutable recording or another explicitly declared input artifact. Assign its digest when the artifact enters the evaluation, preserve the capture format, and make every derived version a new artifact. Resampling, clipping, and channel selection are transformations worth recording.

Document sample rate, channel count, codec, decoder, and the point where audio is captured. A language-policy comparison cannot explain a difference caused by feeding one branch telephone audio and another a wideband recording. The phone-audio evaluation boundary is relevant when the deployed path changes the available speech signal.

Use consented recordings and reviewed task labels for real performance work. The examples in this article are synthetic ledger entries. They demonstrate record behavior and do not establish that any Hindi-English utterance was understood.

A reference should preserve material values and acceptable alternatives. For a date, distinguish the caller's intended calendar and reference context from an arbitrary normalized string. For names, keep the approved entity set and clarification rules rather than requiring one guessed romanization.

Freeze the reference revision before scoring. If a reviewer later finds a mistaken label, preserve the correction and rerun affected comparisons under the new revision. Do not overwrite the reference while retaining a report that pretends all rows used the original ground truth.

Hide reference answers from inference. An experiment can leak its expected answer through filenames, task metadata, retrieval context, candidate contact lists, or a hint generated from the correct transcript. That would make the apparent language-policy difference partly a difference in supplied answers.

A practical separation is to keep the execution bundle and scoring bundle in different records. The execution bundle contains authorized application context. The scoring bundle adds reviewed expected values after recognition has finished. Access controls and trace review must enforce the separation; a Python tuple alone cannot prevent a service from receiving leaked context.

Match the intended deployment context without granting one policy extra information. If the application legitimately knows the caller selected a service language, both policies can receive that same context when the experiment asks about that deployed condition. Label the condition clearly instead of generalizing it to callers who supplied no language preference.

Use data statements to describe collection context, annotation decisions, and population boundaries. Their purpose is documentation that helps others understand the data, not a certificate that a corpus covers all ways of speaking Hindi and English.

Pairing also requires stable trial order or a recorded randomization schedule. Provider changes, transient availability, and task-state changes can interact with execution order. Record order and time without interpreting a small ordering difference as evidence about a language community.

When conversational context matters, declare whether each trial begins with an empty session or a fixed preceding dialog. A policy can appear to recover a switch because an earlier message exposed the intended field. The voice settings isolation contract helps prevent another call's settings from becoming hidden trial context.

Score fields and outcomes independently

Transcript scoring answers a text-comparison question under its own normalization rules. The public NIST Speech Recognition Scoring Toolkit supplies established tooling for that layer. A word-error calculation does not determine whether the correct entity was confirmed or the requested appointment exists.

Define a critical-field scorer beside transcript scoring. It should compare reviewed material values under a versioned equivalence policy. Preserve both the raw recognized form and the normalized form so a future reviewer can recompute the result and challenge the equivalence rule.

Keep missing distinct from incorrect. A timeout with no recognized field is different from a completed result containing the wrong value. If a transcript exists but the downstream action has no receipt, transcript correctness may be known while task completion remains unknown.

Confirmation adds another boundary. A recognizer can extract the intended field, then the agent can read it back ambiguously or proceed after a qualified response. Use distinguishable spoken confirmations and treat a caller's correction as a new revision.

A phrase such as "yes, but later" cannot be scored as approval of the original interval merely because it starts with an affirmative word. The qualified-confirmation repair contract provides a separate check that belongs in a task evaluation, regardless of which language policy produced the text.

For the final operation, collect a receipt from the system that owns the action. Match operation identity, approved values, and the relevant revision. A successful request submission can be retained as transport evidence while the business outcome remains unresolved.

Before calculating an aggregate, choose its denominator. Eligible attempts, completed recognitions, paired complete tasks, confirmed tasks, and receipt-backed outcomes are different sets. Present their counts explicitly so an apparent improvement cannot be caused by excluding one policy's difficult cases.

The reference validator below reports the number of input fixtures, complete pairs, incomplete fixtures, and unsupported attempts. It deliberately avoids a single accuracy score. A production scorer can add reviewed field and outcome measures after it defines the corresponding populations.

Do not convert unsupported into failed recognition. Unsupported tells you the candidate policy cannot be evaluated under the selected configuration. That can be decisive for deployment selection, but it answers an availability question rather than a question about recognition errors on accepted input.

For repeated trials, preserve attempt identity and the sampling plan. Choosing the best of several responses after seeing the reference changes the experiment. Retry rules must be fixed before scoring, just as regression test acceptance rules must remain stable across a release.

Validate an immutable comparison ledger

The artifact accepts mappings at its boundary and immediately copies the supported value shape into frozen dataclasses and tuples. Its trust boundary is intentionally small: fixture IDs, digests, configuration identities, reference values, and observed fields are supplied by trusted evaluation adapters.

It does not verify file hashes against recordings or certify that an adapter used its claimed configuration. Those require captured artifacts and endpoint traces. The validator ensures that the supplied comparison has consistent identities and does not silently pair different tasks.

Save the complete fence as paired_policies.py and run it with Python 3.11 or newer. The included assertions are synthetic negative and boundary fixtures. They exercise the published implementation rather than a provider client.

from dataclasses import dataclass

@dataclass(frozen=True)
class Trial:
    fixture: str
    audio: str
    reference: str
    policy: str
    config: tuple
    status: str
    expected: tuple
    observed: tuple
    transcript: str
    receipt: str

def text(value):
    if type(value) is not str or not value:
        raise ValueError("nonempty string required")
    return value

def fields(value):
    if type(value) is not dict:
        raise ValueError("plain field mapping required")
    return tuple(sorted((text(k), text(v)) for k, v in value.items()))

def configuration(row):
    if row.get("phase") != "final":
        raise ValueError("final-only ledger")
    if row.get("contamination_reviewed") is not True or row.get("reference_visible") is not False:
        raise ValueError("trusted contamination review required")
    config = row["config"]
    mode = text(config["mode"])
    raw_languages = config["languages"]
    if type(raw_languages) not in (list, tuple):
        raise ValueError("languages must be a list or tuple")
    languages = tuple(text(v) for v in raw_languages)
    hint = config["hint"]
    if mode not in {"fixed", "candidates"} or not languages:
        raise ValueError("declared mode and languages required")
    if len(set(languages)) != len(languages) or (mode == "fixed" and len(languages) != 1):
        raise ValueError("invalid language configuration")
    if type(hint) is not str:
        raise ValueError("raw hint text required")
    return (mode, languages, hint, text(row["review_revision"]))

def admit(row):
    status = text(row["status"])
    if status not in {"complete", "timeout", "unsupported"}:
        raise ValueError("unknown status")
    if type(row["transcript"]) is not str:
        raise ValueError("transcript must be raw text")
    receipt = text(row["receipt"])
    if receipt not in {"matched", "mismatched", "unknown"}:
        raise ValueError("unknown receipt state")
    return Trial(*(text(row[k]) for k in
        ("fixture", "audio", "reference", "policy")),
        configuration(row), status, fields(row["expected"]), fields(row["observed"]),
        row["transcript"], receipt)

def summarize(rows, policies):
    policies = tuple(text(p) for p in policies)
    if not policies or len(set(policies)) != len(policies):
        raise ValueError("unique policy identities required")
    admitted = tuple(admit(row) for row in rows)
    by_key, fixtures, configs = {}, {}, {}
    for trial in admitted:
        if trial.policy not in policies:
            raise ValueError("undeclared policy")
        # The final item is per-trial audit evidence, not a recognition setting.
        policy_settings = trial.config[:-1]
        previous_config = configs.setdefault(trial.policy, policy_settings)
        if previous_config != policy_settings:
            raise ValueError("policy configuration changed")
        key = (trial.fixture, trial.policy)
        if key in by_key and by_key[key] != trial:
            raise ValueError("conflicting trial identity")
        by_key[key] = trial  # exact retry is historical evidence
        identity = (trial.audio, trial.reference, trial.expected)
        if fixtures.setdefault(trial.fixture, identity) != identity:
            raise ValueError("fixture or reference changed")
    paired = tuple(f for f in sorted(fixtures) if all(
        (f, p) in by_key and by_key[f, p].status == "complete"
        for p in policies))
    return {
        "state": "empty" if not fixtures else "recorded",
        "fixtures": len(fixtures),
        "complete_pairs": len(paired),
        "incomplete_fixtures": len(fixtures) - len(paired),
        "unsupported_attempts": sum(t.status == "unsupported"
                                    for t in by_key.values()),
        "paired_ids": paired,
        "history": tuple(by_key[k] for k in sorted(by_key)),
    }

def synthetic(policy="fixed", **changes):
    row = dict(fixture="synthetic-task", audio="synthetic-digest",
        reference="ref-A", policy=policy,
        config={"mode": policy, "languages": ["hi-IN"] if policy == "fixed"
                else ["hi-IN", "en-IN"], "hint": ""},
        phase="final", contamination_reviewed=True, reference_visible=False,
        review_revision="synthetic-reviewed-A",
        status="complete", expected={"slot": "slot-A"},
        observed={"slot": "slot-A"}, transcript="synthetic text",
        receipt="unknown")
    row.update(changes)
    return row

def rejects(rows, policies=("fixed", "candidates")):
    try:
        summarize(rows, policies)
    except ValueError:
        return
    raise AssertionError("expected rejection")

a, b = synthetic(), synthetic("candidates")
raw = [a, b]
result = summarize(raw, ["fixed", "candidates"])
assert result["complete_pairs"] == 1
assert result["history"][0].receipt == "unknown"
a["expected"]["slot"] = "mutated"
raw.clear()
assert result["history"][0].expected == (("slot", "slot-A"),)
assert summarize([], ["fixed"])["state"] == "empty"
assert summarize([synthetic()], ["fixed", "candidates"])["incomplete_fixtures"] == 1
r = summarize([synthetic(), synthetic("candidates", status="unsupported")],
              ["fixed", "candidates"])
assert r["complete_pairs"] == 0 and r["unsupported_attempts"] == 1
r = summarize([synthetic(), synthetic("candidates", status="timeout")],
              ["fixed", "candidates"])
assert r["incomplete_fixtures"] == 1
assert len(summarize([synthetic(), synthetic()], ["fixed"])["history"]) == 1
rejects([synthetic(), synthetic(transcript="changed")])
rejects([synthetic(), synthetic("candidates", audio="different")])
rejects([synthetic(), synthetic("candidates", reference="ref-B")])
rejects([synthetic(), synthetic("candidates", expected={"slot": "other"})])
rejects([synthetic(), synthetic(fixture="second", config={
    "mode": "fixed", "languages": ["en-IN"], "hint": ""})])
rejects([synthetic(phase="partial")])
rejects([synthetic(contamination_reviewed=False)])
rejects([synthetic(reference_visible=True)])
rejects([synthetic("candidates", config={
    "mode": "candidates", "languages": "hi-IN", "hint": ""})])
owned = synthetic()
snapshot = summarize([owned], ["fixed"])["history"][0]
owned["config"]["languages"].append("en-IN")
owned["config"]["hint"] = "changed"
assert snapshot.config[:3] == ("fixed", ("hi-IN",), "")
reviewed = summarize([
    synthetic(fixture="first", review_revision="item-review-A"),
    synthetic(fixture="second", review_revision="item-review-B")], ["fixed"])
assert reviewed["complete_pairs"] == 2
assert {t.config[-1] for t in reviewed["history"]} == {"item-review-A", "item-review-B"}
rejects([synthetic(), synthetic(review_revision="changed-item-review")])
rejects([synthetic(status="guessed")])
rejects([], ["fixed", "fixed"])
rejects([synthetic(expected={"slot": []})])
print("Synthetic ledger fixtures passed; no ASR measurements performed.")

The frozen Trial owns its nested values because every supported leaf is a string and every field collection becomes a tuple. The assertions mutate the original mapping and clear the original list after admission. Neither operation changes the preserved history.

Exact repeated records are deduplicated by fixture and policy. A changed transcript, configuration, reference, digest, or expected field under that identity is rejected. Separate experimental attempts need separate fixture identities or an extended explicit attempt key; the example does not silently select among repeated outputs.

The admitted configuration retains the actual mode, ordered language values, raw hint text and contamination-review revision. Only the recognition settings must stay constant across fixtures sharing a policy identity; each item's review revision remains separate audit evidence in its full trial. The ledger accepts final records only; partial transcripts belong to a separate execution trace. Changed final evidence, including review evidence, under the same trial identity is a conflict.

A trusted adapter must bind these values to the actual endpoint request and capture its model and version in the execution bundle. The explicit contamination review is a required attestation, not an automatic semantic detector: a dishonest or mistaken reviewer can still admit an answer-bearing hint. Inspect the raw hint against the scoring reference before granting the attestation, and retain that review evidence.

No temporal expiry is implemented because this is a static comparison validator. If an execution controller waits on live requests, it must separately recheck cancellation, generation, configuration revision, and age after awaiting a result. Do not add an expiry rule to the historical ledger that deletes evidence merely because live authorization ended.

Diagnose differences before choosing a policy

When a pair disagrees, inspect the earliest boundary where evidence differs. If the accepted configuration differs from the manifest, fix the adapter before interpreting the transcript. If the recording digests differ, repair the pairing before computing any policy comparison.

If the transcript differs but material fields agree, review the transcript equivalence rules. A display-script change can alter textual scoring while leaving the intended operation unchanged. Preserve that difference in the report rather than forcing one metric to represent both properties.

If fields differ, inspect the raw transcript and the field extractor independently. A correct name string can be mapped to the wrong record by retrieval or alias handling. The speech retrieval contract helps separate recognition evidence from downstream selection.

Repairs deserve their own trace. Record which material value was superseded, which revision the caller approved, and which revision reached the tool. Spoken self-corrections can expose a task-state bug that changing the language policy would leave untouched.

If a field was recognized correctly but the caller did not understand its confirmation, inspect output delivery. A missing audio tail or unclear boundary can change the interaction. The streaming boundary checklist gives you a way to investigate that path without attributing it to recognition.

If one policy is often unsupported, report which declared endpoint conditions prevent comparison. Do not remove those records and then describe the remaining paired subset as the whole deployment workload. Separate the question "Can this policy run here?" from "How did accepted paired trials behave?"

If outcome receipts remain unknown, reconcile them before calling a task successful. Retry against the same operation identity where the owning system supports idempotency. The workflow enforcement method keeps this reconciliation attached to authorized state instead of a persuasive spoken completion claim.

The result may be that the language policy is not yet the main uncertainty. That is a useful finding. It directs the next engineering action toward reference review, recording integrity, repair handling, or missing receipts rather than another ungrounded model comparison.

Review disagreements with a frozen packet before changing the scorer. Include the raw recording reference, original transcript, expected field, observed field, policy configuration and task receipt. Ask whether the error belongs to recognition, annotation, equivalence rules, extraction, confirmation, or action execution; allow an unresolved classification when the evidence is insufficient.

A second reviewer should be able to reach the same classification without knowing which policy the team prefers. Blind policy labels where practical, and keep the adjudication revision. This reduces one obvious route for expectations to shape the interpretation, though blinding itself does not prove that every source of review bias has been removed.

When the reference changes, retain the previous score and the correction reason in controlled history. Recompute every affected policy under the new reference. Selectively updating the losing policy would corrupt the comparison just as surely as selectively dropping failed requests.

Write a failure note around the smallest reproducible boundary. For example, a synthetic fixture can show that both transcripts preserve the same slot while the action adapter submits an older revision. That is evidence of a revision-handling defect, and it should be routed to the workflow owner instead of being counted as a language-policy error.

Avoid explaining errors through broad claims about how a language or group usually speaks. Your observations identify behavior on collected tasks under recorded conditions. If a pronunciation or construction seems relevant, ask a qualified reviewer to annotate the specific event and then collect a controlled comparison that can test the proposed explanation.

Report what the comparison can support

A defensible report states its task family, source population, collection permission, audio conditions, policy definitions, endpoint versions, reference revision, and execution dates. Include exclusions and reasons. A reader should be able to reconstruct which rows contributed to each reported count.

Datasheets for datasets provides a useful documentation model for composition, collection, intended uses, and limitations. Apply that discipline to the task corpus even when the pilot is internal and small.

Present paired comparisons alongside coverage. Describe eligible fixtures, complete pairs, unsupported attempts, unknown outcomes, and reference corrections. If repeated utterances come from the same participant, explain that dependency before treating each utterance as independent evidence.

A policy can be chosen for documented support, simpler integration, or lower operational burden before a large recognition study exists. State the basis accurately. Do not turn a practical configuration choice into a claim that one language policy is universally more accurate.

Record the operational cost boundary too. Clarification, retry, annotation review, and receipt reconciliation consume work. The cost-per-success method helps connect those costs to verified outcomes while retaining unknown results and excluding fabricated prices.

Preserve a reproduction packet with observable boundaries, including the admitted raw records, configuration mapping, task context snapshot, reference version, scoring version, and artifact digests. Use synthetic or appropriately controlled records when sharing outside the evaluation team. Removing a name from a transcript does not necessarily make the recording or task context safe to publish.

A compact next experiment changes one material uncertainty. Add reviewed within-utterance examples if that behavior matters, extend one task family if field coverage is weak, or fix missing receipts if completion remains unknown. Cohort split checks become necessary before claiming generalization across speakers or collection conditions.

Keep an explicit change log for the comparison itself. A new decoder, revised field matcher, changed candidate list, or corrected reference can each alter results without a new recognizer. The report should make it possible to distinguish a model change from a measurement change.

Before publishing a ranking, decide what uncertainty calculation fits the sampling unit and dependence structure. Repeated clips from one speaker or repeated attempts on one recording do not create the same information as independent tasks from different participants. This guide provides no confidence interval or significance result because no speech experiment was performed.

Finally, separate deployment decisions from benchmark conclusions in the release note. You might choose a configuration because it is supported in the required region and produces inspectable evidence, while still lacking enough observations to rank its recognition performance. That is a defensible engineering decision when its basis and unresolved questions are stated plainly.

Treat the first real run as an audit of the collection path. Inspect one complete fixture from authorized recording through recognition, field review, confirmation and receipt reconciliation. Check that the stored digest identifies the file that was actually submitted and that the configuration identity resolves to the request payload, rather than merely to a default in a source file.

Then deliberately remove one variant from a copied synthetic ledger and confirm that the report exposes an incomplete fixture. Change a reference revision and confirm that pairing fails. These small checks protect the bookkeeping layer before anyone spends time interpreting a chart that was built from incompatible rows.

A useful handoff includes the questions that remain unanswered. For instance, the team may have validated policy availability while still needing reviewed recordings of repairs inside a sentence. Keeping that gap visible prevents a configuration check from becoming an accidental accuracy claim in a later presentation or release note.

---

Frequently asked questions

What does a language hint actually control?

It controls whatever the exact endpoint documents for that configuration. Record the input field, accepted value, model, and endpoint revision. A hint can influence selection without proving every word was recognized, so preserve transcript and task evidence separately from the configuration receipt.

Is autodetection the same as code-switch recognition?

No. Selecting a language from candidates and recognizing switched words inside an utterance are different behaviors. Microsoft's continuous-identification documentation explicitly limits within-sentence switching. Test the behavior your task needs instead of assigning a broad capability label from a successful language-selection response.

Can output language settings constrain input transcription?

Only if the particular interface explicitly documents that relationship. Store output speech language separately from input recognition policy. The ADK discussion linked earlier shows why developers ask this question, but a historical issue cannot establish the current contract of every provider or SDK.

Should every utterance have one language label?

Use one label only when it fits the annotation question. A mixed utterance can require span-level reference information or separate labels for a material token. Preserve uncertainty and annotation rules; a convenient single label should not erase the switch the experiment is intended to test.

How should mixed scripts be represented?

Define a transcript display policy and a separate semantic-field policy. Keep raw provider output before normalization. A script change can affect text scoring without changing an approved entity, while a visually similar spelling can still denote the wrong entity or leave identity unresolved.

Who should review reference fields?

Use reviewers qualified for the language content and the task domain, with a documented disagreement process. Language fluency alone does not establish the intended booking record. Review material values, acceptable alternatives, and clarification requirements before seeing which tested policy produced an output.

Can synthetic prompts establish real-call performance?

They can validate fixture construction and exercise declared logic. They cannot establish performance for a real caller population, acoustic channel, or spontaneous conversation style. Label synthetic examples explicitly and collect appropriately consented, reviewed recordings before making recognition or dialect-performance claims.

How do I pair trials fairly?

Use the same input artifact, external task context, reference revision, and scoring rules for every policy. Record configurations and execution order. If one branch receives extra reference-derived context, you have changed the task and cannot attribute the resulting difference solely to language policy.

What happens when one policy times out?

Keep the fixture unpaired for complete-output comparison and retain its timeout record. Count it in the declared availability or operational denominator when appropriate. Do not invent a transcript, silently drop the fixture from coverage, or assume that an unknown downstream action definitely failed.

Should unsupported requests count as errors?

They should count as unsupported attempts under a stated deployment-coverage measure. They are not observations of recognition correctness on accepted requests. Separate coverage and content measures so readers can understand whether a policy failed to run or ran and produced an incorrect field.

Is word error rate enough?

Word error rate describes text differences under chosen tokenization and normalization. It does not verify a material identifier, confirmation, or booking. Keep transcript scoring alongside critical-field checks and independent outcome evidence, and publish the normalization rules that define which textual differences matter.

How should names and numbers be scored?

Use reviewed task-specific rules. Exact sequences may matter for identifiers, while names can require a scoped alias registry and clarification. Preserve the raw recognized form, expected material value, and equivalence-policy revision so a scorer cannot silently collapse ambiguous entities into one apparent match.

Does a correct transcript prove task completion?

It proves only the reviewed transcript property. The agent may choose the wrong record, mishandle a correction, fail to confirm, or submit the wrong operation. Task completion needs evidence from the action owner that matches the approved values and relevant operation revision.

How do I prevent reference leakage?

Separate inference inputs from scoring references and review the actual execution payload. Check filenames, contextual hints, retrieval results, and reference-derived candidate lists. A clean validator cannot prevent leakage that already occurred in the provider request; payload provenance must be inspected independently.

Can I compare different model versions?

Yes, if version differences are part of the declared experiment. You cannot then attribute the result only to the hint policy. Pin both versions, keep task context and references stable, and describe the comparison as a joint configuration change rather than an isolated language-policy effect.

What should an empty corpus return?

An explicit empty state with zero recorded fixtures and no claimed rate. Empty is not evidence of perfect accuracy, failed recognition, or an expired live operation. The artifact returns a distinct state so a reporting adapter can avoid dividing by zero or inventing a conclusion.

How do retries avoid duplicate evidence?

An exact retry under the same fixture and policy identity can return historical evidence without adding another observation. A changed payload under that identity is a conflict. Genuine repeated experimental attempts require their own declared attempt identities and a sampling plan fixed before inspecting results.

Which raw values belong in a reproduction packet?

Keep artifact identity, configuration mapping, reference revision, raw transcript, raw and normalized fields, attempt status, confirmation evidence, and receipt state. Include captured task context where authorized. Protect recordings and personal information; reproducibility does not automatically authorize public release of the underlying speech.

When should I add another language policy?

Add one when it addresses a documented deployment option or a material uncertainty in the task. Give it a new configuration identity and run paired fixtures. Avoid adding policies solely because an initial small sample looked favorable; that changes the comparison process after seeing the answers.

What claims can a small pilot support?

A small pilot can expose concrete failure modes, unsupported configurations, ambiguous references, and missing evidence. Report those observations with their scope. Broad accuracy rankings or population-level language claims need an appropriate sampling and analysis plan that the synthetic validator does not provide.

Run one defensible comparison

Choose one task family, freeze a reviewed reference, and record two explicit input policies. Run the complete validator on synthetic negative cases first. Then connect authorized adapters that preserve actual configurations, artifact identity, and independent receipts.

Before comparing outcomes, inspect the unpaired and unknown records. Their reasons often identify the next useful engineering fix. The useful result is a comparison you can recompute and explain, including the cases where the available evidence cannot yet support a conclusion.