Emirati Arabic-English task conversations

TL;DR: Evaluate Emirati Arabic-English conversations against a declared speaker-variety and task contract, with separate evidence for transcription, entity capture, repair and confirmation. A supported locale or language-identification result cannot prove dialect accuracy, and a deterministic text fixture cannot establish what a caller said, heard or authorized.

---

The caller names a product in English inside an Arabic sentence, then changes the requested item. The transcript looks plausible, but the application retains the earlier entity and treats an old confirmation as permission to continue.

That failure can survive an apparently successful language configuration. Recognition, matching, correction handling and business completion are different boundaries, each with a reference and an observation of its own.

Begin with a task fixture contract: declared variety, source conditions, expected entity, accepted surfaces, correction events and the confirmation revision. Preserve uncertainty whenever the available evidence cannot resolve one of those fields.

The multilingual evaluation guide establishes why a transcript score cannot substitute for task correctness. This guide supplies a small offline reference and a procedure for building a qualified real-speech trial around it.

Every executable example is synthetic text. It does not recognize speech, classify dialect, validate an Emirati pronunciation, perform a business action or describe private 60db Agent architecture.

Table of contents

1. Define the task before selecting a locale
2. Separate language metadata from speaker evidence
3. Keep transcript and entity references distinct
4. Model corrections as new unconfirmed candidates
5. Run the bounded task fixture
6. Interpret the printed evidence
7. Test within-utterance language switches
8. Choose the evaluation slice
9. Build a consented reference corpus
10. Review uncertainty without guessing
11. Connect recognition adapters carefully
12. Measure the complete task population
13. Qualify a deployment decision
14. Audit the adapter event contract
15. Bind the reference to the intended task
16. Design the first real trial around failure boundaries
17. Frequently asked questions
18. Begin with one repaired entity

Define the task before selecting a locale

A useful task description names what the caller is trying to accomplish and which value the system must preserve. Capturing an item identifier, understanding a requested change and confirming a completed reservation need different evidence, even when the same recognition service handles their audio.

Define the smallest observable outcome first. For entity capture, the result can be a candidate catalog ID and a clarification disposition. For repair, the result includes whether the earlier candidate became obsolete. For a business operation, retain the owning service's outcome separately.

Do not make a general conversation benchmark from a collection of convenient utterances. A set of read product names can test a controlled recognition boundary while leaving spontaneous corrections, interruptions and task follow-through unmeasured.

Write the non-goals beside the task. The reference below accepts already supplied text and explicit correction events. It neither extracts corrections from natural language nor decides whether a speaker is Emirati. Those are adapter and corpus-review responsibilities.

The workflow enforcement guide explains the next boundary: a plausible candidate is not permission to perform an action. Keep the task fixture useful without promoting a text match into authorization.

A product team can use the same separation when evaluating 60db Agent through public behavior. Specify the task and observed result rather than inferring private implementation choices from a successful demonstration.

The figure distinguishes references and outcomes. The article's offline program begins after any audio-to-text adapter and leaves actual task completion unmeasured.

Separate language metadata from speaker evidence

A language tag identifies language metadata within a defined tagging system. BCP 47, specified in RFC 5646, does not turn a region subtag into a measured description of every speaker or every task.

An application configured for an Arabic locale has selected a provider contract. That selection does not establish the speaker's declared variety, the proportion of English spans, or the correctness of recognized names.

The Azure Speech language-support documentation lists capabilities by feature and locale. Read the row for the actual operation and configuration being tested; do not use availability as evidence of dialect-specific accuracy.

Declared speaker variety belongs in corpus metadata with its provenance. If a reviewer cannot establish it under the collection protocol, retain an unknown or explicitly mixed label. Do not infer it from nationality, caller location, account language or an Arabic-script transcript.

The Egyptian Arabic and MSA evaluation guide develops the same reporting discipline for another scope. Separate labels are useful only when their assignment rules and task populations remain visible.

Keep requested recognition language, returned language candidate and reviewed speaker metadata in separate fields. If they disagree, the disagreement is evidence for investigation rather than a reason to overwrite the original annotation.

This distinction also protects the claim in an article title. A guide about Emirati task conversations can explain how to construct the relevant evaluation without pretending that an English-only synthetic fixture is an Emirati acoustic benchmark.

Keep transcript and entity references distinct

A transcript reference describes the words or symbols accepted under the transcription policy. An entity reference describes the intended catalog object under the task policy. Several approved surfaces can refer to one object, while a shared surface can remain ambiguous across multiple objects.

Keep those policies explicit. Do not remove letters, translate product names or choose a nearby spelling merely because the transformed text matches a catalog entry. Such transformations can manufacture an answer rather than preserve the caller's intent.

The W3C guidance on personal names describes why assumptions about name structure fail across cultures. It does not supply a universal transliteration or identity-resolution rule for your catalog.

An approved alias needs a scope and provenance. The reference uses an exact language and task scope, then returns a deterministically ordered immutable tuple of matching entity IDs. It never selects an arbitrary member when more than one entity owns the same approved alias.

Unicode normalization supplies defined textual equivalences, not person identity. The reference deliberately performs literal matching; adding a normalized comparison projection would require an explicit policy and collision fixtures while retaining the original surface.

The names and product-label guide separates canonical spelling, approved aliases and spoken preference. A caller's preferred pronunciation can change the speech rendering without changing the entity identifier.

Use strings for identifiers even when their surfaces contain only digits. RFC 8259's JSON grammar distinguishes strings and numbers, and JSON number syntax does not preserve a literal identifier's leading zeros.

The Arabic digit guide explains why field meaning must precede numeric conversion. A code, a date and an amount can contain similar glyphs while requiring different validation and clarification rules.

Model corrections as new unconfirmed candidates

A correction changes the candidate's meaning or surface under an explicit revision. Confirmation of the earlier revision must not carry over simply because the same connection or conversation remains active.

The spoken self-correction guide explains the conversational boundary. The reference below starts after an adapter has already identified a candidate event or confirmation event and assigned its revision.

A strictly older candidate event is stale. A changed candidate reusing the current revision is a conflict. An exact repeated event identity can be replayed without changing state, while a changed payload reusing that identity is rejected.

Those distinctions matter when a recognition adapter emits partial revisions or an application retries delivery. Treating every repeated value as a fresh confirmation can silently approve work that the caller has already corrected.

The yes-but repair guide covers a particularly useful negative fixture. A positive word followed by a material correction should produce a new unconfirmed candidate, not approval of either the old or new value by accident.

The program uses immutable event values and a fresh local replay state. Its replay result is frozen too. The report function accepts only that exact result type and builds a fresh JSON-ready dictionary; the serialized dictionary is evidence, not a mutable approved proposal.

An adapter still needs to assign event identities correctly. A new confirmation request after repair needs a new identity; an exact retry retains its original immutable request. The fixture does not invent those semantics from natural-language text.

Run the bounded task fixture

Save this entire fence as task_fixture.py and run Python 3.12 or later. It uses only the standard library and prints a reusable result for a synthetic corrected entity, including the distinction between text evidence and unmeasured external completion.

Trusted inputs are curated transcript surfaces, declared task/language scope, reviewed expected entity IDs and adapter-assigned event revisions. The program validates the supported record shape, exact alias matching, correction ordering and confirmation behavior. Audio recognition, annotation expertise and external actions remain outside its boundary.

from dataclasses import dataclass, asdict, FrozenInstanceError
import json


def text(value):
    return type(value) is str and bool(value.strip())


@dataclass(frozen=True)
class Alias:
    language: str
    task: str
    surface: str
    entity: str

    def __post_init__(self):
        if not all(text(x) for x in
                   (self.language, self.task, self.surface, self.entity)):
            raise ValueError("nonempty alias fields required")


@dataclass(frozen=True)
class Event:
    identity: str
    kind: str
    revision: int
    surface: str = ""

    def __post_init__(self):
        if not text(self.identity) or self.kind not in ("candidate", "confirm"):
            raise ValueError("invalid event identity or kind")
        if type(self.revision) is not int or self.revision < 0:
            raise ValueError("nonnegative revision required")
        if type(self.surface) is not str:
            raise ValueError("surface must be text")
        if self.kind == "candidate" and not text(self.surface):
            raise ValueError("candidate surface required")
        if self.kind == "confirm" and self.surface:
            raise ValueError("confirmation carries no replacement surface")


@dataclass(frozen=True)
class Result:
    revision: int
    surface: str
    disposition: str
    candidates: tuple[str, ...]
    entity: str | None
    confirmed: bool
    stale_events: int


def replay(aliases, events, language, task):
    aliases, events = tuple(aliases), tuple(events)
    if not text(language) or not text(task):
        raise ValueError("explicit scope required")
    if any(type(x) is not Alias for x in aliases):
        raise ValueError("typed aliases required")
    if any(type(x) is not Event for x in events):
        raise ValueError("typed events required")
    revision, surface, confirmed, stale = -1, "", False, 0
    seen = {}
    for event in events:
        previous = seen.get(event.identity)
        if previous is not None:
            if previous != event:
                raise ValueError("event identity conflict")
            continue
        seen[event.identity] = event
        if event.revision < revision:
            stale += 1
            continue
        if event.kind == "candidate":
            if event.revision == revision:
                if event.surface != surface:
                    raise ValueError("current revision conflict")
                continue
            revision, surface, confirmed = event.revision, event.surface, False
        elif event.revision == revision:
            matches = {a.entity for a in aliases
                       if (a.language, a.task, a.surface) ==
                       (language, task, surface)}
            confirmed = len(matches) == 1
    matches = {a.entity for a in aliases
               if (a.language, a.task, a.surface) == (language, task, surface)}
    disposition = ("empty" if revision < 0 else
                   "unknown" if not matches else
                   "ambiguous" if len(matches) > 1 else "unique")
    candidates = tuple(sorted(matches))
    entity = candidates[0] if len(candidates) == 1 else None
    return Result(revision, surface, disposition, candidates,
                  entity, confirmed, stale)


def report(case_id, variety, expected_entity, result):
    if not all(text(x) for x in (case_id, variety, expected_entity)):
        raise ValueError("reviewed case metadata required")
    if type(result) is not Result:
        raise ValueError("typed replay result required")
    return {"evidence": "synthetic_text", "case_id": case_id,
            "declared_variety": variety,
            "expected_entity": expected_entity, **asdict(result),
            "entity_exact_match": result.entity == expected_entity,
            "confirmation_required": not result.confirmed,
            "external_task_outcome": "unmeasured"}


def tests():
    source = [Alias("mixed", "item", "demo alpha", "item-a"),
              Alias("mixed", "item", "demo beta", "item-b")]
    catalog = tuple(source)
    first = Event("candidate-a", "candidate", 0, "demo alpha")
    confirm = Event("confirm-a", "confirm", 0)
    repair = Event("candidate-b", "candidate", 1, "demo beta")
    old = replay(catalog, (first, confirm), "mixed", "item")
    assert old.entity == "item-a" and old.candidates == ("item-a",) and old.confirmed
    corrected = replay(catalog, (first, confirm, repair, confirm), "mixed", "item")
    assert corrected.entity == "item-b" and not corrected.confirmed
    final = replay(catalog, (first, confirm, repair,
                            Event("confirm-b", "confirm", 1)), "mixed", "item")
    assert final.confirmed
    assert replay(catalog, (repair, first), "mixed", "item").stale_events == 1
    unknown_scope = replay(catalog, (first,), "other", "item")
    assert unknown_scope.disposition == "unknown" and unknown_scope.candidates == ()
    empty = replay(catalog, (), "mixed", "item")
    assert empty.disposition == "empty" and empty.candidates == ()
    unknown = Event("unknown", "candidate", 0, "unapproved variant")
    assert replay(catalog, (unknown,), "mixed", "item").entity is None
    collision = (Alias("mixed", "item", "demo alpha", "item-c"),) + catalog
    ambiguous = replay(collision, (first, confirm), "mixed", "item")
    assert ambiguous.disposition == "ambiguous" and not ambiguous.confirmed
    assert ambiguous.candidates == ("item-a", "item-c")
    source.clear()
    assert replay(catalog, (first,), "mixed", "item").entity == "item-a"
    for target, field, value in ((first, "surface", "changed"),
                                 (corrected, "confirmed", True)):
        try:
            setattr(target, field, value)
        except FrozenInstanceError:
            pass
        else:
            raise AssertionError("record was mutable")
    for invalid in (
        lambda: replay(catalog, (first, Event("candidate-a", "candidate", 1,
                                            "demo beta")), "mixed", "item"),
        lambda: replay(catalog, (first, Event("conflict", "candidate", 0,
                                            "demo beta")), "mixed", "item"),
        lambda: report("case", "", "item-b", corrected),
        lambda: report("case", "declared", "", corrected),
        lambda: report("case", "declared", "item-b", asdict(corrected)),
        lambda: Event("bad", "candidate", True, "demo alpha"),
    ):
        try:
            invalid()
        except ValueError:
            pass
        else:
            raise AssertionError("invalid fixture accepted")
    output = report("synthetic-repair", "synthetic-unreviewed", "item-b", corrected)
    assert output["entity_exact_match"] is True
    assert output["confirmation_required"] is True
    assert output["external_task_outcome"] == "unmeasured"
    print(json.dumps(output, ensure_ascii=False, sort_keys=True))
    print("synthetic task, alias, repair and confirmation fixtures passed")


if __name__ == "__main__":
    tests()

The printed variety explicitly says synthetic and unreviewed. Changing that label to Emirati would not create acoustic or dialect evidence. A real corpus adapter must supply the declared variety through the approved collection and review process.

Interpret the printed evidence

The corrected fixture prints an exact entity match and a requirement for confirmation. Those outcomes can coexist: the expected entity survived the repair, but the new candidate has not been confirmed under its own revision.

Every replay result also carries the deterministically ordered immutable candidate tuple. A unique result contains its one entity, unknown and empty results contain no candidates, and an ambiguous result retains every distinct matching identity so the owning clarification workflow can present the actual conflict.

It also prints an unmeasured external task outcome. The program has no business service, so it cannot claim that a reservation, purchase or other action completed. Do not fill that field from the entity comparison just to produce a single success flag.

The verified outcome metrics guide provides the larger denominator. A real task report needs a supported terminal observation from the owning workflow, including pending or unknown dispositions when completion cannot be established.

Read the original surface beside the selected entity. The surface can be valid for one task/language scope and unknown in another. This is a useful protection against a product alias leaking into a person-name or account-identifier workflow.

The fixture's immutable tuple snapshot protects catalog membership from a later mutation of the caller's source list. Frozen record fields protect the accepted values. These tests establish the supported value shape; they do not make arbitrary nested Python objects immutable.

The concurrent voice-settings guide describes why ownership matters when moving from an offline replay to a live application. Keep configuration and approved values scoped to the operation that owns them.

Archive the complete output and code version. A test-success sentence alone would not satisfy the promise to produce reusable evidence. The printed record retains the expected entity, candidate, revision, disposition and confirmation state needed to inspect and recompute this specific synthetic comparison.

Test within-utterance language switches

A language switch inside one utterance is different from choosing a language for the beginning of a call. Preserve segment boundaries and the catalog's expected surfaces rather than forcing every word into a single normalized spelling.

The Azure language-identification documentation describes candidate-list behavior and the limits of continuous identification. It states that continuous LID does not identify language changes within the same sentence, so an utterance-level result must not be presented as per-word language evidence.

The documentation also distinguishes a supplied candidate list from an open-set detector. A returned candidate does not establish that the true language was in the list. Keep unsupported or uncertain speech visible in your evaluation rather than relabeling it as the chosen candidate.

A real community request for code-switched Arabic-English recordings asks for appropriate source material and transcripts. It motivates corpus design; it does not supply a measured accuracy result or prove that every Gulf variety behaves alike.

The language-hint evaluation guide offers a related experimental boundary. Hold the source corpus fixed while comparing configurations, and preserve per-item audit evidence without accidentally treating every audit revision as a different policy.

Do not treat an English product insertion as an automatic instruction to translate the product label. The required action depends on the task's catalog and speech policy. Keep an original surface and its approved alias relation available for review.

Choose the evaluation slice

Define a slice with declared variety, task, channel conditions and reference version. If one of those dimensions changes, decide whether the experiment still compares the same workload before averaging its results.

A spontaneous caller correction and a read sentence can test different failure modes. Keep their source conditions visible rather than mixing them under a country-level label and implying one measured capability.

The following table compares evidence types. A check means the source can support the named claim within its declared contract; a cross means it cannot establish that claim by itself.

Keep unknown variety as a valid reporting state when the collection protocol permits it. A missing required label should be rejected during fixture validation. A literal declared label such as unknown must remain distinct from absent metadata, even if both prevent a dialect-specific conclusion.

The accent-cohort guide explains why cohort names and evidence should not become stereotypes. Task-specific findings need the actual tested population and conditions, not a general claim about a group of speakers.

A weighted aggregate can be useful for a declared workload, but keep the slice counts visible. Do not average percentages without their denominators or silently drop difficult tasks because they did not produce a transcript.

Build a consented reference corpus

A real-speech trial needs recordings collected and used under an appropriate consent and retention process. Synthetic text can develop the scorer before that corpus exists, but it cannot replace the acoustic evidence required for a dialect or recognition claim.

Define the permitted collection scope, task prompts and review procedure before recording. Keep personal identifiers out of development fixtures and avoid copying real customer transcripts into a public tutorial or image prompt.

Ask qualified reviewers to label the intended task values and accepted transcription variants independently. Where they disagree, retain both the uncertainty and the resolution policy. Do not make an unsupported reference look certain merely because the scorer requires one expected entity.

Record the source channel and relevant transformations. Resampling, clipping, noise processing and codec changes can alter the material the recognizer receives. Preserve the original approved source and the exact submitted representation under the trial's access controls.

The phone-audio evaluation guide explains why changing the representation changes the experiment. An accuracy difference cannot be attributed to a language setting when one run also received different audio processing.

Keep reviewer instructions versioned. A new spelling convention or entity-alias policy can change the reference without changing the recognizer. Split or rerun affected reports rather than combining outcomes scored against incompatible references.

For read material, avoid treating prompts as proof of what the speaker actually produced. Review the recording. A reader can hesitate, repair a value or pronounce a product differently from the script, and the reference should describe the actual task evidence.

For spontaneous material, preserve the difference between what a caller initially proposed and what they finally intended. The partial-transcript revision guide explains why a convenient intermediate transcript should not become the final command.

Review uncertainty without guessing

Unknown surfaces should produce clarification rather than a nearest catalog match. A fuzzy suggestion may help a reviewer find candidates, but it must not silently become the authoritative entity selected by the task fixture.

Ambiguity is different from absence. When two approved aliases point to different entities, the system has a conflicting candidate set. Preserve that set for the owning clarification workflow; a general yes cannot choose an arbitrary entity from it.

Keep uncertainty localized. A reviewer may know the product but not a nearby name, or may understand the correction while remaining uncertain about the initial value. The reference record should permit those distinctions rather than forcing one transcript-level pass or fail to describe everything.

The unfamiliar-name clarification guide will be part of the same release cohort. Its contract is a useful complement: preserve the caller's surface while asking for the missing evidence instead of silently substituting a familiar name.

When clarification changes a value, create a new candidate revision and invalidate prior confirmation. If a later event from the earlier revision arrives, classify it as stale before evaluating current-state conflicts. This order prevents harmless old delivery from blocking the current task.

Do not treat an interrupted response as a completed confirmation exchange. The interruption-safety guide explains why delivery and task state need explicit ownership. Speech generated by the agent does not establish that the caller received the condition being confirmed.

A review disagreement can also reveal an underspecified task. If reviewers cannot determine whether the caller named a product family or a specific item, improve the task prompt or clarification procedure before using the case to rank recognition configurations.

Connect recognition adapters carefully

The offline example begins with text events. A real adapter must explain how recognition hypotheses become those events, which revisions are final, and which boundaries can legitimately supply confirmation or external completion evidence.

Phrase hints can improve the probability of a term being recognized under a supported provider feature. The Azure phrase-list documentation describes that biasing mechanism; it is not an entity verifier or a guarantee that the requested item was spoken.

Keep hint configuration in the policy snapshot. If the hint list changes between trials, that change belongs in the comparison. If a reviewer merely updates one fixture's audit note, retain it as item evidence rather than inventing a policy difference.

Do not let reference answers leak into recognition input unless that is the explicitly declared experimental condition. Supplying the expected entity as a hint in one run and omitting it from another changes the question the experiment answers.

The cache-key guide is relevant when adapters reuse results. A result from another language/task scope or alias revision cannot be restored as current evidence solely because the surface text looks identical.

Validate event fields before sorting or applying a batch. The fixture receives ordered typed events and rejects Boolean revisions. A live adapter that sorts timestamps needs to validate their types and clock basis before a malformed item can interfere with valid observations.

Preserve raw recognition output alongside the interpreted event within the permitted evidence store. The interpretation can be wrong even when the recognizer's text is faithfully recorded. Separating them makes it possible to reproduce the failure without blaming every task error on ASR.

Measure the complete task population

Define eligibility before examining outcomes. Keep every scheduled case visible, including unsupported configurations, missing references, canceled work and recognition failures. A missing transcript should not disappear from the denominator of an offered-task report.

Use separate numerators for entity exact match, correction handling and supported task completion. Confirmation state can explain why a correct entity was not ready for action. Do not combine those dimensions into a success rate whose meaning changes between cases.

The regression-testing guide explains how to retain the boundary fixtures. Keep the raw case identities, reference revision, configuration snapshot and result evidence so a later implementation can rerun the same task population.

A zero-eligible slice has no rate to report. An observed failure among eligible cases is a real zero numerator. Preserve the distinction, together with the reasons cases were unscored, rather than filling a chart with zeros that imply measured failure.

The cost-per-success guide adds another denominator. Recognition retries, clarifications and interrupted calls can consume resources even when no task completes. Include the actual workload boundary before claiming a configuration is cheaper.

For latency, define the start and end observations separately from quality. A transcript arriving quickly can still lead to the wrong entity, and a completed external operation can leave the spoken response unheard. The latency measurement guide provides the clock and endpoint discipline.

This article reports no accuracy, latency or cost result for a real Emirati-speaking population. It supplies the deterministic task contract and the evidence plan needed before making such a claim.

Qualify a deployment decision

A deployment decision should identify the supported task, the declared speaker-variety scope, tested channel conditions, configuration and remaining uncertainty. A successful demonstration of one product name cannot establish general Arabic-English conversational accuracy.

Preserve the cases that require clarification. They show where the system can continue safely without pretending to know a value. Review whether the clarification wording is understandable under the actual task, rather than treating every extra turn as a failure to eliminate.

Use a small qualified release criterion that can be checked from the retained evidence. For example, require current-revision confirmation before the chosen external action, and require the owning workflow's receipt before reporting completion. Those are contract checks, not invented performance thresholds.

Keep the limitations in the release record. Unreviewed dialect labels, unavailable playback evidence and missing external receipts should remain visible to the person deciding whether the task is ready for real use.

If the reference or alias policy changes, rerun affected fixtures and repeat the relevant listening review. A text-only pass can establish that the revised rules behave as declared, while acoustic and task evidence still need the observation boundary capable of measuring them.

Audit the adapter event contract

The event replay is useful because it separates a supplied correction from the interpretation that produced it. A real system must explain how a recognized utterance becomes a candidate or confirmation event, including the conditions that make the interpretation uncertain.

Keep the raw surface and the interpreted event together under the permitted evidence policy. If a caller says an initial item followed by a repair, the adapter should retain enough provenance to explain why the later candidate superseded the first. The scorer cannot reconstruct that decision from the final entity ID alone.

Give an exact retry the same event identity and immutable payload. Give a new confirmation after repair a fresh identity. Reusing the earlier identity with a changed revision is a conflict, even if the new wording sounds like another agreement.

Test delivery order explicitly. Deliver a newer candidate first, then an older candidate. The older event should remain stale without replacing the current surface. Deliver changed content under the current revision and confirm that it is rejected instead of being accepted as a harmless duplicate.

Test the confirmation boundary separately. An old confirmation replayed after repair should not restore approval. A confirmation for an unknown or ambiguous surface should not select a catalog member. A fresh confirmation of a unique current candidate should progress, so an implementation that rejects everything does not pass the negative cases by accident.

The program's output exposes current state rather than a historical confirmation success. A duplicate earlier event is recorded as a repeated delivery and leaves the newer candidate unconfirmed. That distinction becomes important when a cache or retry layer stores successful past responses.

Keep this behavior in the adapter contract if the application is asynchronous. After waiting for a tool, recognition result or local resource, recheck current task ownership before using its output. The offline replay supplies an ordering model; it does not prove a live adapter performs those final checks.

Bind the reference to the intended task

An expected entity should have a task-specific justification. A product list supplied by a test author can establish the synthetic target, while a real caller's final intended value needs reviewed source evidence and the task's correction policy.

Do not let the reference leak backward into candidate generation. If the test adapter receives the expected entity and then resolves an uncertain surface to that value, the evaluation no longer measures the intended matching behavior. It measures an implementation that was told the answer.

Use separate data paths for the candidate and reference when designing the real harness. The candidate path carries recognition and task interpretation. The reference path carries the approved expectation and review provenance. Join them only at the comparison boundary.

A positive alias fixture should include more than a literal self-comparison. Add an explicitly approved alternate surface for the same synthetic entity, then verify that both resolve within the declared scope. Keep an unapproved nearby surface as a negative control so broad normalization cannot accidentally make every variant succeed.

For a real corpus, a reviewer may accept more than one transcript while the task entity remains unique. Retain that distinction. Conversely, an exact transcript surface can remain ambiguous when the catalog contains several valid objects with that alias.

Record the scope of any pronunciation preference independently. A preferred spoken form can improve the caller's experience while leaving canonical spelling and identity unchanged. The text fixture should not treat that preference as another proof of who the caller meant.

When the reference changes, preserve the earlier release and rerun affected cases. A result scored against the old entity expectation is not directly comparable to one scored against a repaired reference, even if both rows use the same short case label.

Design the first real trial around failure boundaries

Begin with tasks whose intended values can be reviewed reliably and whose external outcomes have a supported observation. Do not start by searching for a flattering aggregate over a large collection of vaguely labeled calls.

Include cases where the English insertion is the critical entity, where it is incidental context, and where a caller changes the value after an apparent agreement. Those cases test different interpretation boundaries without requiring a general claim about all bilingual speech.

Keep read and spontaneous material distinguishable. Read prompts can control lexical content, while spontaneous interaction can expose hesitations and repairs. Neither source style should silently stand in for the other when defining the deployment claim.

Use a predeclared policy for uncertain references and unsupported configurations. Preserve each scheduled identity even when the run fails before recognition produces output. Otherwise difficult conditions can disappear and make the remaining population look stronger than the offered workload.

Review the clarification path as an outcome in its own right. An unknown surface can lead to a useful request for missing evidence, while a confident wrong entity can lead to a harmful action. The task report should preserve those different dispositions rather than rewarding confidence without correctness.

Keep the acoustic observation separate from the printed text assertions. A transcript reviewer can establish what they can recover from a recording under the review protocol. A listening review can establish the perceived speech output under its declared conditions. Neither supplies a business receipt or legal authorization by itself.

Close the trial with an evidence inventory. Identify source conditions, declared variety, reference release, policy configuration, case identities, event interpretations and supported task outcomes. List the observations still missing before making a deployment claim. That inventory turns an initial test into a repeatable engineering method rather than a one-time demonstration.

---

Frequently asked questions

What does an Emirati Arabic-English task fixture measure?

It measures a declared task boundary for a specified corpus and configuration. The offline example measures exact alias matching, revision handling and confirmation state on synthetic text, with acoustic recognition and real completion explicitly unmeasured.

Does ar-AE support prove Emirati dialect accuracy?

No. A support row describes a provider feature and locale configuration. A dialect-specific accuracy claim requires reviewed source material, declared task scope and observed results under the tested conditions.

Can automatic language identification follow every English word?

Do not assume it can. The cited Azure continuous-LID contract does not identify language switches within one sentence, so an utterance-level candidate cannot establish per-word recognition or language assignment.

How should an entity be scored across scripts?

Compare against an explicit expected entity using approved scoped aliases. Retain the original surface and the transcription policy separately, because textual equivalence and business identity answer different questions.

Who should approve reference transcripts?

Use reviewers qualified for the declared variety and task under a documented process. Record uncertainty, disagreements and reference revisions; generated text should not be labeled as verified dialect speech.

Should product names be translated?

Only when the catalog and task policy explicitly support that transformation. Preserve the original captured surface and approved alias relation rather than inventing a translated identifier during evaluation.

How do I represent an uncertain name?

Retain the uncertain surface and a clarification disposition. Do not choose a nearest familiar name or force an expected entity into the record merely to make the test scorable.

What happens after a caller corrects a digit?

The correction produces a new candidate revision and invalidates earlier confirmation. A later confirmation must refer to the current revision with a new request identity, while exact retries retain their original immutable request.

Can two accepted spellings map to one entity?

Yes, when the aliases are explicitly approved for that entity and scope. That mapping is catalog evidence, not proof that every similar spelling or transliteration identifies the same object.

When must an alias remain ambiguous?

When the approved matching scope yields more than one entity. The reference returns ambiguous and prevents confirmation from selecting an arbitrary member of that candidate set.

How should I separate turn completion from recognition?

Record recognition output and conversational turn decisions as different observations. A transcript fragment can be accurate while the caller has not finished a list or correction, so it cannot automatically authorize the next action.

Should MSA examples enter the same cohort?

Only under an explicit reporting design that preserves their label and reference scope. Keep separate slice counts visible; do not relabel unknown or mixed speech as MSA to simplify a report.

How do I report results by task?

Retain eligible counts, observed numerators and unscored reasons for each task and declared variety. A zero-case slice is unavailable, while a zero numerator among eligible cases is an observed result.

What audio conditions belong in the fixture metadata?

Record the relevant channel, encoding, processing and collection conditions under the trial protocol. Preserve which source representation actually reached the recognizer so configuration comparisons do not hide an audio change.

Can synthetic transcripts prove acoustic accuracy?

No. They can validate the scorer and application contract using controlled inputs. Recognition accuracy needs actual audio, reviewed references and an executed experiment with the stated population.

How do I test confirmation without a paid API?

Use the standard-library replay and explicit candidate/confirmation events. It checks the supplied revision contract; it does not prove that a real recognizer, caller or speech-output system generates those events correctly.

What does task completion add beyond WER?

It observes whether the intended workflow reached its supported outcome. WER describes a transcription comparison, while a real task also depends on entity interpretation, correction handling, authorization and external action evidence.

How should bilingual reviewers disagree?

Retain the disagreement and its scope, then apply a documented adjudication or uncertainty policy. Do not silently overwrite one reference or claim certainty solely because a scorer requires a single expected value.

What consent is needed before recording real speakers?

Follow the applicable collection, use and retention process for the actual setting. This article supplies an engineering evidence plan and synthetic fixtures, not a jurisdiction-specific legal consent determination.

Which claims remain unmeasured?

Real recognition quality, dialect accuracy, pronunciation, acoustic playout, comprehension and business completion remain unmeasured by the offline program. Each requires its own supported observation and trial scope.

Begin with one repaired entity

Choose one synthetic entity task and run the complete fence. Inspect the printed candidate revision, exact-match field and confirmation requirement before integrating a recognition adapter.

Then replace the synthetic input only through a reviewed corpus contract. Keep declared variety, source conditions, alias scope and correction evidence visible. Add real task completion only from the owning workflow's supported observation.

The first useful result is a reproducible distinction between recognizing a surface, resolving an entity and acting on current confirmed intent. Preserve that distinction when expanding the task population or evaluating a public voice-agent interface.