Egyptian Arabic versus MSA: choose the right evaluation slices

TL;DR: Compare Egyptian Arabic and MSA using declared variety, task, channel and reference-version slices, with eligible counts and unscored cases visible. Locale availability cannot rank dialect accuracy, and an aggregate is interpretable only when its denominators and any workload weights are explicit.

---

A speech evaluation mixes read MSA prompts with spontaneous Egyptian task calls, then publishes one success percentage. The result moves when the case mix changes, even though neither recognizer nor workflow changed.

The first repair is a reporting contract. Decide what qualifies a case for a named slice, what makes its reference usable and which outcome the numerator actually measures.

Keep a case ledger that preserves every admitted identity, its declared metadata, eligibility and observed outcome. Compute counts before rates, and leave a zero-case rate unavailable rather than inventing a measured zero.

The multilingual benchmark guide separates transcription from task evidence. This article adds a standard-library slice reporter, negative fixtures and a procedure for reviewing the corpus behind any comparison.

All executable examples are synthetic labeled outcomes. They do not recognize audio, determine dialect, measure real recognition quality or report production results for 60db Agent.

Table of contents

1. Define what the comparison is about
2. Declare variety without inferring identity
3. Define the cohort key
4. Keep every case in the ledger
5. Run the bounded slice reporter
6. Recompute the printed counts
7. Keep references comparable
8. Separate transcript and task metrics
9. Use workload weights deliberately
10. Review the source material
11. Diagnose a disappointing slice
12. Control recognition policy changes
13. Report what remains unknown
14. Audit identities and missing metadata
15. Review a changed corpus before rerunning
16. Keep the deployment claim narrower than the corpus
17. Frequently asked questions
18. Begin with an unequal case mix

Define what the comparison is about

An evaluation needs a reader-facing decision before it needs a score. A team choosing a configuration for catalog capture is asking a different question from a team comparing transcription under a fixed spelling convention.

Write the task, population and observation boundary in the evaluation brief. Identify whether the result describes transcript correspondence, intended-entity preservation, confirmation behavior or verified completion of an external workflow.

Keep those outcomes separate when the same case supplies several observations. A correct transcript can feed a faulty entity matcher. A plausible entity can remain unconfirmed. A completed external operation can still produce an inappropriate spoken response.

The outcome metrics guide explains the task boundary. A deployment-oriented report should retain it rather than allowing a convenient transcript metric to become the definition of success.

The phrase Egyptian Arabic versus MSA names reporting scopes in this guide. It does not imply one uniform Egyptian variety, one universal MSA task style or a claim that either label can be inferred reliably from an account's locale.

Begin with the declared corpus labels and document how reviewers assigned them. If source metadata cannot support a label, preserve the uncertainty. A smaller qualified report is more useful than a larger table whose categories are guesses.

Declare variety without inferring identity

Country, nationality, user-interface language and spoken variety are different facts. None supplies a reliable substitute for the corpus's declared variety and review provenance.

BCP 47 defines a language-tagging contract. A tag can describe configured or supplied language metadata without establishing the quality of a recognition result or the identity of the speaker.

The Azure language-support documentation lists feature-specific locale availability. It does not rank Egyptian task recognition against MSA, and a supported locale row cannot be used as a measured acoustic result.

Store requested locale, returned language evidence and reviewed variety separately. An application might request one locale while a corpus contains mixed material. Overwriting the corpus label with the request would hide the mismatch the evaluation should reveal.

The Emirati Arabic-English task guide uses the same distinction for another declared scope. Its synthetic task replay does not become dialect evidence simply by changing a metadata label.

A missing label and a literal declared label must also remain distinct. The Python example uses a typed missing state instead of a reserved display string that could collide with legitimate corpus metadata.

If a reviewer uses the literal label unknown, retain that string as the declared value under the corpus policy. If the required metadata is absent, preserve a separate unscored reason. These states can both limit interpretation while remaining different observations.

Define the cohort key

A cohort key combines the dimensions that matter for the comparison. This guide uses declared variety, task, channel and a consistent reference revision, with recognition-policy settings recorded separately.

Do not put every per-item audit field into the policy identity. A reviewer can update an individual reference note without changing the recognizer's settings. Conversely, a changed hint list or language-selection mode is a real policy difference.

The Hindi-English language-policy guide explains that distinction. Preserve each item's audit evidence while comparing the stable settings that actually define the recognition policy.

Reference revision is part of the scoring contract. The small reporter rejects a mixed reference release rather than guessing whether two versions are compatible. A larger implementation could split them into explicitly separate reports.

Channel is not a decorative label. Phone audio and a controlled wideband recording can produce different source conditions. The phone-audio guide shows why representation changes need to remain visible in an experiment.

Task categories need definitions too. Capturing an identifier, understanding a correction and answering a factual question can expose different errors. Keep their expected values and completion observations scoped to the task rather than sharing an ambiguous success Boolean.

The diagram is a metadata contract. It contains no measured score or claim that one variety outperforms another.

Keep every case in the ledger

Assign an immutable identity before executing the trial. A duplicate identity should not increase a denominator, and a failed attempt should not vanish because no transcript or external receipt was produced.

Eligibility needs a rule declared before reviewing outcomes. A missing required reference can make a case unscored for a particular metric, while a recognition failure under an eligible task remains an observed failure in that metric's denominator.

The bursty workload guide provides a related accounting lesson. Scheduled work, attempted work and completed work describe different populations. Keep the population named whenever reporting a rate.

Retain an explicit exclusion reason with each ineligible case. Do not use a vague filtered label that conceals whether the source was corrupt, metadata was missing or the task fell outside the declared scope.

Keep unknown external completion distinct from a confirmed workflow failure. Both can prevent a stronger success claim, but they need different follow-up evidence. The reporter exposes unknown-completion counts instead of converting missing receipts into definite business outcomes.

The regression-testing guide explains how to preserve a discovered boundary case. Save the exact case identity, configuration and reference release so a later run can compare the same offered population.

Run the bounded slice reporter

Save the complete fence as slice_report.py and run Python 3.12 or later. The program prints a reusable JSON report with raw counts for each synthetic slice and a visible ledger of unscored reasons.

Its trusted inputs are reviewed variety labels, a fixed reference release, task/channel metadata, expected entities and observations from the workflow. It validates only the supported typed record shape and aggregation contract. It does not determine those facts from speech or replace a qualified annotation process.

from collections import defaultdict
from dataclasses import dataclass, asdict
import json


def nonempty(value):
    return type(value) is str and bool(value.strip())


@dataclass(frozen=True)
class Row:
    case_id: str
    variety: str | None
    task: str
    channel: str
    reference: str
    expected: str
    observed: str | None
    completion: str
    eligible: bool = True
    exclusion: str = ""

    def __post_init__(self):
        if not all(nonempty(v) for v in (self.case_id, self.task, self.channel,
                                        self.reference, self.expected)):
            raise ValueError("required corpus metadata missing")
        if self.variety is not None and not nonempty(self.variety):
            raise ValueError("variety must be declared text or None")
        if self.observed is not None and not nonempty(self.observed):
            raise ValueError("observed entity must be text or None")
        if self.completion not in ("completed", "failed", "unknown", "canceled"):
            raise ValueError("unsupported completion observation")
        if type(self.eligible) is not bool or type(self.exclusion) is not str:
            raise ValueError("typed eligibility required")
        if self.eligible and self.exclusion:
            raise ValueError("eligible case cannot carry exclusion")
        if not self.eligible and not nonempty(self.exclusion):
            raise ValueError("excluded case needs a reason")


def summarize(rows, reference):
    rows = tuple(rows)
    if not nonempty(reference) or any(type(r) is not Row for r in rows):
        raise ValueError("typed rows and reference release required")
    ids = [r.case_id for r in rows]
    if len(ids) != len(set(ids)):
        raise ValueError("duplicate case identity")
    if any(r.reference != reference for r in rows):
        raise ValueError("mixed reference releases")
    groups = defaultdict(list)
    for row in rows:
        groups[(row.variety, row.task, row.channel)].append(row)
    output = []
    for (variety, task, channel), members in groups.items():
        eligible = [r for r in members if r.eligible and r.variety is not None]
        unscored = [{"case_id": r.case_id,
                     "reason": "missing_variety" if r.variety is None
                               else r.exclusion}
                    for r in members if not r.eligible or r.variety is None]
        matched = sum(r.observed == r.expected for r in eligible)
        completed_correct = sum(r.observed == r.expected and
                                r.completion == "completed" for r in eligible)
        output.append({
            "variety": variety, "task": task, "channel": channel,
            "offered": len(members), "eligible": len(eligible),
            "entity_matches": matched, "completed_correct": completed_correct,
            "unknown_completion": sum(r.completion == "unknown" for r in eligible),
            "entity_rate": None if not eligible else
                           {"numerator": matched, "denominator": len(eligible)},
            "unscored": unscored,
            "cases": [asdict(r) for r in members],
        })
    return {"evidence": "synthetic_labeled_outcomes", "reference": reference,
            "external_task_outcome": "unmeasured",
            "offered": len(rows), "slices": output}


def tests():
    def row(identity, variety, observed="entity-a", completion="completed", **kw):
        return Row(identity, variety, "capture", "synthetic-channel", "ref-a",
                   "entity-a", observed, completion, **kw)

    cases = (row("e-1", "Egyptian"),
             row("e-2", "Egyptian", None, "failed"),
             row("m-1", "MSA"),
             row("u-1", None),
             row("literal", "(unknown label)"),
             row("excluded", "review-pending", eligible=False,
                 exclusion="reference_under_review"))
    result = summarize(cases, "ref-a")
    assert result["offered"] == 6
    assert result["external_task_outcome"] == "unmeasured"
    by_variety = {s["variety"]: s for s in result["slices"]}
    assert by_variety["Egyptian"]["entity_rate"] == {"numerator": 1, "denominator": 2}
    assert by_variety["MSA"]["entity_rate"] == {"numerator": 1, "denominator": 1}
    assert by_variety[None]["entity_rate"] is None
    assert by_variety["(unknown label)"]["eligible"] == 1
    assert by_variety["review-pending"]["entity_rate"] is None
    assert by_variety[None]["unscored"][0]["reason"] == "missing_variety"
    unknown = summarize((row("pending", "Egyptian", completion="unknown"),), "ref-a")
    assert unknown["slices"][0]["entity_matches"] == 1
    assert unknown["slices"][0]["completed_correct"] == 0
    assert unknown["slices"][0]["unknown_completion"] == 1
    assert summarize((), "ref-a")["slices"] == []
    for bad in (lambda: summarize((cases[0], cases[0]), "ref-a"),
                lambda: summarize(cases, "ref-b"),
                lambda: row("bad", ""),
                lambda: row("bad", "MSA", eligible=1),
                lambda: row("bad", "MSA", eligible=False)):
        try:
            bad()
        except ValueError:
            pass
        else:
            raise AssertionError("invalid corpus accepted")
    print(json.dumps(result, ensure_ascii=False, sort_keys=True))
    print("synthetic slice, denominator and reference fixtures passed")


if __name__ == "__main__":
    tests()

The variety labels in the output are synthetic fixture metadata. They do not describe recordings or measured populations. The deliberately unequal case mix exists to exercise denominator handling, not to suggest a real accuracy difference between Egyptian Arabic and MSA.

Recompute the printed counts

Start with the offered count and the case identities rather than the rate. The reporter prints every admitted row, so a reviewer can reconstruct why a particular slice contains its cases and which ones contribute to the denominator.

Within a slice, eligibility comes from the predeclared rule and required variety metadata. A recognition failure with a missing observed entity still counts when the case is eligible. Missing variety has its own reason and cannot collide with an ordinary string label.

The entity numerator counts exact equality under the supplied task reference. The completed-correct numerator requires that same entity equality plus the external completion observation completed. These are separate fields because a correct candidate can remain pending or fail later in the workflow.

The output preserves unknown-completion counts. Its top-level external task outcome remains unmeasured because this program did not call an owning workflow; each row's completion value is only a supplied synthetic label. A missing receipt can prevent a verified completion claim without proving the external service failed. Keep that uncertainty in the incident and evaluation record until the owning service supplies its supported result.

A slice with no eligible cases prints a null rate. It retains its offered cases and unscored reasons, which is more informative than assigning an accuracy of zero and implying that eligible observations all failed.

JSON's typed representation helps preserve the distinction between null and a literal display label. Export missing metadata as the typed missing state rather than substituting a string that could already appear in the corpus.

Keep references comparable

A reference release defines what the metric considers correct. Changing the accepted spelling, intended entity or ambiguity policy can change the score even when the recognition output is identical.

Version that release explicitly and preserve per-item review history. The artifact rejects rows that claim different reference releases within one report. Its policy is deliberately small: it does not attempt to decide whether two changed references happen to be compatible.

Unicode normalization rules distinguish canonical and compatibility equivalence. If a transcription metric uses a normalization projection, record it as part of the reference contract and retain the original strings for review.

Do not silently strip Arabic marks, fold letters or translate mixed-language product labels. Those choices can alter the comparison's meaning. A transcript-normalization rule should never become an implicit identity rule for people or catalog objects.

The Arabic names guide explains the alias boundary. Explicit approved aliases can connect surfaces to a task entity while retaining ambiguity when several entities share a surface.

A historical SpeechSDK question about Arabic recognition illustrates how configuration uncertainty can become mixed with language-quality claims. Use it as a reproduction prompt, then rely on current configuration evidence and the reviewed reference for the actual experiment.

Keep a review-pending state when annotations are disputed. Excluding the case from a metric may be justified under a declared eligibility rule, but the case and reason should remain in the offered ledger. Otherwise reference uncertainty disappears from the report.

Separate transcript and task metrics

A word or character comparison needs its own tokenization and normalization policy. Task correctness needs an intended value and an observed outcome. Neither should be promoted into the other merely because one is easier to compute.

Personal names expose this difference. A transcript can use an accepted spelling while the entity remains ambiguous, or can differ from the preferred written form while an explicitly approved alias identifies the intended catalog item.

The W3C personal-name guidance explains why assumptions about universal name structure are unreliable. It does not authorize an evaluator to infer a person's identity from a familiar transliteration.

Keep spoken correction handling visible too. A recognizer can capture both the initial and repaired value while the application mistakenly acts on the first. The self-correction guide describes why final intent needs a separate state contract.

A correct entity is still weaker than a confirmed action. The yes-but guide provides a boundary fixture where an apparent agreement contains a correction and invalidates earlier confirmation.

Use criteria to communicate what each evidence type can establish. A check below means the evidence supports the named boundary within its declared scope. A cross means it cannot establish the claim alone.

Use workload weights deliberately

An aggregate answers a question about a population mix. If that mix differs between trials, the aggregate can move without any change in a slice's behavior. Keep the separate reports visible before considering a combined value.

A pooled count uses the observed eligible populations. Add the appropriate success numerators and eligible denominators, then compute the ratio. This answers the question for that collected case mix, whose size and composition should remain in the report.

An equal-slice average answers another question. It gives the same influence to slices that may contain different numbers of cases. That can be a declared analytical choice, but it is not interchangeable with the pooled count.

A deployment-weighted aggregate needs an explicit workload profile. Record the source and version of the weights, the slices they cover and the rule for missing slices. Do not invent weights because the evaluator expects a single score or choose them after observing which configuration wins.

The bounded reporter intentionally emits no weighted aggregate. Its raw counts are sufficient for a separately specified calculation, while the deployment profile remains a decision the fixture cannot infer from labels or case frequency.

A profile that assigns weight to a zero-eligible slice cannot produce a supported complete aggregate without an explicit missing-data policy. Preserve an unavailable result or a qualified partial-coverage report; silently renormalizing the remaining weights changes the workload question.

The cost-per-success guide applies the same discipline to operating costs. A configuration that excludes difficult calls from its success population can appear cheaper while serving less of the intended workload.

Keep sensitivity analysis distinct from the primary result. Exploring several plausible profiles can reveal how dependent a decision is on workload mix. It does not justify presenting the most favorable profile as the only deployment population.

Review the source material

A slice label is only useful when the underlying material matches its description. Review whether cases are read, prompted, spontaneous or adapted from another source, and retain that provenance in the permitted corpus record.

Do not infer the spoken variety from a script supplied to a reader. The recording may contain a repair, an English insertion or a wording change. Reference the actual source evidence rather than assuming the prompt was produced exactly.

A report about Egyptian task calls needs an appropriate collection and annotation process. This article does not provide verified Egyptian utterances or claim that generated Arabic strings are linguistically representative. Synthetic fixtures can test reporting logic while those source requirements remain unmet.

Record consent, permitted uses and retention boundaries through the actual organization's process. Avoid placing real recordings, transcripts or identifiers in public examples or graphic-generation prompts. The public teaching fixture needs only synthetic identities and declared metadata to demonstrate its contract.

Keep the recognizer's actual input representation inspectable. Sample rate, channel count, encoding and any processing can change the experiment. The audio-boundary guide explains why a corrupted source can look like a language or recognition failure.

Review the reference blind to the candidate output where practical under the protocol. Seeing a recognizer's plausible answer can influence a reviewer who is uncertain about a word. Preserve disagreement and adjudication rather than allowing that influence to become an invisible reference revision.

Separate source adequacy from configuration support. A supported locale with unsuitable task material is not a useful dialect evaluation. A carefully reviewed corpus with an unsupported provider configuration also cannot answer the intended deployment question without a different execution plan.

Document which cases exercise corrections, identifiers and conditions. A corpus of easy standalone words can leave the workflow's hardest decisions untested even when the total case count appears substantial.

Diagnose a disappointing slice

Start with the failed cases and locate the first unsupported boundary. Was the source usable? Did the reference establish the intended value? Did recognition preserve the relevant surface? Did the entity matcher select the correct object? Did current confirmation reach the owning workflow?

Do not label every failed case a dialect failure. A wrong task outcome can arise after correct recognition, and an unclear reference can make a correct system response look wrong. Keep the raw observations needed to distinguish those causes.

The workflow enforcement guide helps separate candidate evidence from permission to act. A recognizer may supply a plausible candidate while the workflow correctly asks for clarification. That outcome needs the metric's declared treatment instead of an automatic error label.

Analyze timing separately from meaning. A late result can be unusable for the task while still matching its text reference. The voice latency guide explains how to define the relevant start, end and clock domains before calculating a delay.

Review interruptions and repairs in their event order. A confirmation from an earlier revision can arrive after the caller changed the value. The interruption-safety guide describes why current intent and delivery ownership must survive asynchronous work.

Look for shared source or annotation defects before changing model settings. Several apparently independent failures can originate from the same conversion step, reference convention or alias rule. Repair that cause and retain a regression rather than attributing the cluster to a broad variety label.

The Arabic digits guide provides a concrete example: converting an identifier to a numeric value can remove a leading zero after recognition succeeded. That is a field-contract defect, not evidence about the speaker's variety.

Preserve unresolved cases as unresolved. A report can guide the next experiment by naming the missing observation. It should not force a root-cause label merely because a chart requires every case to have one.

Control recognition policy changes

Hold the corpus fixed when comparing recognition configurations. Record the service or model version where available, requested locale, language-identification mode, hint configuration and preprocessing contract.

The Azure language-identification documentation describes a candidate-list contract and its limitations. A returned candidate is not an independent dialect annotation, and continuous LID should not be described as per-word identification inside a sentence.

The same documentation restricts how locales enter the candidate list. Check the current supported configuration rather than proposing a candidate-list experiment that treats several locales of the same base language as independent dialect detectors.

Azure phrase lists can bias recognition toward supplied terms under the documented feature. Preserve which terms were supplied. A reference answer leaked into only one trial can change the evaluation question instead of demonstrating a generally better configuration.

Keep policy settings separate from per-item annotation history. If two cases have different reviewer revisions but identical recognition settings, they can still share a recognition policy. The reference release used for scoring must nevertheless remain compatible under the report's explicit rule.

Version changes should invalidate affected comparisons deliberately. Do not reuse a prior transcript as though it were produced under a new hint list or locale mode. The cache-scoping guide explains why apparently identical input text does not establish equivalent policy context.

For a provider comparison, preserve execution failures and unsupported conditions. Availability can decide whether a configuration belongs in the experiment, but it cannot supply a made-up score for cases that were never executed.

A configuration change is a testable intervention. State which failure boundary it is intended to repair, rerun the same eligible cases and retain both successful and unsuccessful results. Avoid searching settings until a favorable aggregate appears without reporting the search procedure.

Report what remains unknown

The strongest result should use only evidence the experiment actually collected. The synthetic reporter establishes denominator and reference behavior on supplied labels; it cannot establish acoustic quality or the correctness of those labels.

A real trial should identify unmeasured source populations and task types. If no spontaneous correction cases were collected, say so. If a channel or mixed-language condition was absent, do not generalize into that condition from a locale table or a nearby variety label.

Keep confidence in the annotation separate from the recognizer's confidence. The two values describe different sources of uncertainty. A high recognition confidence cannot repair an uncertain expected entity, and a confident reviewer cannot establish that the workflow completed externally.

Report missing completion evidence without converting it to a definite business failure. A client can lose the receipt after the external operation completed. The owning service's reconciliation contract decides that outcome; the slice reporter only retains the supplied unknown state.

The playback evidence guide adds another limit. A correct task result does not prove the caller heard the spoken response or understood a condition. A listening or acoustic trial requires separate observations and should retain that boundary in its title and method.

Do not claim statistical certainty from the illustrative counts printed here. Their numeric values are chosen synthetic fixtures for testing logic. Real uncertainty analysis needs the actual sampling design, case dependencies and measurement procedure.

The eventual release decision should name the observed task scope, remaining gaps and conditions that would reopen the choice. A careful bounded conclusion lets another engineer identify whether a new task population is covered or needs another trial.

Audit identities and missing metadata

Duplicate detection must happen before aggregation can inflate a denominator. An identity repeated with identical content is still one case under this reporter's contract, so the program rejects the repeated corpus entry instead of treating it as another observation.

A larger execution system may distinguish a logical case from multiple attempts. If so, give both identities explicit fields and declare whether the metric counts cases, attempts or sessions. Retrying a case must not silently increase the population that the report calls distinct tasks.

Missing metadata needs a representation that cannot collide with any admitted real label. The code uses None for missing variety and retains every valid string literally, including the deliberately awkward label used in its negative fixture.

That choice is about the key's type, not the display wording. A UI can display missing variety using a helpful phrase while the underlying grouping key remains separate from a corpus that happens to contain the same phrase as a legitimate label.

Review exclusions at the same level. An explicit eligibility flag with a reason is different from an absent variety field. The report preserves raw cases so a reviewer can identify both conditions even when its concise unscored row names the first blocking reason.

Validation precedes aggregation. The small artifact validates immutable rows when constructed, then rejects mixed reference releases and duplicate IDs before counting. A live adapter must preserve that order rather than sorting or grouping malformed metadata before checking it.

Keep a failure test for every collision rule. Missing versus literal labels, duplicate IDs and mixed references are especially useful because they can produce plausible-looking totals instead of obvious crashes. A readable report is not proof that its population is correct.

Review a changed corpus before rerunning

A new corpus release can add material, repair references or revise labeling rules. Record which change occurred rather than treating every new report as a pure system improvement comparison.

For overlapping cases, retain stable identities when the case remains the same logical observation. Record a new reference revision when the expected value or annotation contract changes. If the source material itself changes, identify that change so a reviewer does not compare two different recordings as one unchanged case.

Compare offered identities before comparing scores. A higher aggregate after removing difficult cases does not establish better recognition. A lower aggregate after adding a new task type can reflect broader coverage rather than a regression in the original workload.

Use a fixed regression subset when the question is whether a code change broke earlier behavior. Use a separately labeled expanded corpus when the question is whether the system covers new tasks. The same release can include both reports without merging their purposes.

The spoken-answer consistency guide provides a related source-control lesson: keep the content and observation boundary stable while testing one declared intervention.

Reference repairs should be reviewed independently of a desired ranking. A disputed annotation can change which configuration appears stronger. Preserve the earlier reference, review reason and affected reports so the change remains auditable.

No corpus revision can retroactively establish uncollected acoustic or workflow evidence. If the earlier run never recorded an external outcome, a new annotation should not invent that outcome from the transcript alone. Re-execute or retain the missing state.

Keep the deployment claim narrower than the corpus

A corpus can exercise selected tasks under declared conditions without representing every caller who shares a broad language label. Describe the tested task and channel before considering a broader rollout statement.

Separate a deployment acceptance rule from a measured result. A team can require current-revision confirmation or a minimum evidence completeness rule as policy. This article does not supply invented accuracy thresholds or claim that a standard dictates one universal acceptable error rate.

Review the consequence of an error. Misrecognizing a conversational filler and changing a critical identifier can have different task effects even when both count as text differences. Preserve the task-specific measures rather than relying on a single transcript aggregate to describe every failure.

The identifier-capture guide explains why an alternate capture or clarification route can be part of the task contract. Its availability is a workflow choice to evaluate, not proof that voice recognition is correct.

A qualified decision also includes a fallback for unknown or ambiguous input. Clarification can preserve task correctness when the current evidence is insufficient. Evaluate whether the fallback works for the declared tasks instead of excluding its cases from the report.

Keep deployment monitoring tied to the same meanings. If production counts a completed task differently from the trial, the two rates cannot be compared directly. Document the mapping and preserve any unavailable observations rather than forcing the populations to align by name.

Before approving the report, ask another engineer to rebuild one slice from its printed raw cases. They should identify every offered identity, apply the declared eligibility rule, count exact entity matches and distinguish completed-correct outcomes from pending completion evidence. The derived fields should follow from that reconstruction without access to a hidden dashboard or a remembered exclusion.

Repeat the exercise with the zero-eligible slice and the literal missing-looking label. Those controls check the meaning of unavailable output and demonstrate that display text cannot merge with typed absence. Keep them in the regression suite when changing export formats or introducing a database layer, because serialization can accidentally collapse distinctions that the original in-memory grouping preserved.

Then inspect the report title and summary against the evidence actually collected. A synthetic logic check should say synthetic. A bounded audio trial should name its task and source conditions. A deployment decision should name the population and uncertainty it considered. These wording checks prevent a technically correct count from becoming an unsupported broader claim during presentation.

---

Frequently asked questions

Why separate Egyptian Arabic from MSA in evaluation?

They are distinct declared evaluation scopes whose source styles and task populations can differ. Separate reporting makes those conditions visible without claiming that either label describes one uniform speaker population.

Does ar-EG identify the actual spoken variety?

No. It can identify a requested or supplied locale under a provider contract. Reviewed source metadata and annotation provenance must establish the variety label used by the evaluation.

What belongs in a task cohort?

Declare variety, task, channel and a compatible reference release, together with the recognition policy being tested. Keep unrelated per-item audit revisions as evidence rather than accidentally turning them into different recognition settings.

How should mixed Egyptian-MSA speech be labeled?

Use the corpus's explicit mixed or uncertainty policy. Preserve that label and its provenance instead of silently assigning the material to whichever slice makes the aggregate simpler.

Who assigns variety labels?

Use qualified reviewers under the declared corpus process. Record how labels were assigned and disputed; the offline reporter does not classify speech or validate a reviewer label.

Should unknown variety be excluded?

It depends on the predeclared metric scope. Preserve unknown or missing cases in the offered ledger with their reasons, and do not use outcome quality to decide eligibility after the run.

Why can aggregate accuracy hide a failure?

Its value depends on the case mix and weighting. A large easy slice can dominate a pooled count while another task remains weak, so retain each slice's numerator and denominator.

How are slice denominators defined?

Count cases eligible under the declared task and reference rules. Eligible recognition failures remain in the denominator; missing required metadata or out-of-scope cases need visible unscored reasons.

Can I average two percentage scores?

Only when that averaging rule answers the intended question. A pooled count, equal-slice average and deployment-weighted aggregate describe different populations; preserve their counts and weights explicitly.

When are workload weights appropriate?

Use them when a declared deployment profile supplies the intended case mix. Record their source, version, coverage and missing-slice rule instead of selecting favorable weights after examining results.

What should a zero-case slice show?

Show an unavailable rate and its offered or unscored context. A zero numerator among eligible cases is different from having no eligible observations at all.

How do channel and noise affect the cohort?

They describe source conditions that can change the experiment. Retain the actual input representation and processing so a configuration comparison does not hide a channel or preprocessing change.

Should read speech and spontaneous calls share a score?

Only under an explicit reporting design that preserves those source styles and their counts. A read prompt does not cover spontaneous corrections merely because both contain Arabic words.

How do I prevent duplicate test cases?

Assign stable case identities and reject duplicates before aggregation. If repeated attempts are legitimate, model attempt identity separately and state whether the metric counts cases or attempts.

How should I version reference transcripts?

Preserve a reference release and per-item review evidence. Split or reject incompatible releases rather than comparing outputs against a mixture of changing expected values.

Can WER compare different spelling conventions?

The comparison needs a declared transcription and normalization policy. A change in accepted spelling can move the score without changing recognition, so keep the original strings and reference version.

Should entity correctness have its own metric?

Yes, when the task depends on an intended catalog object or identifier. A transcript match alone does not establish a unique entity, current confirmation or verified external completion.

Does a locale table rank systems?

No. It documents feature availability under its own provider scope. Ranking task quality requires an executed trial with reviewed references and a consistent workload.

How do I report reviewer disagreement?

Retain the disagreement, its scope and the resolution or uncertainty policy. Do not overwrite it silently or let the candidate system's output become an unrecorded tie-breaker.

What evidence supports a deployment decision?

Use the declared task outcomes, case populations, configuration, source conditions and remaining gaps. The synthetic program proves its reporting logic only; real speech and workflow claims need their own observations.

Begin with an unequal case mix

Run the complete fence and recompute the synthetic slice counts from its printed case ledger. Confirm that a failed eligible case stays counted, missing variety remains distinct from every string label and a zero-eligible slice has no invented rate.

Then choose one reviewed task population for the real experiment. Keep the reference release and recognition policy fixed while comparing configurations, and retain separate entity and completion observations.

Use the result to make a bounded decision about that task and channel. Expand the corpus deliberately when the deployment question expands, while preserving the evidence needed to explain whether the case mix, reference or system behavior actually changed.