Model, prompt, or carrier regression?

TL;DR: Compare frozen cases and list every declared configuration change before assigning a regression to a component. Preserve missing and failed runs, isolate one variable in a controlled follow-up, and treat carrier warnings or score shifts as evidence for hypotheses rather than proof of cause.

---

A voice release changes the model and prompt on the same day that telephone calls become slower. A carrier warning appears in the trace. The evidence supports several hypotheses, but the dashboard assigns the incident to the model because that version changed most visibly.

Start with a change matrix, not a component verdict. Pair observations by frozen input identity, retain each run's configuration and outcome, and classify whether the declared comparison isolates one factor. Missing provenance makes a comparison unclassifiable; equal missing fields do not prove equal settings.

The runnable tool below emits that matrix and a next-experiment suggestion for every scheduled case. It performs no provider calls, transport replay or counter subtraction. Its synthetic fixtures preserve model-plus-prompt confounding, evaluator-only changes, missing observations and aborted work.

For a voice application such as 60db Agent, recovery can proceed while causal certainty remains limited. The critical-field evaluation guide explains why transcript, field and task outcomes must also remain distinct during that investigation.

Table of contents

1. Define the observed regression
2. Freeze cases before comparing components
3. List the whole declared configuration
4. Keep execution and task outcomes separate
5. Run the paired change matrix
6. Interpret the emitted synthetic evidence
7. Treat missing provenance as missing
8. Keep network evidence within its boundary
9. Route evaluator changes to an instrument audit
10. Make diagnostic criteria explicit
11. Choose the next controlled experiment
12. Retain failed and interrupted work
13. Separate local replay from live channels
14. Preserve evidence without exposing callers
15. Record recovery separately from attribution
16. Frequently asked questions
17. Begin with the smallest comparable pair

Define the observed regression

A regression needs an observable outcome and a comparison boundary. It might be a longer caller wait, more incorrect literal values, more failed tasks or a larger fraction of missing responses. These are different outcomes and may have different causes.

Write the symptom before naming a component. A phrase such as slower calls should identify the event interval, clock source, workload and incomplete-case policy. A phrase such as lower quality should identify the rubric, reference evidence and task population.

OpenAI's evaluation best practices emphasize evaluating the actual task. That principle also applies to diagnosis: a convenient model metric should not replace the outcome that triggered the incident.

Do not confuse coincidence with an intervention. A model release timestamp and a carrier warning appearing near the same incident can motivate investigation. They cannot allocate the observed change between those components without comparable evidence and appropriate controls.

The cold-start guide illustrates how a plausible model-latency explanation can conceal queueing or readiness behavior. Define the observed stage before deciding which subsystem should own the repair.

Likewise, the WER and task guide distinguishes transcript edits from completed work. A lower transcript metric can coexist with a task regression, so diagnosis should preserve the separate signals rather than reducing them to one composite label.

Specify the comparison population before selecting examples. A few memorable failures can reproduce a bug without estimating its deployment frequency. A production trend requires a defined scheduled population and an observation policy that keeps missing or unresolved outcomes visible. The same evidence bundle can contain both kinds of material, but it should label the reproduction and empirical study separately so their conclusions do not merge.

Freeze cases before comparing components

Pair old and new observations against the same declared case input. Preserve the canonical input representation, task configuration and permitted source lineage. If the input changes, the comparison changes more than the component under investigation.

The artifact uses a case identity and SHA-256 input digest. Each observation must match the scheduled digest. Reordered result collections remain comparable because the join uses validated identity, not list position or arrival order.

A digest binds only the declared representation. Hashing a transcript does not establish that two audio inputs are identical, and hashing a stored file does not establish identical delivery timing. State whether the input identity covers bytes, decoded samples, transcript or structured task fields.

The synthetic-versus-real speech guide explains source and transformation lineage. A local replay can freeze a file while still differing from the original microphone, network or endpoint path.

Keep case identity separate from run identity. A stable case can be executed many times, while each attempt retains its own run reference. Reusing a run identifier for changed input or configuration weakens the audit trail even when a convenient dashboard groups them together.

The program rejects changed input digests rather than trying to repair the pair automatically. The surrounding collector should preserve that failed comparison and its reason. An invalid pair is evidence about the study's comparability, not proof that the application produced a worse answer.

Canonicalization must be stable and declared. If one side hashes a JSON display and another hashes a differently ordered serialization, their mismatch may describe the collector rather than changed task content. Conversely, overaggressive normalization can hide a meaningful identifier or unit change. Bind the digest to the exact representation appropriate for the declared study and keep its serialization policy with the external manifest.

List the whole declared configuration

The matrix compares model, prompt, tool schema, carrier, codec, sample rate, region, retry policy, workload and evaluator. These are a deliberately bounded set of declared factors, not a complete description of every possible provider or runtime change.

Missing values remain typed None. If either side lacks a required factor, the pair is unclassifiable for isolated-change purposes. Two missing model versions are not evidence that the same model ran, and two missing retry policies are not evidence of equal retry behavior.

Keep stable policy settings separate from run references. A repeated execution should have a new run identity without being classified as a new policy merely because its audit ID differs. Conversely, identical display names do not establish identical prompt or tool-schema contents.

The cache-key guide applies the same principle to reused outputs. Meaningful identity needs the settings that affect the result. A short alias is often a reference to evidence, not the evidence itself.

For prompts and schemas, use controlled content digests or immutable revision references in the real adapter. The synthetic fixture uses short version labels to make comparisons readable. The code validates their type and equality but cannot authenticate their underlying contents.

If the declared factors show only a model change, the tool says single declared change. That wording matters. Undocumented provider changes, collection differences and execution variability remain outside the local matrix, so the classification is not a causal verdict.

The factor list should be extended only when a concrete diagnostic boundary requires it. Adding dozens of nullable settings can produce an elaborate table without improving evidence. Omitting a relevant observed change is equally misleading. Start with the smallest configuration contract that covers the experiment, record known limitations and revise the schema deliberately when a new condition becomes material to the claim.

Keep execution and task outcomes separate

An execution can complete while the business task remains unknown. A request can fail before producing an answer, or be aborted during a replay. Those outcomes belong in execution accounting rather than being silently mapped to poor model quality.

The artifact uses completed, failure, unknown, aborted and not-attempted execution states. Missing observations are another explicit collector state. Independently supplied task labels use success, failure or unknown, with evidence references required for known outcomes.

A failed or aborted execution cannot claim task success through this bounded schema. For a completed execution, the task may still be unknown if the owning system has not supplied the required evidence. The adapter is trusted to interpret that evidence correctly.

The truthful waiting-speech guide explains why a timeout should not be spoken as a confirmed failed action. An unobserved result can remain unknown even while the local request has stopped waiting.

The duplicate-booking guide adds an important retry boundary. Retrying an unknown action can create a second side effect unless the owning system provides the appropriate request-identity contract.

The matrix does not compute an overall quality score. It retains raw dispositions and provenance so another appropriately designed analysis can evaluate the observed symptom. That keeps configuration comparability from being mistaken for successful task completion.

Execution completion should also have a declared boundary. A model response received, a tool operation accepted and an endpoint playback observed are different completion events. The external collector must state which one its completed disposition means. This reference treats it as the completion of the declared replay operation and retains task evidence separately, so the code cannot accidentally award business success for receiving response bytes.

Run the paired change matrix

Save this complete fence as regression-change-matrix.py and run it with Python 3.11 or newer. It uses only the standard library, runs boundary fixtures and prints the full synthetic evidence bundle. No model, carrier or evaluation provider is contacted.

All configuration and outcome declarations come from a trusted external collector. The program checks exact scalar and record types before joining or sorting, owns immutable snapshots and preserves every scheduled identity. Network notes are retained as observations, never converted into causal labels or counter deltas.

from dataclasses import dataclass, replace
from hashlib import sha256
import json


FACTORS = ("model", "prompt", "tool_schema", "carrier", "codec", "sample_rate",
           "region", "retry_policy", "workload", "evaluator")
STATES = ("completed", "failure", "unknown", "aborted", "not_attempted", "missing")


def text(value):
    if type(value) is not str or not value.strip():
        raise ValueError("nonempty text required")


def digest(value):
    return sha256(value.encode("utf-8")).hexdigest()


def hash_text(value):
    text(value)
    if len(value) != 64 or any(c not in "0123456789abcdef" for c in value):
        raise ValueError("lowercase SHA-256 required")


@dataclass(frozen=True)
class Configuration:
    model: str | None
    prompt: str | None
    tool_schema: str | None
    carrier: str | None
    codec: str | None
    sample_rate: int | None
    region: str | None
    retry_policy: str | None
    workload: str | None
    evaluator: str | None

    def __post_init__(self):
        for name in FACTORS:
            value = getattr(self, name)
            if value is None:
                continue
            if name == "sample_rate":
                if type(value) is not int or not 1 <= value <= 384000:
                    raise ValueError("bounded integer sample rate required")
            else:
                text(value)


@dataclass(frozen=True)
class Case:
    case_id: str
    input_hash: str

    def __post_init__(self):
        text(self.case_id)
        hash_text(self.input_hash)


@dataclass(frozen=True)
class Observation:
    case_id: str
    input_hash: str
    run_id: str
    configuration: Configuration
    state: str
    task: str
    task_evidence: str | None
    network_notes: tuple[str, ...]

    def __post_init__(self):
        text(self.case_id)
        text(self.run_id)
        hash_text(self.input_hash)
        if type(self.configuration) is not Configuration:
            raise ValueError("exact immutable configuration required")
        if type(self.state) is not str or self.state not in STATES[:-1]:
            raise ValueError("typed execution state required")
        if type(self.task) is not str or self.task not in {"success", "failure", "unknown"}:
            raise ValueError("typed task outcome required")
        if self.state != "completed" and self.task != "unknown":
            raise ValueError("incomplete execution cannot supply task outcome here")
        if self.task == "unknown":
            if self.task_evidence is not None:
                raise ValueError("unknown task has no authoritative outcome reference")
        else:
            text(self.task_evidence)
        if type(self.network_notes) is not tuple:
            raise ValueError("immutable notes required")
        for note in self.network_notes:
            text(note)


def index(values, expected):
    owned = tuple(values)
    if any(type(value) is not expected for value in owned):
        raise ValueError("exact records required")
    result = {}
    for value in owned:
        if value.case_id in result:
            raise ValueError("duplicate identity")
        result[value.case_id] = value
    return result


def configuration_report(value):
    return {name: getattr(value, name) for name in FACTORS}


def observation_report(value):
    if value is None:
        return {"state": "missing", "task": "unknown", "run_id": None,
                "configuration": None, "network_notes": [], "task_evidence": None}
    return {"state": value.state, "task": value.task, "run_id": value.run_id,
            "configuration": configuration_report(value.configuration),
            "network_notes": list(value.network_notes), "task_evidence": value.task_evidence}


def audit(cases, old_results, new_results, target="model"):
    if type(target) is not str or target not in FACTORS:
        raise ValueError("declared target factor required")
    scheduled = index(cases, Case)
    old = index(old_results, Observation)
    new = index(new_results, Observation)
    for observations in (old, new):
        for case_id, value in observations.items():
            if case_id not in scheduled or value.input_hash != scheduled[case_id].input_hash:
                raise ValueError("unscheduled or changed frozen input")
    ledger = []
    states = {side: {state: 0 for state in STATES} for side in ("old", "new")}
    tasks = {side: {label: 0 for label in ("success", "failure", "unknown")}
             for side in ("old", "new")}
    classes = {}
    for case_id in sorted(scheduled):
        first, second = old.get(case_id), new.get(case_id)
        changed, missing = [], []
        if first is None or second is None:
            classification = "unpaired"
            next_experiment = "obtain the missing declared observation; retain this attempt"
        else:
            for name in FACTORS:
                a, b = getattr(first.configuration, name), getattr(second.configuration, name)
                if a is None or b is None:
                    missing.append(name)
                elif a != b:
                    changed.append(name)
            if missing:
                classification = "unclassifiable_missing_provenance"
                next_experiment = "recover provenance or narrow the comparison"
            elif changed == ["evaluator"]:
                classification = "evaluator_only"
                next_experiment = "freeze outputs and audit judge disagreement separately"
            elif not changed:
                classification = "no_declared_change"
                next_experiment = "check repeatability and undeclared execution conditions"
            elif changed == [target]:
                classification = "single_declared_change"
                next_experiment = "repeat the controlled comparison with independent outcome evidence"
            elif len(changed) == 1:
                classification = "other_factor_changed"
                next_experiment = "test the intended target while holding other declared factors fixed"
            else:
                classification = "confounded"
                next_experiment = "isolate one changed factor while preserving all observed hypotheses"
        classes[classification] = classes.get(classification, 0) + 1
        observations = {"old": observation_report(first), "new": observation_report(second)}
        for side, record in observations.items():
            states[side][record["state"]] += 1
            tasks[side][record["task"]] += 1
        ledger.append({"case_id": case_id, "input_hash": scheduled[case_id].input_hash,
                       "old": observations["old"], "new": observations["new"],
                       "changed_factors": changed, "missing_factors": missing,
                       "classification": classification, "next_experiment": next_experiment,
                       "causal_verdict": "not_established", "transport_delta": "not_computed"})
    return {"evidence": "synthetic declared-configuration comparison", "target": target,
            "scheduled": len(scheduled), "paired_observations": len(set(old) & set(new)),
            "paired_completed": sum(old[key].state == new[key].state == "completed"
                                    for key in set(old) & set(new)),
            "execution_states": states, "task_outcomes": tasks,
            "classifications": classes, "ledger": ledger,
            "provenance_authenticity": "trusted external collector; not verified here"}


def fixture():
    config = Configuration("model-1", "prompt-1", "schema-1", "carrier-1", "pcm", 16000,
                           "region-1", "retry-1", "workload-1", "judge-1")
    cases = [Case(name, digest("synthetic-input:" + name))
             for name in ("isolated", "confounded", "judge", "aborted", "missing")]
    old = [Observation(c.case_id, c.input_hash, "old:" + c.case_id, config,
                       "completed", "success", "synthetic-outcome:" + c.case_id, ()) for c in cases]
    new = [replace(old[0], run_id="new:isolated", configuration=replace(config, model="model-2")),
           replace(old[1], run_id="new:confounded", configuration=replace(config, model="model-2", prompt="prompt-2"),
                   task="failure", task_evidence="synthetic-failure", network_notes=("carrier_warning",)),
           replace(old[2], run_id="new:judge", configuration=replace(config, evaluator="judge-2")),
           replace(old[3], run_id="new:aborted", state="aborted", task="unknown", task_evidence=None,
                   network_notes=("counter_reset_observed; no delta computed",))]
    return cases, old, new


def expect_rejected(function):
    try:
        function()
    except ValueError:
        return
    raise AssertionError("expected rejection")


def tests():
    cases, old, new = fixture()
    report = audit(cases, old, new)
    assert report["scheduled"] == 5 and report["paired_observations"] == 4
    assert report["paired_completed"] == 3
    assert report["classifications"]["single_declared_change"] == 1
    assert report["classifications"]["confounded"] == 1
    assert report["classifications"]["evaluator_only"] == 1
    assert report["classifications"]["no_declared_change"] == 1
    assert report["execution_states"]["new"]["aborted"] == 1
    assert report["execution_states"]["new"]["missing"] == 1
    assert report["task_outcomes"]["new"]["unknown"] == 2
    assert report == audit(reversed(cases), reversed(old), reversed(new))
    assert all(row["causal_verdict"] == "not_established" for row in report["ledger"])
    assert all(row["transport_delta"] == "not_computed" for row in report["ledger"])
    warning = next(row for row in report["ledger"] if row["case_id"] == "confounded")
    assert warning["new"]["network_notes"] == ["carrier_warning"]
    assert warning["changed_factors"] == ["model", "prompt"]
    unknown = replace(new[0], configuration=replace(new[0].configuration, carrier=None))
    assert audit(cases, old, [unknown])["classifications"]["unclassifiable_missing_provenance"] == 1
    # Both sides missing is still unknown, never an equality control.
    missing_old = replace(old[0], configuration=replace(old[0].configuration, carrier=None))
    assert audit([cases[0]], [missing_old], [unknown])["ledger"][0]["missing_factors"] == ["carrier"]
    other = replace(new[0], configuration=replace(old[0].configuration, prompt="prompt-2"))
    assert audit([cases[0]], [old[0]], [other])["classifications"]["other_factor_changed"] == 1
    assert audit([cases[0]], [], [new[0]])["classifications"]["unpaired"] == 1
    expect_rejected(lambda: audit(cases + [cases[0]], old, new))
    expect_rejected(lambda: audit(cases, old, new + [new[0]]))
    expect_rejected(lambda: audit(cases, old, [replace(new[0], input_hash=digest("changed"))]))
    expect_rejected(lambda: replace(new[0].configuration, sample_rate=True))
    expect_rejected(lambda: replace(new[0], state=True))
    expect_rejected(lambda: replace(new[0], network_notes=["mutable"]))
    expect_rejected(lambda: replace(new[0], task_evidence=None))
    expect_rejected(lambda: replace(new[0], state="failure"))
    expect_rejected(lambda: audit(cases, old, new, target=True))
    not_attempted = replace(new[0], state="not_attempted", task="unknown", task_evidence=None)
    assert audit(cases, old, [not_attempted])["execution_states"]["new"]["not_attempted"] == 1
    for state in ("failure", "unknown"):
        partial = replace(not_attempted, state=state)
        assert audit(cases, old, [partial])["execution_states"]["new"][state] == 1
    assert audit([], [], [])["scheduled"] == 0
    snapshot = report["ledger"]
    cases.clear()
    assert len(snapshot) == 5


if __name__ == "__main__":
    tests()
    print(json.dumps(audit(*fixture()), indent=2, allow_nan=False))

The sample-rate bound is an admission limit of this teaching tool, not a claim about every supported codec or provider. The tool does not validate media formats. Configuration references and outcome evidence remain trusted declarations whose authenticity must be established outside this function.

Run the full fence before connecting a collector. Its boundary fixtures cover every classification branch as well as changed input, duplicate identity, missing provenance and unsupported scalar shapes. A surrounding job runner should record an audit exception as a failed comparison attempt, not return an empty green matrix. The local report's determinism applies only to admitted records under this explicitly bounded schema.

Interpret the emitted synthetic evidence

The fixture schedules five identities. Four have observations from both sides and three have completed executions on both sides. The missing fifth result and aborted fourth execution remain visible instead of disappearing from the workload.

One case changes only its declared model. Another changes model and prompt while retaining a carrier warning. The tool labels the latter confounded and preserves both changed factors plus the warning; it does not choose a model, prompt or carrier cause.

Another case changes only the evaluator. The report routes it toward a judge-disagreement audit over frozen outputs. That is a diagnostic next step, not evidence that either evaluator is better or that the application regressed.

The aborted case retains a counter-reset note, but every ledger row explicitly says transport delta not computed. This example intentionally has no typed raw-counter snapshots and therefore performs no transport arithmetic. A note cannot substitute for identity-bound counter evidence.

The report prints old and new configuration values, execution and task states, run identities, input digests, changed or missing factors and next-experiment suggestions. Every causal verdict remains not established. A reader can reproduce the local classification without inheriting an unsupported incident conclusion.

Use the full ledger in review. A single total for confounded cases cannot explain which factors differ or which scheduled attempts are absent. The value of the small tool is that its limited classification remains connected to the declared evidence.

The next-experiment text is deliberately attached to each case classification rather than emitted as one global verdict. Different cases can have different evidence gaps in the same replay. One may lack a baseline, another may have a judge-only change and another may contain several application changes. Keeping those distinctions prevents the most common class from overwriting a more consequential unresolved boundary.

Treat missing provenance as missing

Equality requires observed comparable values. If the carrier field is absent on both sides, the tool marks carrier provenance missing. It does not assume that the same carrier ran because two null values happen to compare equal in the programming language.

Likewise, a moving model alias can conceal a provider update. Preserve the most specific authoritative identity available and record what cannot be pinned. A locally invented snapshot name would make the report look stronger while weakening its truthfulness.

OpenAI's advanced-usage guide describes reproducibility considerations for the interfaces it covers. Documented controls and response metadata have limits; do not generalize them into a guarantee that every model or audio pipeline can be replayed identically.

The artifact keeps missing factors distinct from the literal text unknown. A real adapter can use that string as an actual version label, but it should explain what the label means. Typed absence must remain absence rather than colliding with an admitted text value.

Before comparing runs, review which required factors are actually observed and which are declared by the collector. A revision reference can be useful, but it needs an owning configuration record. A string equality check cannot prove that external settings stayed unchanged.

If provenance cannot be recovered, narrow the conclusion. A historical comparison can remain useful as a symptom record or a hypothesis source without supporting isolated attribution. An honest unclassifiable result is more useful than a component name chosen to complete a dashboard field.

Keep provenance repair separate from result correction. Recovering a missing configuration reference can make a pair classifiable without changing its observed outcome. Correcting an erroneous input digest can invalidate an earlier pair. Both require a report revision with the original review references preserved. Do not edit a historical bundle in place and leave downstream readers believing that its evidence was always complete.

Keep network evidence within its boundary

Transport observations have their own identities, units, clock domains and reset behavior. A carrier warning can indicate a relevant condition, but it does not establish that the condition caused a model response error or the full caller-perceived delay.

The WebRTC Statistics specification defines statistical objects and fields for WebRTC observations. The cited publication is a Candidate Recommendation Draft, so implementation and specification scope should remain explicit. A field name alone does not supply a valid interval calculation.

RFC 3550 defines RTP and RTCP transport and monitoring concepts. It does not promise quality of service or establish that one observed warning explains an application's task outcome. Use the protocol evidence under its actual scope.

The jitter-buffer guide shows why cumulative delay needs the corresponding emitted-count delta from the same receiver, clock domain and epoch. Subtracting across a restart or using a packet denominator can create misleading diagnostics.

The packet-loss guide distinguishes loss structure from aggregate loss rate. Two traces with similar totals can affect speech differently. A generic warning that loss occurred cannot identify the pattern or prove its contribution to a task failure.

This artifact stores notes only. If a future extension computes deltas, it must admit immutable raw snapshots with receiver or stream identity, counter name, unit, clock information and reset evidence. Without that shape, leave the arithmetic unavailable rather than guessing a baseline.

Absence of a network warning is not evidence of a healthy channel unless the collection contract establishes coverage of the relevant condition. A collector may fail to receive statistics during the very interval that matters. Preserve missing observations and collector limitations, and avoid converting a quiet log into a positive network-quality result. The matrix retains notes without interpreting their absence as a measurement.

Route evaluator changes to an instrument audit

A changed evaluator can move a quality score while the application's outputs stay fixed. Treat that as a measuring-instrument question before declaring that the model or prompt produced worse behavior. Preserve both evaluator identities and the frozen outputs they judged.

The judge-update audit provides a paired transition method. It retains independent adjudicated labels, shared judge errors and missing results. Agreement with the old judge is not the same as correctness.

OpenAI's evals guide documents evaluation data and grader configuration. Those capabilities support controlled studies, but using an evaluation interface does not make two differently configured runs comparable automatically.

The matrix classifies evaluator-only declared changes separately from the intended model target. If evaluator and model both change, the comparison is confounded under the declared factors. Run an instrument audit or freeze a common evaluator before making an application quality claim.

The confidence calibration guide adds another distinction: a score shift is not automatically a calibrated probability change. Keep the event definition and label provenance with any numerical evaluation output.

Do not repair a judge comparison by regenerating application answers and calling them frozen. The new answers introduce another variable. Preserve the original objects and start a separately versioned study for changed application outputs if that is the question the team needs to answer.

Make diagnostic criteria explicit

In this table, ✓ marks checks or reports provided by the local matrix. ✗ marks an external observation or experiment. The criteria make a successful metadata audit distinct from a successful causal investigation.

NIST's AI Risk Management Framework provides context for evaluation and measurement under intended conditions. It does not turn a matching configuration table into proof of causality or supply a universal rollback rule.

Use the table to assign follow-up work. A digest mismatch needs input-provenance review. Missing settings need collector repair. A confounded comparison needs a controlled experiment. Asking one model score to solve all three problems hides the actual evidence gap.

Choose the next controlled experiment

When several declared factors differ, choose an intervention that isolates the question relevant to recovery. Hold other declared settings fixed, reuse comparable cases and evaluate the same outcome boundary. Preserve the original confounded run as part of the evidence history.

A model-only comparison can retain the same prompt, tool schema and input collection. A prompt-only comparison can retain the same declared model configuration. A carrier investigation needs appropriate channel controls rather than merely replaying a stored transcript through a model.

The phone-audio guide explains why sample rate and channel representation can affect input. The companding guide shows a codec transformation that should not be hidden inside a nominally model-only test.

If repeated runs vary, study that variability under the same declared configuration. A single improved retry does not establish that a rollback repaired the cause. Keep attempt identities and avoid selecting only favorable outcomes after reviewing them.

The tool emits next-experiment suggestions as bounded diagnostic guidance. They are not executable provider instructions, approval to access private recordings or guarantees that a particular study will identify the cause. The owning engineer must choose a feasible, permitted experiment with the needed controls.

Define the expected observation for the follow-up before running it. If the question concerns a particular field substitution, preserve that literal and its approved context. If the question concerns caller wait, specify the measured interval and incomplete-case policy. A controlled configuration with a changed outcome definition can still answer the wrong question, even when the factor matrix looks clean.

A genuine developer question about telephony latency appears in LiveKit's issue discussion. It is historical community context showing the need to distinguish application and telephone paths, not proof of a current carrier or platform defect.

Retain failed and interrupted work

A replay that completes only easy cases can look better than a complete baseline. Keep every scheduled identity visible and distinguish completed, failed, aborted, not-attempted and missing observations. Outcome denominators should explain which of those states contribute.

The batch cancellation guide explains why interruption cleanup and caller-owned accounting matter. A ledger trapped inside a canceled task cannot reliably report the work that never reached the final gather step.

This tool consumes an already collected ledger. It does not schedule asynchronous work or recover a killed process. Its missing observation state preserves the absence of a result, but an external runner must retain the actual cancellation and execution evidence.

Execution failures and unknown task outcomes remain distinct. A provider request can fail before producing a response; a completed response can still lack downstream completion evidence. Neither should silently become an accepted or rejected task score through a truthy conversion.

The bursty arrival guide shows why workload shape also matters. A replay under different admission pressure can change missing and failed fractions even when each completed response looks comparable.

Before interpreting a quality difference, inspect scheduled counts, paired observations and paired completed executions. Their denominators answer different questions. The emitted report keeps all three visible rather than letting the survivor subset impersonate the whole workload.

Avoid best-attempt selection after a failed run. A successful retry may be useful for recovery, but replacing the original failure with the retry hides execution variability and changes the evaluated population. Retain both attempts in the external ledger and supply the observation chosen by a predeclared policy. This function intentionally rejects duplicate case observations rather than implementing an undocumented retry-selection strategy.

Separate local replay from live channels

A local replay can isolate a model or parser boundary, but it cannot automatically reproduce microphone capture, carrier routing, packet timing or endpoint playback. Declare which part of the path the replay exercises and which parts remain external.

The WebSocket head-of-line guide distinguishes ordered-message behavior from application read-loop stalls. A slow received message does not by itself prove a network byte gap or identify the source of delay.

The network-change guide involves generation and clear-barrier ownership. A stored audio file can remain identical while a stale-generation bug changes what an endpoint receives after reconnecting.

The playback acknowledgment guide separates local writer acceptance from endpoint observations. A successful submission cannot stand in for acoustic audibility or caller comprehension in a regression claim.

Keep transport and application timestamps in their own clock domains. The matrix computes no timing intervals. A broader harness must preserve raw events, units, identity and clock assumptions before subtracting timestamps or adding stage durations.

Reproduce one boundary at a time when a live failure has many moving parts. A stored waveform can help test recognition on the captured samples; a transport simulation can test a declared delay pattern; a business-system stub can test local state handling. Those experiments each have value, but combining their successful outcomes does not reconstruct an unobserved end-to-end event or establish what the caller heard.

When live-channel evidence is unavailable, retain the limitation. A green local replay can narrow the suspected boundary or verify a software fix without explaining the original carrier event. State that useful result directly rather than broadening it into a full-path guarantee.

Preserve evidence without exposing callers

A regression bundle should contain the minimum permitted evidence needed to reproduce the diagnosis. Public examples can use synthetic inputs and version labels. Restricted operational evidence needs its own retention, access and redaction policy.

The telemetry-redaction guide discusses preserving diagnostic structure without publishing sensitive values. Content digests and opaque references can help, but they do not make every combination of metadata nonidentifying.

Keep raw caller recordings, private tool payloads and internal service details out of public reports. A minimal reproduction can often replace literal values while preserving the failure mechanism. Document that substitution so the reproduction is not mistaken for representative deployment data.

The consented dataset guide explains why approved recording use and source lineage matter. An incident recording being available to an engineer does not automatically authorize its reuse in a shared benchmark.

Preserve original failures and review references under the permitted policy. If a report is corrected, create a new revision instead of deleting the earlier observation. The history should show whether evidence changed, the configuration changed or the interpretation changed.

Keep source retention decisions separate from diagnosis status. A case can remain unresolved after its raw source must be deleted under policy. Preserve only the permitted summary and deletion record, and mark which reproduction steps are no longer available. Do not imply that an old digest grants continuing access to the underlying recording or overrides the collection's retention boundary.

Do not publish private architecture to make the explanation seem more concrete. Public interface contracts, synthetic fixtures and observed boundary behavior are enough to explain this diagnostic method. The artifact intentionally uses generic factor names and no real provider account or customer identifiers.

Record recovery separately from attribution

An incident team may roll back a configuration before it can establish cause. That can be a reasonable recovery action under its operating policy. Record the action and observed effect separately from the causal explanation.

A rollback followed by recovery supports an association under the observed conditions. Other things may have changed, including load, carrier path, caches or provider behavior. Preserve those uncertainties instead of rewriting the timeline into a universal verdict about the reverted model.

The admission-control guide and warm-capacity guide illustrate operational changes that can improve caller wait without changing model inference itself. Recovery can target the symptom while investigation continues.

After recovery, run the smallest permitted controlled comparison that can distinguish the remaining hypotheses. Retain failures, unknowns and missing provenance. If a causal explanation remains unavailable, say so and preserve the evidence needed for future review.

The immediate deliverable is an auditable matrix and a concrete next experiment. A component name is useful only when evidence supports it. Until then, the report should explain what changed, what was observed and what remains unresolved.

Assign the next experiment an owner, an admitted input manifest and a specific outcome boundary. That turns uncertainty into a reviewable engineering action without requiring the matrix to invent an answer.

Preserve the practical recovery result even when attribution remains open. Engineers can truthfully report that caller wait returned to the defined operating range under a restored configuration while acknowledging incomplete causal evidence. That statement is more actionable than either overstating certainty or discarding a useful operational observation merely because a complete explanation is unavailable.

Frequently asked questions

How can a regression be attributed to a component?

Define the outcome, preserve comparable inputs and isolate the intended intervention with appropriate controls. A configuration difference supports a hypothesis, not automatic attribution. This local matrix reports declared comparability and explicitly leaves causal verdicts unestablished.

What should a paired replay freeze?

Freeze the declared case input, task definition, non-target configuration and evaluation boundary. Preserve exact run provenance and attempts. State whether input identity refers to audio bytes, decoded samples, text or structured fields because those boundaries are not interchangeable.

Why does a model alias weaken provenance?

Its underlying implementation can change without a new visible name. Record authoritative snapshots or available response metadata and disclose what cannot be pinned. Do not invent a stable identity to make a historical comparison appear stronger.

Can temperature zero make replay identical?

Do not assume it. Use the documented contract of the actual interface and measure repeatability under the declared conditions. The matrix is deterministic over supplied records but does not reproduce or control upstream model execution.

How should a prompt hash be recorded?

Hash a declared canonical representation and retain an authorized immutable reference to its contents. Include relevant examples and policy text. A matching digest establishes identity of that representation, not authenticity of a provider execution or correctness of the prompt.

What if the judge model changed too?

The comparison includes a measuring-instrument change. Freeze outputs and run a separate judge audit, or use a common evaluator for the application comparison. A changed judge score alone cannot establish a model-quality regression.

How should missing replay cases be counted?

Keep them in the scheduled ledger with an explicit missing observation. Distinguish missing from aborted, failed and not-attempted work when the collector knows the state. Report paired and paired-completed denominators separately from scheduled cases.

Can a carrier warning prove causation?

No. It records a potentially relevant condition whose scope needs investigation. Preserve it alongside all configuration changes and task evidence. A controlled channel study or other appropriate evidence is needed before assigning a causal contribution.

Which WebRTC counters belong in evidence?

Use the counters relevant to the declared question with receiver or stream identity, units, clock information and raw snapshots. This tool computes no counter deltas. A warning string is not an adequate substitute for typed counter evidence.

How should counter resets be handled?

Do not subtract across a reset or identity boundary. Start a new valid baseline under the counter's documented contract. The synthetic fixture preserves a reset note and leaves transport arithmetic explicitly not computed because raw snapshots are absent.

Does local replay reproduce a phone network?

It exercises only the path included in the replay. Stored audio or transcripts cannot reproduce every carrier, endpoint and timing condition. State the boundary and retain live-channel evidence separately when the claim requires it.

How do codec changes confound ASR comparisons?

They can change the recognizer's input representation while the model changes too. Preserve codec and sample-rate provenance, validate actual media separately and isolate transformations when evaluating a model-only hypothesis. Metadata equality alone does not validate waveform equivalence.

Why retain canceled and unattempted work?

Otherwise a run can look better by scoring only survivors. Keep every scheduled identity and distinguish execution states from task outcomes. This tool consumes those observations; an external runner must preserve them through actual interruption.

Can several component changes be tested together?

Yes, as a declared bundle comparison. It can reveal an outcome difference under its conditions, but individual attribution remains confounded. Preserve all changes and use controlled follow-ups when the engineering decision needs a component-specific explanation.

How do tool-schema changes affect replay?

They can change what the application requests or interprets even with the same model and prompt. Treat schema as a declared factor and preserve its immutable content reference. Do not hide it under a generic application-version label.

What should an evidence bundle exclude?

Exclude unnecessary personal data, private recordings, credentials, internal hosts and proprietary architecture from public artifacts. Use synthetic reproductions and permitted opaque references. Retain authorized restricted evidence only under the appropriate access and retention policy.

How should regional differences be sliced?

Use explicit task, language, channel and deployment-region definitions with support counts. Do not infer caller identity from an accent label. A regional association motivates investigation but does not establish a carrier or model cause without controls.

Which experiment follows a confounded result?

Choose one relevant factor and hold other declared settings fixed over comparable inputs and outcomes. Preserve the original run. The tool's suggestion is diagnostic guidance; it does not guarantee feasibility, permission or causal identification.

What does a green synthetic replay establish?

That the declared software checks passed on those fixtures under the recorded conditions. It can verify a narrow invariant or reproduce accounting. It does not establish real-caller representativeness, live-carrier robustness or universal provider behavior.

When is rollback justified by limited evidence?

Apply the operating team's explicit recovery policy and record the observed impact and uncertainty. Recovery need not wait for complete attribution, but a successful rollback should not be rewritten as proof of a universal component defect.

Begin with the smallest comparable pair

Take one observed failure and freeze its permitted input representation. Record both runs' declared configuration, execution state and independent task evidence, then run the complete matrix and inspect missing provenance before selecting a hypothesis.

Choose a controlled follow-up that can change the engineering decision. Preserve failed and interrupted work, keep network observations within their scope and record recovery separately from attribution. The useful result is a reproducible path from observed symptom to the next defensible experiment.