Detect evaluator drift after a judge-model update

TL;DR: Freeze the cases, agent outputs and rubric before comparing judge versions. Join results by identity and content provenance, preserve missing and invalid outcomes, and adjudicate disagreements before calling a judge-score change a quality regression.

---

The agent deployment has not changed, yet an evaluation dashboard suddenly reports more failures. The judge-model alias was updated between runs. The dashboard is measuring with a different instrument, and the new score alone cannot say whether the agent became worse.

Begin with a paired audit over frozen outputs. Keep both judgments for each case, the exact rubric and prompt provenance, and any independent adjudicated label. Count transitions before comparing aggregate pass rates. A case moving from pass to fail deserves investigation; it does not explain its own cause.

The runnable artifact below produces a transition table, separate human-agreement denominators and an all-case review ledger. Its fixtures show a shared judge error, reordered results, a missing result and changed content. All are synthetic; no model calls or real quality measurements are implied.

For voice systems, the evaluation object matters especially. A transcript-only judge cannot establish acoustic playback or completed business work. The playback-evidence guide defines one of those separate observation boundaries.

Table of contents

1. Treat the judge as a measuring instrument
2. Freeze the object being measured
3. Define the paired comparison population
4. Keep adjudication independent
5. Run the disagreement audit
6. Interpret the printed transition report
7. Retain errors ties and absent results
8. Version prompts rubrics and model snapshots
9. Separate variability from instrument change
10. Make release criteria explicit
11. Inspect task and language slices
12. Adjudicate disagreements without rewriting history
13. Avoid calibration claims from agreement alone
14. Refresh the reference collection deliberately
15. Connect the audit to a release investigation
16. Frequently asked questions
17. Start with one frozen comparison

Treat the judge as a measuring instrument

A judge maps an input bundle and rubric to an evaluation output. Changing the model, prompt, examples, decoding settings or output parser can change that mapping. The resulting score is an observation made by the instrument, not an intrinsic property detached from it.

OpenAI's evaluation best-practices guide describes task-specific evaluation and the need to calibrate automated evaluation against human judgment. The engineering implication is to version the evaluation process alongside the application being evaluated.

Distinguish three propositions. The new judge disagrees with the old judge. The new judge disagrees with adjudicated reference labels. The agent itself produces worse task outcomes under comparable conditions. The first proposition can be checked with paired judgments; the others need additional evidence.

A shift might reveal a flaw in the old judge rather than a flaw in the application. It might also reflect a new parsing bug, an accidentally edited rubric or a lost context field. Inspect the complete measuring path before attributing the result to the model update.

The model, prompt and carrier guide handles the broader attribution problem. This article deliberately isolates one part: comparing declared judge instruments over exactly the same admitted evaluation objects.

The bounded audit does not decide which judge is better. It makes the evidence needed for that decision available in a form a reviewer can recompute. That separation prevents an attractive dashboard difference from quietly becoming an unsupported causal conclusion.

Name the instrument change precisely in the release record. A provider model update, a prompt revision and a parser repair are different interventions, even if the dashboard groups them under one evaluator name. If the team intentionally changes several together, describe the comparison as a bundle change. The audit can still show disagreement, but no local transition table can allocate that disagreement among the bundled interventions without additional controls.

Freeze the object being measured

For an instrument comparison, preserve the case input, agent response, rubric and reference label version. If the agent regenerates its response during the second run, the comparison now contains both an output change and a judge change.

The code uses SHA-256 digests of synthetic input and output text to bind result records to frozen cases. In a real adapter, specify the canonical serialization and retain the authorized evidence reference. A digest is only useful when both sides hash the same declared representation.

Do not hash a transcript and describe it as an audio identity. Audio bytes, decoded waveform, transcript and task record are different objects. A transformation can preserve one while changing another. The synthetic-versus-real speech guide explains why source and transformation provenance also matter.

The audit rejects a judgment whose input or output digest differs from the corresponding frozen case. It does not silently treat that result as a valid pair. The surrounding runner should retain the rejected attempt with its error reason rather than replacing it with a success-shaped empty record.

Keep a stable case identity separate from the content digest. Identity supports lineage and review references; the digest detects declared content changes. Reusing an identity with changed content requires an explicit revision and a new comparison contract, not an unnoticed overwrite.

Canonicalization belongs in the adapter contract. Decide whether line endings, Unicode representation and serialized field order are part of the evaluated object before hashing. A well-intended cleanup between runs can change content or hide a meaningful literal difference. For a transcript rubric, preserve the admitted transcript representation; for a structured task rubric, retain the typed field representation and its schema version rather than hashing a convenient display string.

The spoken self-correction guide offers a related example. A corrected candidate is a new evaluation object. A judgment bound to the earlier value cannot automatically evaluate the corrected one simply because both belong to the same call.

Define the paired comparison population

The primary transition table includes cases with a valid binary judgment from both instruments. Its denominator is jointly valid pairs. Pass-to-pass, pass-to-fail, fail-to-pass and fail-to-fail are separate cells whose total must equal that denominator.

All scheduled cases remain in the ledger even when they lack a valid pair. A missing new result is not a new failure label; a judge error is not an application failure; an abstention is not a pass. Preserve these states so the reader can evaluate coverage separately from agreement.

Join records by unique case identity after validating their scalar fields. Positional zipping is unsafe: providers, batch jobs and retry collectors can return results in a different order. The boundary fixture reverses one collection and asserts that the complete report stays identical.

Reject duplicate identities within an instrument's supplied results. Choosing the last result would silently impose a retry policy and discard the earlier evidence. If repeated judgments are part of the experiment, give them explicit repetition identities and evaluate repeatability under a separate protocol.

The webhook ordering guide addresses an analogous ingestion problem. Delivery order and identity are separate facts. An evaluation collector should not let incidental arrival order define which observation counts.

When presenting agreement, print its numerator and denominator together. A stable agreement fraction with fewer valid pairs can conceal lost difficult cases. The audit reports scheduled, paired and per-instrument usable counts so a reviewer can see that change before interpreting a trend.

Decide how the scheduled collection is admitted before starting either run. A case excluded because its evidence is unavailable should have an explicit exclusion record outside the bounded function. A case scheduled successfully but missing a judge result belongs inside its ledger. Mixing those two boundaries makes it difficult to tell whether collection policy or execution reliability changed between versions.

Keep adjudication independent

The old judge is a previous instrument, not ground truth. If old and new judges agree on an incorrect answer, agreement can look perfect while both miss the task requirement. Include independent adjudication where available and retain its version and uncertainty.

Here, human labels have the narrow values pass, fail or unknown under one declared rubric. The adapter is responsible for obtaining those labels through an appropriate review process. The code does not verify reviewer competence, independence, consent or actual task completion.

Agreement against human labels has a separate denominator for each instrument: cases with a valid binary judgment and a known adjudicated reference. A missing result in one instrument need not erase a usable result in the other, but their denominators must remain explicit.

The critical-field evaluation guide shows why a rubric should name the actual outcome. A judge that rewards fluent paraphrasing can still miss an altered identifier or negation that invalidates the task.

Human review can also disagree. Mark unresolved adjudication unknown rather than forcing consensus in the data importer. If the rubric itself is ambiguous, revise it deliberately and treat the revision as an instrument change requiring a separate comparison.

Keep adjudication provenance separate from the judge's own explanation. A plausible rationale generated by the judge is part of its output, not independent corroboration. The review process should point to the task requirement and permitted evidence that support its disposition. Where that evidence is insufficient, an unknown label is more informative than a confident narrative attached to an unsupported binary decision.

For consequential outcomes, preserve the authoritative evidence source. A human reading a transcript may judge whether an answer follows a rubric, but that review does not establish that a downstream booking exists. Keep observable external outcomes separate from evaluator opinions.

Run the disagreement audit

Save this complete fence as judge-disagreement-audit.py and run it with Python 3.11 or newer. It uses frozen records, validates exact types before joining, executes negative fixtures and prints the full synthetic audit. It performs no network requests.

The scope is intentionally binary judgment transitions with explicit nonbinary dispositions. A real adapter must preserve provider responses, parser versions, canonical content hashing and adjudication provenance. Do not convert arbitrary model text to pass or fail through truthiness.

Run the whole file before adapting it. The boundary fixtures are part of the declared contract, including empty denominators and mismatched per-observation instruments. Removing them for a shorter demonstration would remove evidence for behavior the surrounding article relies on.

from dataclasses import dataclass, replace
from datetime import datetime
from hashlib import sha256
import json


def text(value):
    if type(value) is not str or not value.strip():
        raise ValueError("nonempty text required")
    return value


def digest(value):
    return sha256(value.encode("utf-8")).hexdigest()


def hash_text(value):
    text(value)
    if len(value) != 64 or any(c not in "0123456789abcdef" for c in value):
        raise ValueError("lowercase SHA-256 text required")


@dataclass(frozen=True)
class Instrument:
    model: str
    prompt_hash: str
    rubric: str
    settings_revision: str
    sampling_settings: str

    def __post_init__(self):
        for value in (self.model, self.rubric, self.settings_revision, self.sampling_settings):
            text(value)
        hash_text(self.prompt_hash)


@dataclass(frozen=True)
class Case:
    case_id: str
    input_hash: str
    output_hash: str
    human: str
    rubric: str
    slice_name: str | None

    def __post_init__(self):
        text(self.case_id)
        text(self.rubric)
        hash_text(self.input_hash)
        hash_text(self.output_hash)
        if type(self.human) is not str or self.human not in {"pass", "fail", "unknown"}:
            raise ValueError("typed human disposition required")
        if self.slice_name is not None:
            text(self.slice_name)


@dataclass(frozen=True)
class Judgment:
    case_id: str
    input_hash: str
    output_hash: str
    label: str
    run_id: str
    instrument: Instrument
    observed_at: str

    def __post_init__(self):
        text(self.case_id)
        text(self.run_id)
        if type(self.instrument) is not Instrument:
            raise ValueError("immutable instrument required")
        text(self.observed_at)
        try:
            observed = datetime.fromisoformat(self.observed_at)
        except ValueError:
            raise ValueError("ISO observation timestamp required") from None
        if observed.utcoffset() is None:
            raise ValueError("timestamp offset required")
        hash_text(self.input_hash)
        hash_text(self.output_hash)
        if type(self.label) is not str or self.label not in {
            "pass", "fail", "tie", "abstain", "error"}:
            raise ValueError("typed judge disposition required")


def index_exact(values, expected):
    owned = tuple(values)
    if any(type(value) is not expected for value in owned):
        raise ValueError("exact record type required")
    result = {}
    for value in owned:
        if value.case_id in result:
            raise ValueError("duplicate identity")
        result[value.case_id] = value
    return result


def instrument_report(value):
    return {"model": value.model, "prompt_hash": value.prompt_hash,
            "rubric": value.rubric, "settings_revision": value.settings_revision,
            "sampling_settings": value.sampling_settings}


def audit(cases, old_results, new_results, old_instrument, new_instrument):
    if type(old_instrument) is not Instrument or type(new_instrument) is not Instrument:
        raise ValueError("exact instruments required")
    if old_instrument.rubric != new_instrument.rubric:
        raise ValueError("rubric changed; not this instrument-only audit")
    frozen = index_exact(cases, Case)
    old = index_exact(old_results, Judgment)
    new = index_exact(new_results, Judgment)
    for record in frozen.values():
        if record.rubric != old_instrument.rubric:
            raise ValueError("reference rubric mismatch")
    for results, instrument in ((old, old_instrument), (new, new_instrument)):
        for case_id, judgment in results.items():
            if judgment.instrument != instrument:
                raise ValueError("per-observation instrument mismatch")
            case = frozen.get(case_id)
            if case is None:
                raise ValueError("result has no scheduled case")
            if (case.input_hash, case.output_hash) != (
                judgment.input_hash, judgment.output_hash
            ):
                raise ValueError("frozen content changed")
    binary = {"pass", "fail"}
    transitions = {a + "->" + b: 0 for a in ("pass", "fail") for b in ("pass", "fail")}
    agreement = {side: {"matched": 0, "eligible": 0} for side in ("old", "new")}
    dispositions = {side: {key: 0 for key in (
        "pass", "fail", "tie", "abstain", "error", "missing")} for side in ("old", "new")}
    ledger = []
    shared_wrong = 0
    for case_id in sorted(frozen):
        case = frozen[case_id]
        labels = {"old": old[case_id].label if case_id in old else "missing",
                  "new": new[case_id].label if case_id in new else "missing"}
        for side, label in labels.items():
            dispositions[side][label] += 1
            if label in binary and case.human in binary:
                agreement[side]["eligible"] += 1
                agreement[side]["matched"] += label == case.human
        paired = labels["old"] in binary and labels["new"] in binary
        if paired:
            transitions[labels["old"] + "->" + labels["new"]] += 1
            shared_wrong += (labels["old"] == labels["new"] and
                             case.human in binary and labels["old"] != case.human)
        review = (not paired or labels["old"] != labels["new"] or
                  case.human == "unknown" or
                  any(label in binary and label != case.human for label in labels.values()))
        ledger.append({"case_id": case_id, "input_hash": case.input_hash,
                       "output_hash": case.output_hash, "human": case.human,
                       "slice": case.slice_name, **labels, "paired_binary": paired,
                       "old_run": old[case_id].run_id if case_id in old else None,
                       "new_run": new[case_id].run_id if case_id in new else None,
                       "old_observed_at": old[case_id].observed_at if case_id in old else None,
                       "new_observed_at": new[case_id].observed_at if case_id in new else None,
                       "review": review})
    for value in agreement.values():
        value["fraction"] = (value["matched"] / value["eligible"]
                             if value["eligible"] else None)
    paired_count = sum(transitions.values())
    changed = transitions["pass->fail"] + transitions["fail->pass"]
    return {"evidence": "synthetic frozen-output audit",
            "old_instrument": instrument_report(old_instrument),
            "new_instrument": instrument_report(new_instrument),
            "scheduled": len(frozen), "paired_binary": paired_count,
            "transitions": transitions, "changed": changed,
            "disagreement_fraction": changed / paired_count if paired_count else None,
            "human_agreement": agreement, "dispositions": dispositions,
            "shared_wrong": shared_wrong, "ledger": ledger,
            "external_task_outcome": "unmeasured"}


def expect_rejected(function):
    try:
        function()
    except ValueError:
        return
    raise AssertionError("expected rejection")


def fixture():
    cases = [Case(name, digest("input:" + name), digest("response:" + name),
                  human, "rubric-1", group)
             for name, human, group in (
                 ("a", "pass", None), ("b", "fail", "telephone"),
                 ("c", "fail", "unknown"), ("d", "unknown", "telephone"))]
    first = Instrument("judge-old-snapshot", digest("prompt-1"), "rubric-1", "settings-1",
                       "temperature=0; seed=not_requested")
    second = replace(first, model="judge-new-snapshot")
    def judgments(labels, run, instrument):
        return [Judgment(c.case_id, c.input_hash, c.output_hash, label, run, instrument,
                         "2026-11-10T00:00:00+00:00")
                for c, label in zip(cases, labels) if label is not None]
    old = judgments(("pass", "pass", "pass", "abstain"), "old-run", first)
    new = judgments(("pass", "fail", "pass", None), "new-run", second)
    return cases, old, new, first, second


def tests():
    cases, old, new, first, second = fixture()
    result = audit(cases, old, new, first, second)
    assert result["scheduled"] == 4 and result["paired_binary"] == 3
    assert result["transitions"]["pass->fail"] == 1
    assert result["shared_wrong"] == 1
    assert result["dispositions"]["new"]["missing"] == 1
    assert result["human_agreement"]["old"]["matched"] == 1
    assert result["human_agreement"]["new"]["matched"] == 2
    assert result == audit(reversed(cases), reversed(old), reversed(new), first, second)
    expect_rejected(lambda: audit(cases + [cases[0]], old, new, first, second))
    expect_rejected(lambda: audit(cases, old + [old[0]], new, first, second))
    expect_rejected(lambda: audit(cases, old, [replace(new[0], output_hash=digest("changed"))], first, second))
    expect_rejected(lambda: audit(cases, old, new, first, replace(second, rubric="rubric-2")))
    expect_rejected(lambda: replace(old[0], label=True))
    expect_rejected(lambda: replace(old[0], label="0.9"))
    expect_rejected(lambda: replace(old[0], observed_at="2026-11-10T00:00:00"))
    expect_rejected(lambda: audit(cases, old, [replace(new[0], instrument=first)], first, second))
    expect_rejected(lambda: audit(cases, old, new + [replace(new[0], case_id="unscheduled")], first, second))
    expect_rejected(lambda: audit(cases, old, new, first, object()))
    empty = audit([], [], [], first, second)
    assert empty["disagreement_fraction"] is None
    assert empty["human_agreement"]["old"]["fraction"] is None
    for label in ("tie", "error", "abstain"):
        changed = [replace(new[0], label=label)] + new[1:]
        assert audit(cases, old, changed, first, second)["paired_binary"] == 2
    snapshot = result["ledger"]
    cases.clear()
    assert len(snapshot) == 4


if __name__ == "__main__":
    tests()
    print(json.dumps(audit(*fixture()), indent=2, allow_nan=False))

settings_revision identifies the configuration record, while sampling_settings retains a declared settings snapshot as text. Every judgment binds its own immutable instrument, run and offset-bearing observation timestamp. The adapter must ensure that each supplied result truly came from that configuration; this local audit cannot authenticate provider provenance or reproduce a model execution.

Interpret the printed transition report

The synthetic collection schedules four cases. Three have binary results from both instruments. One changes from pass to fail, two remain pass, and the fourth retains an old abstention and a missing new result. The changed fraction uses the three jointly valid pairs.

One unchanged pass is wrong under its adjudicated fail label. The shared_wrong count exposes this case even though the two judges agree. The review ledger marks it for inspection rather than treating agreement as sufficient evidence of correctness.

Old and new human agreement are computed over their respective binary, adjudicated cases. The emitted matched and eligible counts permit independent recomputation. These synthetic fractions demonstrate the accounting contract; they do not establish that a real judge update improved quality.

Every ledger entry preserves case identity, input and output digests, human disposition, slice, both labels and run references. An absent result has a null run reference. That distinction allows a reviewer to tell a missing observation from an actual provider error disposition.

The report explicitly marks external task outcome unmeasured. A judge may decide that a response follows a rubric, but this program has no business-system adapter. The CRM authorization guide illustrates why an evaluated response cannot replace actual permission and operation evidence.

When storing real reports, retain the approved canonical input manifest alongside the digest references. Otherwise a later reviewer can verify that two hashes match without knowing which task boundary they represent. Identity consistency is necessary, but it does not explain the semantics of the evaluated content.

Retain errors ties and absent results

Nonbinary outcomes answer operational questions that binary scores cannot. An error can indicate a provider or parser failure. An abstention can indicate insufficient evidence. A tie may be meaningful under a comparison rubric. A missing result means the collector has no admitted observation for that scheduled identity.

The code preserves these states in per-instrument disposition counts and the all-case ledger. They do not enter the binary transition table. That exclusion is explicit and does not remove the cases from the scheduled population.

For a real collector, preserve the raw provider response under the permitted retention policy and the parser version that classified it. A parser update can change a label without any model change. Do not replace malformed responses with fail labels unless that is a separately declared operational metric.

The interrupted-workload guide explains why cancellation accounting needs a caller-owned ledger. An evaluation interrupted midway must retain unattempted and canceled work instead of reporting quality only on completed cases.

Decide retry policy before examining the results. Retrying only failures and keeping the best judgment can bias an audit. If retries are needed for operational recovery, retain attempt identities and explain which admitted attempt the comparison uses.

Do not confuse audit failure with application failure. A content-digest mismatch means this instrument-only comparison is invalid for that input bundle. It can justify stopping the audit, but it does not by itself establish that the voice application gave a wrong answer.

The typed disposition vocabulary is intentionally closed. A provider may return additional states, but the adapter must map them through an explicit documented policy before admission. An unsupported state should cause visible adapter review rather than becoming the nearest familiar label. Preserve the original response reference so a later parser revision can be evaluated without pretending that the earlier classification never happened.

Version prompts rubrics and model snapshots

A judge version is more than a model name. Retain the prompt, examples, rubric, input transformation, response schema, parser and sampling configuration. Use controlled references or hashes where appropriate, while keeping private evaluation content out of public artifacts.

OpenAI's evals guide describes configuring evaluation data and graders through the documented interface. Those interface features support an evaluation workflow; they do not make differently configured runs comparable automatically.

The instrument record uses separate fields for model, prompt digest, rubric and settings revision. The example requires a shared rubric between instruments because its claim is a comparison under one label contract. It can expose prompt or model differences, but it does not identify their individual causal contributions when both change.

Use a stable settings revision for a stable configuration and a distinct run identity for each execution. Comparing run IDs as if they were policy settings would falsely classify every repeated trial as an instrument change. Discarding run IDs would lose useful execution lineage.

The cache-key guide demonstrates the same identity discipline in another context. A key must bind the settings that actually affect meaning; a convenient display name is not enough.

When a provider exposes only a moving alias, record that limitation. A claimed snapshot without an authoritative source is weaker than an honest alias plus observation time and available response metadata. Do not invent a stable model identifier merely to make the report appear reproducible.

Observation timestamps describe when results were recorded, not a proof that provider infrastructure was unchanged between them. The code requires an explicit offset so the timestamp has a clear representation. It performs no cross-machine latency subtraction and makes no clock-synchronization claim. If timing becomes part of the evaluation, introduce a separate clock-domain contract instead of subtracting unrelated wall-clock strings.

Separate variability from instrument change

Repeated runs of the same declared judge can disagree. Before attributing every label transition to an update, understand the instrument's repeatability under the relevant task and settings. A deterministic local audit does not make the upstream model deterministic.

OpenAI's advanced-usage guide discusses nondeterminism and the documented role and limitations of reproducibility controls for the interfaces it covers. Do not generalize a seed or fingerprint contract to every model, audio path or API surface.

A repeatability study should hold the case, response, rubric and declared configuration fixed while retaining each repeated result. Compare within-version variation separately from between-version changes. The bounded program deliberately accepts one admitted judgment per instrument and case, so repeated-result analysis remains an external study.

If a collector supplies duplicate identities, the program rejects them instead of silently choosing a winner. That behavior protects the small artifact's contract. A separate repetition-aware adapter could use case and repetition identity together while preserving the same frozen content checks.

The bursty-load testing guide highlights another confounder. Under load, timeouts and incomplete requests can change the observed evaluation population. A change in the fraction of usable judge results can precede any interpretable shift in judgment quality.

Report repeatability limits candidly. A small set of matching runs does not prove that future outputs will match, and setting a low sampling parameter does not establish exact reproducibility. The release decision should use observed evidence under its declared scope.

Make release criteria explicit

Use ✓ for criteria that the local audit checks or reports directly. Use ✗ for requirements that need external evidence or another experiment. This table describes the artifact's boundary rather than a universal approval policy.

Choose release rules before seeing the update's result. A team may require review of certain critical-case disagreements, acceptable evidence coverage or a resolved parser change. Those are explicit operating decisions, not universal thresholds supplied by this article.

NIST's AI Risk Management Framework and its Generative AI Profile provide context for measurement, evaluation and risk management. They do not supply a universal judge-disagreement cutoff for a voice-agent release.

Inspect task and language slices

An aggregate transition table may hide a concentrated problem. Preserve relevant task and language slices before analysis, and inspect support counts as well as transitions. A missing slice is an evidence gap, not evidence of unchanged behavior.

Use narrowly defined cohorts. The Emirati task guide and Tamil-English name guide illustrate distinct task and representation challenges. A broad multilingual label can conceal what the judge actually received and what the rubric expected.

Do not infer speaker identity or capability from a slice label. Metadata should come from the approved evaluation design, and missing values should remain typed missing values. The example keeps None distinct from the admitted string unknown in the review ledger.

Examine transitions that affect important literals or repairs. A judge that prefers smooth summaries might reward an answer that silently substitutes a name. The unfamiliar-surname guide shows why clarification can be a correct outcome even when it looks less fluent.

For speech-sensitive rubrics, state whether the judge has audio, transcript or both. Text alone cannot establish pronunciation, timing or audibility. The brand-pronunciation guide separates spelling evidence from what was actually spoken.

The local artifact retains a slice label but does not automatically estimate slice-specific uncertainty. A reviewer can group its ledger under a declared analysis plan. Avoid producing a dense grid of tiny fractions and presenting them as stable comparative results.

Adjudicate disagreements without rewriting history

Build a review queue from changed labels, nonbinary dispositions, unknown reference labels and disagreements with known references. The emitted ledger's review flag is deliberately inclusive. It prioritizes evidence review; it does not assign blame to either judge.

For each reviewed case, retain the original judgments and review references. An adjudicator may decide the old judge was right, the new judge was right, both were wrong or the rubric was insufficient. Preserve that decision as a new adjudication revision.

If the rubric changes, do not overwrite its old version and recompute a supposedly unchanged comparison. The new label contract requires its own evidence record. The program rejects mismatched rubric versions for this bounded instrument-only audit.

The confirmation-repair guide offers a useful rubric stress case. An answer that treats a qualified correction as final consent may be wrong even when a shallow conversational judge finds it helpful.

Keep review blind to the extent practical for the evaluation design. Revealing which result came from the proposed update can influence adjudication. This article does not implement blinding or reviewer assignment; those obligations belong in the external review process.

After review, report what changed in the evidence, including resolved unknowns and revised reference labels. A better agreement fraction after adjudication may reflect improved labels rather than improved judge behavior. Preserve the prior report so that distinction remains visible.

Do not discard unchanged cases from the review evidence. They establish the paired denominator and can reveal shared failures that a disagreement-only queue misses. A practical review policy can sample unchanged cases under a declared design while inspecting all predefined critical transitions. The sampling policy itself needs to be recorded, because a reviewed subset cannot silently stand in for adjudication of the whole collection.

Avoid calibration claims from agreement alone

Binary agreement measures whether labels match under a defined denominator. It does not establish that a numerical judge confidence is calibrated. A pass label and a probability forecast are different evaluation outputs with different validation needs.

The confidence calibration worksheet defines fixed-bin comparisons and accepted-risk accounting. Do not take a judge's self-reported certainty and treat it as the probability that its label is correct without supporting evidence.

Scikit-learn's calibration guide explains the relationship between probability forecasts and outcome frequencies. Its Brier-score documentation also makes score conventions explicit. Neither property follows from old-new agreement alone.

Shared errors are particularly instructive. Two instruments can use similar shortcuts or miss the same requirement. Their agreement can increase while agreement with independently adjudicated task evidence decreases. Keep both analyses and the case-level review evidence.

If the judge emits ordinal or continuous scores, declare a separate contract for score scale, missingness and any conversion to binary labels. An arbitrary text score such as a numeric-looking string is rejected here because this artifact expects typed label dispositions.

A genuine developer question about inconsistent custom evaluation results appears in DeepEval's discussion. It motivates preserving prompts, reference evidence and outcomes, but the discussion does not establish the behavior of a current model or supply a universal remedy.

When a numeric judge score is part of another workflow, retain the original score and the policy that maps it to a label. A threshold revision changes that policy even when the judge's raw output stays identical. Comparing only the final pass/fail field would otherwise make a threshold adjustment resemble a model update. This artifact avoids that ambiguity by accepting only an already declared typed label contract.

Refresh the reference collection deliberately

A frozen collection is valuable for comparability, but it can become narrow or repeatedly optimized against. Maintain a stable reference set for instrument comparison and a separately governed process for adding new task evidence. Do not quietly replace difficult cases between runs.

The regional dataset guide explains provenance and consent obligations for real speech. A reference collection should retain permitted use, source lineage and relevant collection conditions rather than treating public availability as unrestricted evaluation permission.

When adding cases, keep the previous cohort identifiable. Report paired results on the shared cohort and separate evidence for newly added cases. A changed total pass rate over a changed workload is not an isolated measure of judge drift.

Protect the holdout from repeated prompt tuning. If engineers inspect failures and alter the judge prompt until the same collection passes, the collection has influenced selection. Retain a separate evaluation boundary for claims that require untouched evidence.

Record why a case was retired. It may contain invalid evidence, lack permitted retention or represent an obsolete task contract. Retirement can be appropriate, but deleting only embarrassing failures produces a misleading historical trend.

Separate collection refresh from rubric repair where possible. Adding a new channel and changing what counts as a successful answer at the same time can make historical comparisons hard to interpret. If the combined change is necessary, preserve the old cohort, old rubric and explicit mapping to the new study. A truthful discontinuity in the report is preferable to a continuous-looking line assembled from incompatible measurements.

The accent-cohort guide discusses leakage and scope. Similar care applies to evaluator examples: a collection dominated by familiar templates may measure recognition of those templates rather than general judgment of the task.

Connect the audit to a release investigation

When the dashboard changes, first establish whether the evaluated objects and workload stayed fixed. Then verify the instrument provenance and result coverage. Only after those boundaries are clear should the team interpret disagreement patterns or investigate application changes.

If outputs are frozen and the rubric is unchanged, a judge-label shift is evidence of measuring-instrument behavior. It can reveal a prior blind spot, but it cannot show that an unchanged application deployment regressed during the update.

If application outputs changed too, preserve both changes and run a controlled comparison where feasible. The transport diagnosis guide demonstrates why a plausible symptom is only a starting hypothesis until the relevant observation boundary is isolated.

Keep incident recovery separate from causal certainty. A team may choose to roll back a judge configuration to restore a stable release gate while adjudication proceeds. That operational decision should be recorded as restoring comparability, not as proof that the newer model is universally worse.

For a voice workflow, supplement rubric judgments with deterministic field checks and independently observed task outcomes. The leading-zero capture guide shows a concrete invariant that a free-form judge should not be asked to infer from fluency alone.

Keep the release decision and the diagnostic conclusion in separate fields. The team may pause deployment because reference coverage is insufficient even while the application's observed behavior is unchanged. It may also deploy a well-supported judge improvement while retaining unresolved low-priority questions. Recording the reason makes later review possible without rewriting operational caution into a universal model-quality claim.

Frequently asked questions

What is evaluator drift?

It is a change in how the evaluation process maps comparable objects to judgments. Model updates, prompts, rubrics, parsers and input transformations can all contribute. A changed application score is an observation requiring investigation, not automatic proof of evaluator drift.

How can a judge update change a quality score?

The new instrument may interpret the same rubric differently or attend to different response features. Compare frozen outputs and retain case-level transitions. Aggregate pass-rate differences alone cannot identify whether the new interpretation is more correct.

What should a frozen reference set contain?

Keep stable case identities, canonical input and response provenance, rubric version, permitted evidence references, task slices and adjudicated labels with unknowns. The local fixture uses synthetic text digests and typed labels to demonstrate that contract.

Should the old judge be treated as ground truth?

No. It is a prior instrument that may share errors with the new one. Independent adjudicated labels can expose shared mistakes. Preserve unresolved human evidence as unknown rather than using either judge to manufacture certainty.

Why should agent outputs remain frozen?

Otherwise the comparison changes both the object and the measuring instrument. A label transition could reflect either. Frozen outputs allow a narrower instrument comparison, while a separate experiment can evaluate changed application behavior.

How do rubric changes differ from model changes?

A rubric revision changes the declared evaluation target; a model revision changes the instrument applying that target. Both need versioning. This bounded audit rejects differing rubric versions rather than pretending they are the same comparison.

Which denominator measures paired disagreement?

Jointly valid binary pairs. The program divides changed binary labels by that count and reports the four transition cells. Scheduled cases without valid pairs remain in the disposition counts and ledger so evidence coverage stays visible.

How should missing judge results be counted?

Retain them as missing observations for scheduled identities, separate from errors, abstentions and failures. They do not enter binary transitions. A missing result can indicate incomplete collection and must not become a favorable default.

Can temperature zero guarantee repeatability?

Do not assume it. Use the documented behavior of the specific interface and measure repeated runs under declared conditions. The local audit is deterministic over supplied records, but it makes no determinism claim about an upstream model.

What provenance should be retained?

Preserve model identity, prompt and rubric versions, settings, input and output identity, parser version and run references. Keep stable configuration separate from each execution's identity. Record limitations when a provider exposes only a moving alias.

How should ties and abstentions be represented?

As explicit typed dispositions under a documented rubric. They remain in the all-case ledger and per-instrument counts. Do not coerce them to binary labels merely because a dashboard expects a pass-rate field.

Why should joins use identifiers and hashes?

Identifiers align the intended cases; content hashes bind those identities to declared frozen objects. Reordered result collections then remain comparable. A mismatch is rejected because the instrument-only comparison no longer evaluates the same content.

How do human disagreements affect the reference set?

Unresolved labels should remain unknown, with review provenance retained externally. If adjudication changes a reference, create a new revision. Rewriting the original label without history can make the judge appear to improve retroactively.

Should a judge score its own model family?

That choice needs evaluation for the actual rubric and potential shared failure modes. This audit does not establish independence between model families. Use adjudicated evidence and suitable comparisons instead of assuming either automatic bias or automatic neutrality.

How should multilingual slices be reviewed?

Declare language, task and representation boundaries, retain support counts and use appropriately qualified adjudicators. A transcript-only judge cannot verify acoustic pronunciation. Missing or tiny slices should remain explicit evidence limitations rather than broad multilingual claims.

Can agreement hide shared judge errors?

Yes. The fixture contains a case both judges pass while its adjudicated label is fail. The shared-wrong count and review ledger expose that pattern. Old-new agreement is therefore distinct from correctness against independent evidence.

When should a judge update block a release?

Apply predefined task-specific criteria for unresolved critical disagreements, coverage and provenance. This article supplies no universal threshold. A blocked comparison may mean the measuring process is invalid rather than that the application itself is worse.

How should a reference set be refreshed?

Version additions, retirements and label revisions, retain the shared comparison cohort and preserve permitted source lineage. Keep newly added cases separate when comparing old and new results so workload changes do not masquerade as instrument drift.

Can unpaired scores establish judge drift?

They can motivate investigation, but changed case composition and result coverage remain confounders. A paired frozen-output audit supports a narrower comparison. Keep unmatched observations rather than discarding them to manufacture a balanced-looking table.

What does this audit fail to prove?

It does not prove correct human labels, authentic provider provenance, model repeatability, acoustic quality or completed tasks. It validates declared records and prints reproducible disagreement accounting. External evidence remains necessary for those broader claims.

Start with one frozen comparison

Select a small approved reference collection and freeze its exact input and agent-output representation. Run both declared instruments, retain every scheduled case and execute the audit's negative fixtures before interpreting the transition report.

Review changed labels, shared errors and missing evidence with the people who own the task rubric. Preserve the original judgments and record adjudication revisions separately. The immediate result should be a defensible instrument comparison and a concrete review queue, with application quality claims reserved for evidence that actually measures them.