Voice agent regression testing: a real-call harness

TL;DR: Build voice agent regression testing in layers: deterministic state and tool checks, versioned audio replay, then authorized calls through the real media route. Assert business effects and forbidden behavior, retain failure evidence, and repeat stochastic conversations without turning reruns into a way to hide defects.

A prompt edit passes a text conversation test. On the next phone call, the agent speaks over a correction, waits indefinitely for a tool, and creates the wrong appointment.

The text test was useful, but it exercised a smaller contract. A release harness needs to identify which behaviors can be proven locally and which require speech, playback, transport, or an external business system.

As of November 9, 2026, LiveKit's testing overview distinguishes turn-level unit checks from multi-turn simulations and audio runs. That separation is a useful starting point for a vendor-neutral suite, regardless of the framework you use.

The harness design below is a proposal you can adapt. It includes runnable local assertion code, but it does not claim a live-call experiment, a measured pass rate, or a public regression SDK for 60db Agent.

Start with a latency measurement contract so your audio tests have a defined beginning and end. Then decide which failures should block a release even when the generated words sound reasonable.

Table of contents

1. Define a release contract around behavior
2. Separate local checks from real-call evidence
3. Version fixtures and scenario timing
4. Build the smallest useful test runner
5. Assert effects and forbidden behavior
6. Inject silence, interruptions, and tool failures
7. Evaluate stochastic responses with calibrated judgments
8. Run safely in CI and through real routes
9. Frequently asked questions
10. Turn failures into durable release evidence

Define a release contract around behavior

A voice agent regression test specifies an input, an observable result, and a forbidden effect. The result should describe the task or protocol behavior you need, rather than require an exact sentence when equivalent wording is acceptable.

For an appointment workflow, the result can be a booking with the confirmed date and service. A forbidden effect can be a second booking, a booking before confirmation, or an appointment created from a superseded request.

The same scenario can carry a conversational assertion. If the caller changes the date during an answer, the agent must stop obsolete speech and continue using the corrected date.

Those assertions belong to different owners. The business system proves what was booked, while the media trace proves what played and the conversation evidence explains how the correction was handled.

Do not use the final transcript as a substitute for every layer. It can omit audio glitches, buffering, speech that played after cancellation, and tool effects that never appear in the assistant's final wording.

Write the contract before generating examples. Otherwise it is easy to select cases that the existing system already handles and mistake coverage of those cases for coverage of the actual task.

A useful scenario contains a purpose, preconditions, fixture references, caller actions, expected effects, forbidden effects, deadlines, and cleanup ownership. Keep fields only when they change execution or interpretation.

For example, a delayed inventory scenario can require the agent to avoid inventing stock while the read is unresolved. The expected continuation can be a supported fallback or a later grounded answer, depending on the workflow you have approved.

The workflow enforcement guide explains why required authorization and confirmation steps must survive prompt variation. The test should inspect the enforcing boundary rather than only checking whether the model recites the right policy.

Distinguish a regression from an invalid test

An agent regression means a valid scenario violates its acceptance contract. An invalid test means the harness failed to establish the conditions or collect the evidence needed to judge the agent.

A microphone permission refusal can invalidate a browser audio test. A tool returning an intentional timeout is a valid scenario when the test asked for that fault.

Keep the distinction explicit in the result record. If every harness error is reported as an agent failure, the suite becomes noisy; if every agent failure is dismissed as infrastructure, it becomes useless.

Define who can reclassify a result and what evidence they must preserve. Reclassification should change the report transparently, not delete the original failed attempt.

Use a small number of terminal outcomes, such as pass, behavioral failure, setup failure, and insufficient evidence. Attach a specific reason code and artifact references rather than growing an unstructured list of vague statuses.

This record later supports outcome analysis. It also lets a release owner distinguish a candidate defect from an unavailable test route without losing either fact.

Identify the change surface

A prompt change can alter tool selection, response length, disclosure delivery, and turn timing. A synthesis change can alter audio onset, pronunciation, and interruption boundaries while leaving text-level behavior intact.

A carrier or transport change can affect connection setup, audio format, packet timing, and teardown. Choose tests from the affected contracts, then keep a small cross-cutting suite for failures that span several layers.

Do not rerun every expensive phone scenario for a typo in an internal report. Do not run only a text snapshot when the change affects media buffering.

The suite is a map of evidence, not a contest to accumulate test counts. A few scenarios with clear failure meaning can be more useful than many conversations graded by a broad impression of quality.

Separate local checks from real-call evidence

Start with deterministic logic tests for state transitions, argument validation, authorization, deduplication, and event handling. These checks should run without depending on a live model's choice of words.

Next use text-level agent tests for tool selection and grounded continuation. They exercise model behavior while leaving audio and transport outside their scope.

Audio replay adds recognition, turn boundaries, synthesis, playback, and timing. An authorized call through the intended route then adds the carrier or gateway conditions your product uses.

This layering is not a claim that one level replaces another. Each level supplies a different observation boundary and a different cost of investigation when it fails.

A check mark describes coverage available when that assertion is instrumented. Merely placing a phone call does not establish that business arguments or resulting records were inspected.

LiveKit's unit-testing guide provides one framework example of explicit message and tool assertions. Verify the current installed API before adopting framework snippets; the local examples here use only Python's standard library.

Local assertions, audio replay, and authorized calls prove different boundaries.

Use doubles at the correct boundary

A controlled tool double can return a timeout, conflict, or delayed response on demand. It is useful because a production booking service should not need to malfunction just to test the agent's error handling.

Keep the double faithful to the real contract. Validate its response shape against the supported schema, and add a smaller contract test against a sandbox or authorized service environment.

A fake service that always returns success hides the failure paths the harness exists to explore. A double should model the specific behavior under test, including errors and unknown outcomes.

Do not describe results from doubles as live business-system success. Label the test layer and record which external boundaries were simulated.

Keep a route-specific smoke suite

A route-specific smoke test checks whether the deployed path works at all under the tested conditions. Include connection, bidirectional audio, one grounded tool operation where permitted, hangup, and settled cleanup.

For a SIP route, inspect the distinction between signaling and media described in the SIP troubleshooting guide. A successful connection event is insufficient evidence of understandable audio in both directions.

For browser media, test permissions and tab closure. For a stream route, test audio framing and the application's forwarding queues.

The transport comparison identifies these ownership differences. Keep a common scenario intent while letting each adapter collect the route-specific evidence it can support.

Version fixtures and scenario timing

A fixture is more than an audio file. It includes provenance, expected language, format, channel conditions, relevant speech boundaries, and the behavior the scenario is meant to test.

Store a digest of the audio bytes and a fixture version. If the recording changes, the next run must be able to establish that it used different input.

Use synthetic test content and consented speakers. Avoid importing production customer calls into a general CI artifact store just because those calls contain realistic pauses.

A real defect can inspire a new synthetic fixture. Preserve the failure class, such as a date correction during playback, without retaining the original person's voice or private details.

For each clip, record sample rate, channel count, sample width, encoding, frame count, and any conversion applied before delivery. A mislabeled sample rate can alter pitch, duration, or recognition and make unrelated failures appear at once.

Python's wave module can inspect supported WAV files using the standard library. Check your Python version and the actual format support rather than assuming every file ending in .wav contains the same encoding.

Choose timing relative to observable events

A fixed delay after test start is fragile. The greeting may begin later under load, so a caller clip scheduled at that delay can interrupt silence in one run and active speech in another.

Prefer event-relative actions. For an interruption scenario, wait until assistant playback is established, then inject the caller correction after a documented offset.

For a slow-tool scenario, hold the tool response behind a barrier and release it after the caller changes the request. That produces a reproducible race without depending on a real network being slow on demand.

The interruption state machine supplies the ownership invariants for those schedules. Keep the test's trigger and the expected stop boundary in the artifact record.

Avoid triggering exclusively from a partial transcript substring. Transcription can revise text, and a wording change can prevent the test from reaching its intended condition.

Use protocol or playback events where available. When text is the only trigger surface, label the limitation and fail the scenario as insufficient evidence if the trigger never occurs.

Preserve realistic silence

Silence is part of a conversation. A person may pause to find a document, decide between options, or correct a sentence before finishing it.

Do not trim every clip tightly around speech if endpointing behavior matters. Retain annotated pauses and include scenarios where the caller resumes after the agent begins answering.

The VAD versus turn-detection guide explains why the same acoustic silence can have different conversational meanings. Your fixture labels should describe the intended continuation rather than merely the duration of quiet audio.

A long silent clip is not a complete silence test. The expected agent policy must say whether to wait, clarify, offer help, or end after a supported timeout.

Include language and entity cohorts

Names, account references, dates, quantities, and addresses deserve explicit checks. A response can sound fluent while using one wrong digit in the business operation.

The multilingual benchmark separates transcript error from task completion. Apply that distinction even in a mostly single-language suite when callers mix languages or pronounce entities differently.

Do not publish an overall pass rate that hides a failing language cohort. Report the corpus distribution and per-cohort outcomes with enough sample context to judge the evidence.

Build the smallest useful test runner

The runner needs to establish preconditions, drive the scenario, collect evidence, evaluate assertions, and clean up what it created. It does not need to become a second voice platform.

Keep provider adapters narrow. They should create or connect the supported test session, send media in the documented format, expose relevant events, and close the session.

Put business assertions in a separate evaluator that reads the test ledger and sandbox records. This keeps a transport adapter from deciding whether an appointment or payment was correct.

A scenario engine can start as an ordinary function with explicit waits and timeouts. Add a declarative language only when repeated scenarios reveal a real need for it.

flowchart TD
    A[Versioned scenario] -->|prepare| B[Fixture and sandbox setup]
    B -->|start supported route| C[Call adapter]
    C -->|audio and events| D[Evidence ledger]
    C -->|tool operations| E[Test business system]
    D -->|observations| F[Assertions]
    E -->|verified effects| F
    F -->|record outcome| G[Release report]
    C -->|always close| H[Cleanup]
    H -->|verify terminal resources| G

This diagram separates evidence collection from judgment. A failed call still reaches cleanup, and a cleanup failure still appears in the final report.

A runnable event assertion

The example below is a vendor-neutral test for one invariant: no old-response frame may be admitted after that response is invalidated. Its event names are a local test schema, not a claimed API from any vendor.

import unittest


def stale_output_violations(events):
    invalid = set()
    violations = []
    for index, event in enumerate(events):
        kind = event["kind"]
        response = event["response"]
        if kind == "invalidate":
            invalid.add(response)
        elif kind == "admit_audio" and response in invalid:
            violations.append(index)
    return violations


class InterruptionEvidenceTests(unittest.TestCase):
    def test_rejected_late_audio_is_safe(self):
        events = [
            {"kind": "invalidate", "response": "a"},
            {"kind": "reject_audio", "response": "a"},
            {"kind": "admit_audio", "response": "b"},
        ]
        self.assertEqual(stale_output_violations(events), [])

    def test_admitted_late_audio_fails(self):
        events = [
            {"kind": "invalidate", "response": "a"},
            {"kind": "admit_audio", "response": "a"},
        ]
        self.assertEqual(stale_output_violations(events), [1])


if __name__ == "__main__":
    unittest.main()

Save the example as test_interruption_evidence.py and run python -m unittest test_interruption_evidence.py. Both tests should pass because the second test confirms that the evaluator detects an intentionally bad event sequence.

This is a test of the evaluator, not evidence that your agent passed the invariant. The adapter must produce ordered, complete events from the actual playback-admission boundary before the same function can evaluate a real run.

Schema validation belongs before this function. Reject missing identifiers, unsupported event kinds, duplicate sequence numbers, and incomplete streams rather than quietly treating unrecognized evidence as safe behavior.

Python's unittest documentation covers the runner and assertion methods used here. No extra dependency or hidden hosted test service is required for this local check.

Give every wait a reason and a deadline

A scenario waits for events because those events establish preconditions. Each wait should name the expected event, the deadline, and the failure meaning if it never arrives.

A connection timeout differs from an agent that never answers after a valid turn. Preserve that distinction instead of throwing the same generic timeout exception for both.

Bound the whole run as well as individual waits. A chain of retries and cleanup waits can otherwise exceed the time budget even when every local timeout looks reasonable.

Use a monotonic clock for local deadlines. Cross-process timestamps can provide trace context, but they should not define a local wait unless the clock relationship is established.

Assert effects and forbidden behavior

The strongest assertion is often outside the transcript. If the caller requested a booking, inspect the sandbox booking record and verify its confirmed arguments and operation identity.

A spoken promise is evidence of what the agent said. It does not prove the service accepted the operation, and it does not prove that the accepted operation used the right date or account.

Assert the count as well as the content. A correct record accompanied by an unintended duplicate is a failed scenario.

For a refusal or a missing confirmation, assert the absence of the write. That negative assertion catches systems that say they will wait while a background callback proceeds anyway.

Keep authorization in the tested path

Do not bypass the real policy boundary to make the harness convenient. If production tools require signed requests and argument validation, the appropriate contract tests should exercise those checks.

Use sandbox credentials and permitted test identities. The harness should never need to impersonate a real customer or weaken a production authorization rule.

The speculative tool guide adds tests for a resolved tool whose actual capability differs from the initial label. This catches a dangerous class of errors that transcript-only checks cannot see.

A test double can record proposed arguments before dispatch. A separate service contract check then verifies that the actual backend rejects invalid, unauthorized, stale, or duplicate operations according to its contract.

Test partial delivery

An agent can generate a required statement without delivering it. Interrupt the statement and inspect whether the workflow still considers the requirement incomplete.

Then continue the conversation and verify the approved recovery behavior. Repeating a short required statement may be correct, while repeating a long unrelated introduction may indicate poor context repair.

Do not equate playback completion with user agreement. A confirmation check should rely on the policy's actual evidence for agreement and the specific arguments being confirmed.

Distinguish transport health from task health

A call can connect, exchange audio, and end normally while failing the task. Conversely, a task can complete before an abrupt disconnect makes the conversational finish unsatisfactory.

Keep both outcomes. The successful-call cost model needs the business denominator, while a reliability report still needs the transport failure and any recovery work.

This separation avoids a misleading green suite where every call has status completed but none of the intended business records were checked.

Inject silence, interruptions, and tool failures

Fault injection should target a named assumption. Delay a read to test stale-result handling, drop an acknowledgment to test unknown operation state, or interrupt queued playback to test output invalidation.

A random collection of failures can reveal defects, but a release regression case needs a reproducible schedule and an acceptance contract. Keep exploratory fault testing and fixed regression checks as related but distinct activities.

Barge-in scenarios

Interrupt during the greeting, before first response audio, in the middle of a long answer, and while a tool result is pending. Each case exercises a different boundary.

Assert that old speech stops at the defined observation boundary. Then assert that the caller's correction changes the active task and that the replacement response does not resume obsolete content.

Include duplicate and delayed speech-start events in local tests. They find controllers that accidentally cancel the new response when an old detector event arrives late.

Silence scenarios

Test a thinking pause within an unfinished request, silence after a complete request, and silence after the agent asks a question. The expected behavior should depend on context rather than use one rule for all quiet periods.

Add background speech and short acknowledgments to expose false stops. A silence threshold tuned only against clean clips can pass replay tests while performing poorly in a shared room.

Retain the exact fixture and timing labels. Without them, a failed silence test can become an argument about whether the caller had finished rather than a reproducible observation.

Tool failure scenarios

Return an explicit error, a timeout, malformed data, an authorization refusal, and a result that arrives after interruption. These are different failures and should have different expected continuation where the workflow requires it.

For writes, test the unknown-outcome case: the service may have accepted the operation before the connection failed. Require reconciliation or supported idempotency rather than an automatic second operation with a new identity.

If the agent offers a transfer after repeated failure, exercise both successful and unavailable handoff paths. The warm handoff guide describes the context and lifecycle assertions needed there.

Disconnect and cleanup scenarios

End the call while synthesis is active and while a tool is waiting. Assert that media stops, call-bound tasks settle, and only approved post-call work remains.

The shutdown guide separates completion events from later artifacts such as recording availability. Your harness should not fail a valid delayed recording merely because it expected every artifact at hangup.

Also verify that the test runner itself releases sockets, browser contexts, temporary files, and created sandbox records. A suite that leaks resources can contaminate later trials and make failures appear random.

Evaluate stochastic responses with calibrated judgments

Model-backed conversations can vary across runs even when the fixture and configuration stay fixed. Exact string equality is appropriate for exact entities or protocol fields, but it is often too narrow for a natural-language answer.

Use deterministic assertions wherever the contract permits them. Compare structured tool arguments, verified operation state, forbidden-action counts, and defined media events directly.

Use a semantic judge only for bounded questions that require interpretation. Examples include whether the response answered the caller's question or explained a supported fallback without making an unsupported promise.

OpenAI's evaluation guidance recommends task-specific evaluations and continuous evaluation practices. Treat any automated judge as another component that needs validation against examples your team has reviewed.

Calibrate the judge

Create a small set of reviewed pass, fail, and ambiguous responses. Include cases where fluent wording hides a wrong entity, an unsupported promise, or a missing limitation.

Compare the judge's decisions with the reviewed labels. Inspect disagreements and refine the question or rubric before using the score to block releases.

Keep the judge prompt, model, and version in the run record. A changed evaluator can shift scores without any change in the agent under test.

For high-consequence business actions, the judge should not replace the deterministic policy assertion. It can assess the explanation while the business system proves whether the action was permitted and correct.

Repeat without laundering failures

Repeat stochastic scenarios according to a policy chosen before the run. Keep every attempt and report the observed failure fraction with sample context.

A retry that happens to pass does not erase the failed attempt. Classify whether the failure exposed a deterministic defect, model variation, a setup issue, or insufficient evidence.

Do not keep rerunning until the suite turns green. That turns the release criterion into a measure of patience and obscures intermittent failures that callers will still encounter.

Compare baseline and candidate using the same repeat policy. A candidate tested once and a baseline tested until a favorable run are not comparable evidence.

Retain every stochastic attempt; a later passing run adds evidence without erasing the failure.

Keep aggregate scores subordinate to invariants

A single quality score can help rank exploratory variants. It should not average away an unauthorized write, a privacy violation, or stale audio after hangup.

Keep blocking invariants visible beside summary metrics. A release owner should be able to see which hard checks passed and where a softer conversational trade-off changed.

Report per-scenario and per-cohort outcomes. The overall number can hide a new failure concentrated in one language, route, or interruption pattern.

Run safely in CI and through real routes

Put cheap deterministic checks on ordinary code changes. Schedule more expensive audio and authorized phone runs according to the change surface and the cost of missing a defect.

A nightly suite can broaden coverage, but a release that changes the media adapter should still run relevant media checks before deployment. Frequency follows risk and evidence needs.

Isolate test resources from production data. Use a dedicated workspace or sandbox where supported, with least-privilege credentials and destinations approved for automated calling.

Store secrets in the CI platform's supported secret store. Redact authorization headers, one-time stream URLs, webhook secrets, and tool payload fields that can expose private data.

The data-boundary comparison explains why test artifacts deserve the same access and retention discipline as other application data. A failed recording can be more sensitive than a passing metric.

Qualify 60db Agent through public surfaces

60db Agent's public Agents documentation describes browser test calls and WebSocket stream calls. Use those supported contracts where enabled for the workspace, and verify the current limits before scheduling automated runs.

The documentation does not establish a built-in regression harness for every scenario described here. Your runner must supply fixture playback, evidence capture, business assertions, and cleanup unless a current public feature explicitly owns them.

A browser test establishes behavior on that test path. An authorized outbound call establishes behavior on the tested phone route, while inbound availability still needs workspace and environment qualification.

Do not claim that a text-only API test passed barge-in or audio quality. State the tested boundary in the report and leave untested media behavior marked unverified.

Use the call-trace design to keep the evidence ledger consistent across local tests and deployed calls. Matching event names and ownership identifiers makes a failed CI scenario easier to compare with an operational incident.

Preserve reproducibility without overcollecting

A useful failure bundle includes scenario version, fixture digest, agent configuration version, route, attempt status, event ledger, verified business effects, and redacted diagnostic references. Add audio only under the test's consent and retention policy.

Use relative paths or artifact identifiers rather than embedding secret-bearing URLs in the report. Access-controlled storage should enforce the same scope as the source test data.

Record tool and provider versions you can observe. For a managed service with undisclosed internals, preserve the date and public configuration while marking unavailable details unknown.

Record the first divergence

When several assertions fail in one call, identify the first event where observed behavior departs from the scenario contract. A lost correction can explain wrong tool arguments, an unsupported answer, and an unsuccessful outcome without requiring four unrelated fixes.

Preserve later failures as consequences rather than deleting them. The release report can name the earliest confirmed defect and link the downstream evidence, helping reviewers distinguish a shared cause from independent regressions.

Compare the failing event sequence with a passing baseline under the same fixture and configuration policy. If the two runs diverge before the intended fault is injected, investigate setup or uncontrolled variation before attributing the outcome to the fault.

Identify the earliest confirmed departure while preserving the later consequences in the failure bundle.

Close every resource you create

Use cleanup that runs after pass, fail, timeout, and interruption. Track created call identifiers and browser sessions explicitly so the runner knows what it owns.

Verify terminal state after requesting hangup. A successful request to close is evidence of intent, while the subsequent terminal event establishes the observed outcome.

If cleanup fails, preserve that result and stop adding load until the leak is understood. Do not solve a busy port by launching another server on the next number and forgetting the first one.

Keep a bounded artifact retention policy and a process for promoting failure cases into the maintained corpus. This prevents both resource leaks and a permanent archive of unnecessary recordings.

Frequently asked questions

What is voice agent regression testing?

It is a repeatable check that a voice-agent change preserves required behavior across defined scenarios. A useful suite covers business effects, conversational state, media behavior, and the supported route instead of relying only on transcript wording.

Are text tests enough for a voice agent?

Text tests are useful for model and tool behavior, but they do not exercise recognition, playback, endpointing, or transport. Use audio and route-specific tests when those contracts are affected by the change.

Do I need real phone calls on every commit?

Not for every change. Run cheap deterministic checks frequently, then choose audio and authorized phone tests according to the affected surface and the consequence of a missed defect.

What should a test scenario contain?

Include its purpose, preconditions, fixture version, caller actions, expected effects, forbidden effects, deadlines, and cleanup owner. Keep fields concrete enough that another engineer can reproduce the scenario without interpreting an informal story.

How do I test interruption handling?

Trigger caller speech relative to established assistant playback, then check stopping behavior and continuation. Add local controlled schedules where old audio or tool results arrive after response invalidation.

How do I test a slow tool reliably?

Use a controlled tool boundary that holds the response until the scenario releases it. That creates the intended race without depending on an external service to become slow at the right moment.

Should I compare exact response strings?

Use exact comparisons for entities, protocol fields, and outputs that require an exact value. For equivalent natural-language answers, use a bounded semantic rubric while keeping business and safety assertions deterministic.

What is a negative assertion?

It checks that a forbidden effect did not occur. Examples include no duplicate booking, no write before confirmation, no stale audio after invalidation, and no playback restart after hangup.

Can I use a model to grade conversations?

Yes, for bounded interpretive questions after calibrating it against reviewed examples. Keep its configuration versioned and do not let a broad quality score replace deterministic authorization or business-result checks.

How should I handle flaky tests?

Preserve every attempt and classify the cause using evidence. A rerun can add information, but a passing retry should not erase a real intermittent failure from the release report.

What audio fixtures should I start with?

Start with a normal request, a correction during playback, a thinking pause, a required tool lookup, and a supported error path. Use consented or synthetic content and record format, timing, and expected behavior.

Can production incidents become test fixtures?

Yes, by recreating the failure class with synthetic content where possible. Avoid copying private recordings or transcripts into CI simply because they reproduce the problem conveniently.

Why does fixture sample rate matter?

An incorrect sample rate changes how bytes map to time and sound. Record source and delivered formats, and validate conversions so recognition or timing defects are not caused by a broken test input.

How do I prove a booking succeeded?

Inspect the supported business-system record and verify its confirmed arguments and operation identity. The agent saying that the booking succeeded is a conversational claim, not proof of the external effect.

What if the call succeeds but the recording is missing?

Distinguish task outcome from missing evidence. A recording may have a later availability contract, so wait according to that contract and mark insufficient evidence when the required artifact never arrives.

How should I test multiple languages?

Use labeled language and entity cohorts with task-level assertions. Report cohort outcomes separately so a strong aggregate result cannot hide a language-specific regression.

Can I test 60db Agent without a phone line?

The public documentation describes browser testing and stream calls where enabled for the workspace. Verify current access and limits, then label the result according to the path you exercised.

Does a successful browser test prove SIP works?

No. A browser test and a SIP route exercise different media and connection boundaries. Keep a small authorized smoke suite for each route your product promises to support.

What belongs in a failure artifact?

Preserve the scenario and configuration versions, fixture digest, event ledger, assertion results, business effects, and redacted diagnostic references. Retain audio only when consent, access, and retention policies allow it.

What is the smallest useful first harness?

Use one versioned scenario, one supported adapter, a deterministic business assertion, and cleanup that runs on every outcome. Add layers when a real failure requires evidence the current harness cannot collect.

Turn failures into durable release evidence

A real-call regression harness earns its place by explaining failures. Its value comes from a stable contract, meaningful fixtures, and assertions that distinguish what the agent said from what the system did.

Build the local invariant checks first. Add audio replay for speech behavior, then qualify the real route with authorized calls and verified cleanup.

Keep stochastic evaluation honest by preserving all attempts and calibrating semantic judgments. Keep hard invariants visible so an attractive average score cannot conceal an unauthorized effect or an interruption race.

Your first deliverable can be small: one synthetic correction scenario, one delayed read, one forbidden-effect assertion, and one failure bundle. If 60db Agent is the system under test, use its documented supported surfaces and add your own runner around the evidence those surfaces expose.