Rolling deployments without truncating active conversations

TL;DR: Close application admission atomically with the start of draining, then let existing conversations finish within an explicit budget. Readiness changes only influence routing; a retained per-session ledger must still distinguish completed, canceled, never-started and unknown work when cleanup or the process ends.

---

A rolling update reports healthy replicas while a caller's existing conversation stops mid-sentence. The replacement worker is ready, but the old worker owned a long-lived media session and a submitted business operation. Infrastructure availability alone does not describe what happened to either one.

Start with separate contracts for new admission, existing work and cleanup evidence. A worker can stop accepting new sessions while continuing conversations it already owns. The shutdown semantics guide explains why ending local tasks must not erase unknown external outcomes.

Kubernetes Pod lifecycle documentation describes termination hooks, signals, the grace period and eventual forced termination. Those mechanisms provide an execution window, not proof that application drain or every carrier leg completed. Your protocol must decide what to admit and what evidence remains before that window closes.

The runnable program below models a single event loop, one drain controller and cooperative synthetic sessions. It tests local admission and ledger behavior, including cancellation before a session coroutine starts. It does not perform a Kubernetes rollout, reconnect a caller or establish zero-loss behavior for 60db Agent on a live carrier.

Table of contents

1. Separate readiness from application admission
2. Register ownership at one admission boundary
3. Give existing work an explicit drain budget
4. Retain every planned and admitted identity
5. Run a bounded drain simulation
6. Close resources without inventing remote completion
7. Integrate the protocol with deployment routing
8. Validate the conversation and business boundaries
9. Frequently asked questions
10. Deploy with an auditable drain result

Separate readiness from application admission

Readiness is a signal used by surrounding infrastructure to decide whether a worker should receive new traffic. Application admission is the local decision to accept responsibility for a particular session. Existing conversations are already-owned work, often attached to long-lived sockets, media paths and external task identities.

These states can change at different times. A worker may become unready while a load balancer still has an old endpoint view. A connection accepted earlier may submit another logical request after drain starts. The application needs its own gate even when infrastructure health checks behave as designed.

Kubernetes probe documentation distinguishes readiness, liveness and startup checks. Do not use a deliberate admission closure as evidence that the process is unhealthy and must be restarted immediately. A draining worker may need to remain alive long enough to finish or account for existing work.

The local model has starting, ready, draining and stopped phases. Starting rejects admission until the trusted controller marks it ready. Beginning drain closes admission under the same lock used to register sessions. Stopped cannot return to ready through the model's startup method.

A worker's ready flag should not be inferred from the presence of a process ID or an open port. Initialization can still be incomplete, required adapters may be unavailable, and a process can already be draining. Define the exact conditions under which your application is prepared to own a new conversation.

The cold-start capacity guide separates arrival, startup and readiness. A caller arriving during startup has a different ownership and wait history from a caller already admitted to a warm worker. Preserve those distinctions when interpreting rollout delays.

A historical readiness-versus-liveness issue raises this operational question in a real deployment context. Use current Kubernetes documentation for the probe contract, and test the actual selected chart and SDK versions. The issue is background evidence, not proof that current releases still share its reported behavior.

Readiness false also does not mean every active conversation ended. Avoid a controller that tears down all connections merely because a readiness endpoint now returns unavailable. Closing new admission and retiring existing work should have distinct implementation and diagnostic paths.

The admission-control article provides the broader capacity rule: accepting responsibility should be an explicit decision with a bounded ownership contract. During a rollout, that contract must remain valid even while the infrastructure's routing view catches up.

Register ownership at one admission boundary

The race to solve is small and specific. An admission request checks ready, then a drain signal closes the worker, then the request registers a session. The worker can now own work that was absent from its drain snapshot, even though both code paths appeared to check the right state.

Use one critical section to check phase and register ownership. The drain transition uses that same critical section. A request that linearizes before closure belongs to the existing-work set; a request that linearizes after closure is rejected locally, including traffic arriving through a stale route.

The artifact registers the identity and task reference before leaving the admission lock. It does not wait for the session coroutine to begin before acknowledging local admission. This matters because task creation and first coroutine execution are separate scheduler events.

A task can be canceled before its body starts. No finally block inside that body can run if execution never entered it. The outer registry and done callback retain the identity and classify canceled-before-start without claiming that a resource context was entered or closed.

Validate the planned identity collection before building or sorting the ledger. Duplicate or malformed IDs should fail before admission begins. The model copies the supported list into its own dictionary and later returns frozen scalar receipts rather than exposing the mutable internal rows to callers.

The trusted session object is a synthetic adapter, not caller-controlled application code. Production admission should create owned session resources through a reviewed adapter factory and bind their references to the registry. Do not let a mutable external object change its tenant or ownership identity after admission.

Per-call settings isolation addresses a related ownership risk. Registering a task is insufficient if its configuration still aliases another conversation's mutable values. Keep the values and resources required for that session attached to the correct identity.

An admission result must distinguish rejection from accepted-but-not-started work. A caller or upstream router may safely choose another worker after a true rejection. Retrying an already-admitted session elsewhere requires a separate ownership transfer protocol, especially when external actions might have begun.

The regional failover boundary extends that issue beyond one event loop. A local lock cannot establish exclusive ownership across regions. If a different worker takes over business actions, the authoritative fencing and idempotency rules must apply at the protected write boundary.

Repeated drain signals should not reopen the gate or reset the work set. The local begin_drain transition is idempotent, and a repeated completed drain call returns the retained snapshot. The model assumes one drain controller; it does not implement a multiprocess supervisor election.

Give existing work an explicit drain budget

A drain budget defines how long the worker may continue already-owned work before moving to cleanup or handoff. It should fit inside the deployment's actual termination window, leaving room for signaling, resource closure and evidence persistence. It is not a promise that every conversation can finish within that time.

The sample receives a trusted deadline event. A deployment adapter must set it when the configured drain deadline is reached, using an appropriate monotonic clock and preserving the original deadline across repeated signals. The fixture sets it explicitly to exercise boundary behavior without a long wall-clock wait.

Existing sessions can complete while the event remains unset. If all owned tasks finish, the drainer proceeds without waiting for the deadline. If the deadline occurs first, it requests cancellation of unfinished local tasks and waits only within its separate cleanup budget.

Python's asyncio task documentation describes cooperative cancellation and waiting behavior. A cancellation request is not a guarantee that arbitrary code stops immediately. A coroutine can delay cleanup or an adapter can fail to cooperate, so retained references and an explicit unresolved state remain necessary.

The fixture's cleanup timeout is a synthetic test configuration. It measures no production termination latency or carrier behavior. Its purpose is to show that an incomplete close can be reported as unknown cleanup instead of hanging the drain controller forever or claiming successful closure.

A bounded controller cannot safely force an arbitrary Python coroutine to finish. The model therefore gives an entered resource's close operation its own retained task and shields that operation from session cancellation. If closure remains pending or fails, the worker stays draining and retains both session and closure references for an outer supervisor; later owner cancellation cannot overwrite failed-close evidence. After waiting on closures, phase classification filters owners that completed during that second wait instead of treating a stale pending snapshot as live work.

A close task that finishes with an exception is not a close acknowledgment. The model retains that failed task and keeps the resource disposition unknown and the worker draining. By contrast, cancellation before the session body starts never enters the resource context and therefore creates no close task or false close claim.

A process supervisor can eventually kill the process, but that action changes the evidence boundary. It does not prove remote cancellation or durable local receipt. Design durable external-operation records and carrier observations for the failure modes where in-process cleanup cannot run at all.

The audio backpressure guide explains why queued speech can outlive its conversational usefulness. A drain deadline should not become permission to flush every stale audio chunk before exit. Apply current generation, age and cancellation policy at the final submission boundary.

For already-buffered provider audio, local cancellation may not retract what has left your process. The playback acknowledgment boundary distinguishes writer acceptance, endpoint playback evidence and caller audibility. Keep those observations separate from the drain controller's task status.

Retain every planned and admitted identity

A useful drain report begins with the workload universe. If a scheduler planned several sessions but stopped partway through, the unscheduled identities must remain visible. Counting only tasks that reached a final gather hides work that was never attempted or was canceled before execution.

The model initializes an outer ledger for every planned identity. Local admission changes a row to admitted, the session body marks it running, and completion or cancellation supplies a distinct disposition. A never-admitted row remains not attempted even when the worker stops.

The report separates work status, started flag, resource-closed acknowledgment and external-effect status. Those fields answer different questions. A canceled coroutine can have a closed local resource while an earlier synthetic submitted effect remains unknown.

The fixture marks one session as hypothetically submitted to demonstrate that distinction. No external request is actually made. The report is explicitly labeled synthetic, and its unknown effect is a modeled uncertainty rather than a claim about any real booking or carrier operation.

A completed local session also does not establish a verified successful business task. The outcome-metrics guide separates observable conversation events from independently confirmed outcomes. Preserve that distinction when a rollout dashboard counts drained sessions.

If the drain controller itself is canceled, the outer ledger must remain accessible. The sample's finally path cancels and accounts for owned tasks under the declared single cooperative cancellation assumption. It then re-raises the controller cancellation instead of converting it into an ordinary success return.

The batch-cancellation guide examines why one canceled request should not erase unrelated work. A drain ledger needs the same discipline: preserve each identity and disposition instead of replacing the entire batch with one generic canceled result.

Keep historical completed rows after active tasks leave the work set. Draining an empty active set should not erase the creation or completion evidence required to explain the deployment. A queue size or active-task count alone cannot serve as a durable workload history.

The criteria below use ✓ for behavior supplied by the named mechanism under its stated assumptions and ✗ for behavior it does not establish. These are contract checks, not measured uptime or a claim of lossless carrier delivery.

Run a bounded drain simulation

Save the complete fence as worker_drain.py and run it with Python 3.12 or later. It opens no sockets and starts no server or child process. The synthetic sessions expose synchronization barriers and resource-close acknowledgments so the fixture can inspect specific local race outcomes.

The model assumes one event loop and one drain controller. Session adapters cooperate with cancellation, except for the explicitly delayed close that remains separately owned until the test releases and reaps it. Cancellation after serve() returns cannot convert a pending resource close into a stopped worker. Multiple cancellation requests during cleanup, interpreter termination and process kill are outside this in-memory protocol.

The deadline event is a trusted adapter input, not a clock implementation. A production controller must derive it from the configured budget and ensure repeated signals do not extend that budget. The separate cleanup timeout bounds how long this local controller waits for cooperative closure.

import asyncio
from dataclasses import dataclass, asdict
import json

@dataclass(frozen=True)
class Receipt:
    identity: str
    work: str
    started: bool
    resource_closed: bool
    effect: str


class Session:
    def __init__(self,submitted=False,fail=False,cleanup_release=None,cleanup_fail=False):
        self.entered,self.release = asyncio.Event(),asyncio.Event()
        self.submitted,self.fail,self.closed = submitted,fail,False
        self.cleanup_release,self.cleanup_fail = cleanup_release,cleanup_fail

    async def __aenter__(self):
        return self

    async def serve(self):
        self.entered.set()
        await self.release.wait()
        if self.fail:
            raise RuntimeError('synthetic session error')

    async def __aexit__(self,*unused):
        if self.cleanup_release is not None:
            await self.cleanup_release.wait()
        if self.cleanup_fail:
            raise RuntimeError('synthetic resource close error')
        self.closed = True


class Worker:
    # One event loop and one drain controller; adapters are cooperative.
    def __init__(self,planned):
        if type(planned) not in (tuple,list) or not all(type(x) is str and x and len(x)<=80 for x in planned):
            raise ValueError('planned identities')
        if len(set(planned)) != len(planned):
            raise ValueError('duplicate planned identity')
        self.rows = {x:dict(work='not_attempted',started=False,resource_closed=False,effect='not_attempted') for x in tuple(planned)}
        self.tasks,self.sessions,self.closures = {},{},{}
        self.phase = 'starting'
        self.lock = asyncio.Lock()
        self.drain_started = asyncio.Event()

    async def ready(self):
        async with self.lock:
            if self.phase != 'starting':
                return False
            self.phase = 'ready'
            return True

    async def admit(self,identity,session):
        if type(identity) is not str or identity not in self.rows or type(session) is not Session:
            raise ValueError('admission')
        async with self.lock:
            if self.phase != 'ready':
                return 'rejected'
            if self.rows[identity]['work'] != 'not_attempted':
                return 'duplicate_identity'
            self.rows[identity]['work'] = 'admitted'
            self.sessions[identity] = session
            task = asyncio.create_task(self._run(identity,session))
            self.tasks[identity] = task
            task.add_done_callback(lambda done,key=identity:self._done(key,done))
            return 'admitted'

    def _done(self,identity,task):
        row = self.rows[identity]
        if not row['started'] and task.cancelled():
            row['work'] = 'canceled_before_start'
        if not task.cancelled():
            task.exception()  # Observe unexpected failures without dropping identity.
        self._maybe_stopped()

    def _maybe_stopped(self):
        unresolved = any(row['work'] == 'unknown_cleanup' and not row['resource_closed'] for row in self.rows.values())
        if self.phase == 'draining' and not unresolved and all(task.done() for task in self.tasks.values()) and all(task.done() for task in self.closures.values()):
            self.phase = 'stopped'

    async def _close(self,identity,session):
        try:
            await session.__aexit__(None,None,None)
        finally:
            self.rows[identity]['resource_closed'] = session.closed

    def _closed(self,identity,task):
        row = self.rows[identity]
        error = None if task.cancelled() else task.exception()
        if task.cancelled() or error is not None or not self.sessions[identity].closed:
            row['work'] = 'unknown_cleanup'
        elif row['work'] == 'unknown_cleanup':
            owner = self.tasks[identity]
            row['work'] = 'canceled' if row['started'] and (owner.cancelled() or owner.cancelling()) else row['work']
        row['resource_closed'] = self.sessions[identity].closed
        self._maybe_stopped()

    async def _run(self,identity,session):
        row = self.rows[identity]
        row['started'],row['work'] = True,'running'
        if session.submitted:
            row['effect'] = 'unknown'
        entered = False
        try:
            await session.__aenter__()
            entered = True
            await session.serve()
            row['work'] = 'completed'
        except asyncio.CancelledError:
            if row['work'] != 'unknown_cleanup':
                row['work'] = 'canceled'
            raise
        except Exception:
            row['work'] = 'failed'
        finally:
            if entered:
                closer = asyncio.create_task(self._close(identity,session))
                self.closures[identity] = closer
                closer.add_done_callback(lambda done,key=identity:self._closed(key,done))
                try:
                    await asyncio.shield(closer)
                except asyncio.CancelledError:
                    if row['work'] != 'unknown_cleanup':
                        row['work'] = 'canceled'
                    raise
                finally:
                    row['resource_closed'] = session.closed

    async def begin_drain(self):
        async with self.lock:
            if self.phase in {'starting','ready'}:
                self.phase = 'draining'
                self.drain_started.set()
            return self.phase

    def snapshot(self):
        return tuple(Receipt(identity,**row) for identity,row in sorted(self.rows.items()))

    async def _finish_drain(self,remaining,signal,cleanup_seconds):
        finish_at = asyncio.get_running_loop().time()+cleanup_seconds
        if signal is not None:
            await asyncio.gather(signal,return_exceptions=True)
        if remaining:
            _,pending_tasks = await asyncio.wait(remaining,timeout=max(0,finish_at-asyncio.get_running_loop().time()))
        else:
            pending_tasks = set()
        closures = {task for task in self.closures.values() if not task.done()}
        if closures:
            _,pending_closures = await asyncio.wait(closures,timeout=max(0,finish_at-asyncio.get_running_loop().time()))
        else:
            pending_closures = set()
        # The closure wait can give an owner from the first wait time to finish.
        pending = {task for task in pending_tasks|pending_closures if not task.done()}
        for identity,task in self.tasks.items():
            closer = self.closures.get(identity)
            if task.done() and (closer is None or closer.done()):
                self._done(identity,task)
            else:
                self.rows[identity]['work'] = 'unknown_cleanup'
        # Session and closure references remain owned if cleanup is unresolved.
        unresolved = any(row['work'] == 'unknown_cleanup' and not row['resource_closed'] for row in self.rows.values())
        self.phase = 'draining' if pending or unresolved else 'stopped'

    async def drain(self,deadline,cleanup_seconds=0.1):
        if type(deadline) is not asyncio.Event or type(cleanup_seconds) not in (int,float) or not 0 < cleanup_seconds <= 1:
            raise ValueError('drain budget')
        await self.begin_drain()
        if self.phase == 'stopped':
            return self.snapshot()
        tasks = set(self.tasks.values())
        signal = None
        try:
            if tasks and not deadline.is_set():
                signal = asyncio.create_task(deadline.wait())
                all_done = asyncio.gather(*tasks,return_exceptions=True)
                await asyncio.wait({all_done,signal},return_when=asyncio.FIRST_COMPLETED)
        finally:
            # Shield one bounded accounting pass from controller cancellation.
            if signal is not None:
                signal.cancel()
            remaining = {task for task in tasks if not task.done()}
            for task in remaining:
                task.cancel()
            cleanup = asyncio.create_task(self._finish_drain(remaining,signal,cleanup_seconds))
            try:
                await asyncio.shield(cleanup)
            except asyncio.CancelledError:
                await cleanup
                raise
        return self.snapshot()


async def main():
    worker = Worker(['complete','held','never-admitted'])
    normal,held = Session(),Session(submitted=True)
    assert await worker.admit('complete',normal) == 'rejected'
    assert await worker.ready()
    assert await worker.admit('complete',normal) == 'admitted'
    await normal.entered.wait()
    normal.release.set()
    await worker.tasks['complete']
    assert normal.closed
    assert await worker.admit('held',held) == 'admitted'
    await held.entered.wait()
    deadline = asyncio.Event()
    drain = asyncio.create_task(worker.drain(deadline))
    await worker.drain_started.wait()
    assert await worker.admit('never-admitted',Session()) == 'rejected'
    assert await worker.begin_drain() == 'draining'
    deadline.set()
    rows = {r.identity:r for r in await drain}
    assert rows['complete'].work == 'completed'
    assert rows['held'].work == 'canceled' and rows['held'].resource_closed
    assert rows['held'].effect == 'unknown'
    assert rows['never-admitted'].work == 'not_attempted'
    assert worker.phase == 'stopped'
    assert await worker.drain(deadline) == worker.snapshot()
    assert not await worker.ready()
    before = Worker(['before-start'])
    await before.ready()
    session = Session()
    await before.admit('before-start',session)
    expired = asyncio.Event(); expired.set()
    result = await before.drain(expired)
    assert result[0].work == 'canceled_before_start'
    assert not result[0].started and not session.closed
    assert 'before-start' not in before.closures
    canceled = Worker(['active','not-scheduled'])
    await canceled.ready()
    active = Session()
    await canceled.admit('active',active)
    await active.entered.wait()
    task = asyncio.create_task(canceled.drain(asyncio.Event()))
    await canceled.drain_started.wait()
    task.cancel()
    try:
        await task
        raise AssertionError('controller cancellation swallowed')
    except asyncio.CancelledError:
        pass
    assert {r.identity for r in canceled.snapshot()} == {'active','not-scheduled'}
    assert active.closed and canceled.rows['active']['work'] == 'canceled'
    failed = Worker(['failure'])
    await failed.ready()
    broken = Session(fail=True)
    await failed.admit('failure',broken)
    await broken.entered.wait()
    broken.release.set()
    await failed.tasks['failure']
    assert failed.rows['failure']['work'] == 'failed' and broken.closed
    # A delayed close exceeds the cleanup budget; the outer owner later reaps it.
    cleanup_release = asyncio.Event()
    slow = Worker(['slow-close'])
    await slow.ready()
    delayed = Session(cleanup_release=cleanup_release)
    await slow.admit('slow-close',delayed)
    await delayed.entered.wait()
    expired = asyncio.Event(); expired.set()
    unknown = await slow.drain(expired,cleanup_seconds=0.001)
    assert unknown[0].work == 'unknown_cleanup' and not delayed.closed
    assert slow.phase == 'draining' and not slow.tasks['slow-close'].done()
    assert not slow.closures['slow-close'].done()
    cleanup_release.set()
    await asyncio.gather(*slow.closures.values(),return_exceptions=True)
    await asyncio.gather(*slow.tasks.values(),return_exceptions=True)
    await asyncio.sleep(0)  # Run retained-task completion callbacks.
    assert delayed.closed and slow.phase == 'stopped'
    # Cancellation during bounded cleanup still finishes the accounting pass.
    cleanup_release = asyncio.Event()
    interrupted = Worker(['interrupted-cleanup'])
    await interrupted.ready()
    delayed = Session(cleanup_release=cleanup_release)
    await interrupted.admit('interrupted-cleanup',delayed)
    await delayed.entered.wait()
    expired = asyncio.Event(); expired.set()
    controller = asyncio.create_task(interrupted.drain(expired,cleanup_seconds=0.001))
    while not interrupted.tasks['interrupted-cleanup'].cancelling():
        await asyncio.sleep(0)
    controller.cancel()
    try:
        await controller
        raise AssertionError('cleanup cancellation swallowed')
    except asyncio.CancelledError:
        pass
    assert interrupted.rows['interrupted-cleanup']['work'] == 'unknown_cleanup'
    assert interrupted.phase == 'draining'
    cleanup_release.set()
    await asyncio.gather(*interrupted.closures.values(),return_exceptions=True)
    await asyncio.gather(*interrupted.tasks.values(),return_exceptions=True)
    await asyncio.sleep(0)
    assert delayed.closed and interrupted.phase == 'stopped'
    # Cancellation after serve returns cannot abandon the owned close task.
    cleanup_release = asyncio.Event()
    post_serve = Worker(['post-serve-close'])
    await post_serve.ready()
    delayed = Session(cleanup_release=cleanup_release)
    await post_serve.admit('post-serve-close',delayed)
    await delayed.entered.wait()
    delayed.release.set()
    while 'post-serve-close' not in post_serve.closures:
        await asyncio.sleep(0)
    expired = asyncio.Event(); expired.set()
    snapshot = await post_serve.drain(expired,cleanup_seconds=0.001)
    assert snapshot[0].work == 'unknown_cleanup' and not snapshot[0].resource_closed
    assert post_serve.phase == 'draining' and not post_serve.closures['post-serve-close'].done()
    cleanup_release.set()
    await post_serve.closures['post-serve-close']
    await asyncio.gather(*post_serve.tasks.values(),return_exceptions=True)
    await asyncio.sleep(0)
    assert delayed.closed and post_serve.phase == 'stopped'
    # A failed close remains retained and unresolved rather than looking stopped.
    failed_close = Worker(['failed-close'])
    await failed_close.ready()
    failed_release = asyncio.Event()
    unclosed = Session(cleanup_release=failed_release,cleanup_fail=True)
    await failed_close.admit('failed-close',unclosed)
    await unclosed.entered.wait()
    unclosed.release.set()
    while 'failed-close' not in failed_close.closures:
        await asyncio.sleep(0)
    failed_owner = failed_close.tasks['failed-close']
    failed_closer = failed_close.closures['failed-close']
    failed_closer.add_done_callback(lambda unused:failed_owner.cancel())
    failed_release.set()
    await asyncio.gather(*failed_close.tasks.values(),return_exceptions=True)
    expired = asyncio.Event(); expired.set()
    failed_snapshot = await failed_close.drain(expired)
    assert failed_snapshot[0].work == 'unknown_cleanup' and not failed_snapshot[0].resource_closed
    assert failed_close.phase == 'draining'
    assert failed_close.closures['failed-close'].done()
    assert type(failed_close.closures['failed-close'].exception()) is RuntimeError
    # An owner snapshotted pending can finish while its closure is awaited.
    filtered = Worker(['closing'])
    await filtered.ready()
    close_release = asyncio.Event()
    closing = Session(cleanup_release=close_release)
    assert await filtered.admit('closing',closing) == 'admitted'
    await closing.entered.wait()
    closing.release.set()
    while 'closing' not in filtered.closures:
        await asyncio.sleep(0)
    real_wait,calls = asyncio.wait,0
    async def controlled_wait(awaitables,*args,**kwargs):
        nonlocal calls
        calls += 1
        owned = set(awaitables)
        if calls == 1:
            return set(),owned
        close_release.set()
        result = await real_wait(owned,*args,**kwargs)
        await asyncio.sleep(0)
        return result
    asyncio.wait = controlled_wait
    try:
        expired = asyncio.Event(); expired.set()
        filtered_snapshot = await filtered.drain(expired)
    finally:
        asyncio.wait = real_wait
    assert calls == 2 and filtered.tasks['closing'].done() and filtered.closures['closing'].done()
    assert filtered_snapshot[0].resource_closed and filtered.phase == 'stopped'
    for reverse in (False,True):
        race = Worker(['race'])
        await race.ready()
        admission = race.admit('race',Session())
        closing = race.begin_drain()
        values = await asyncio.gather(*([closing,admission] if reverse else [admission,closing]))
        status = values[1] if reverse else values[0]
        assert status in {'admitted','rejected'}
        expired = asyncio.Event(); expired.set()
        await race.drain(expired)
        assert (race.rows['race']['work'] == 'not_attempted') == (status == 'rejected')
    print(json.dumps({'fixture_class':'Synthetic example',
                      'ledger':[asdict(r) for r in worker.snapshot()],
                      'before_start':[asdict(r) for r in result],
                      'cleanup_budget_snapshot':[asdict(r) for r in unknown],
                      'carrier_observation':'not_tested','process_kill_durability':'not_tested'},indent=2))
    print('All local admission, drain and cleanup fixtures passed')


if __name__ == '__main__':
    asyncio.run(main())

The normal session closes successfully before drain begins. The held session starts, carries a synthetic unknown external effect marker and is canceled when the deadline event is set. The unadmitted planned identity remains in the same report as not attempted.

The immediate-deadline fixture admits a task and drains before its first coroutine execution. Its receipt says canceled before start, with no close acknowledgment. That is correct because no resource context was entered; claiming cleanup there would manufacture evidence from a finally block that never ran.

Another fixture cancels the drain controller while an active session is waiting. The session registry remains available and the started adapter closes cooperatively. The final snapshot is retrieved from the worker after cancellation, so the evidence does not disappear with the canceled task's return value.

The delayed-close fixture crosses the cleanup budget and returns unknown cleanup while retaining the task reference. An outer owner then releases the close barrier and gathers the task. This is an explicit handoff of still-owned cleanup, not a mechanism for abandoning arbitrary background tasks.

The delayed-close receipt is an immutable snapshot of what the controller knew at its cleanup boundary. Reaping the task later can add a newer close observation to the live ledger, but it does not retroactively make that earlier report a confirmed close. Keep both observation time and receipt identity when exporting such evidence.

The admission/drain race is exercised in both submitted coroutine orders. The lock determines which transition wins, and the ledger must agree with that result. Scheduler order can affect admission disposition, but it cannot produce an admitted session that is missing from the owned registry.

Close resources without inventing remote completion

Resource closure has several layers. Your coroutine may exit, a local socket may close, an SDK may acknowledge shutdown and a carrier may later report a call-leg state. Those observations should not be compressed into one boolean called graceful.

The synthetic adapter sets its close flag inside its asynchronous context exit. The fixture checks the actual adapter object after normal completion, exception and cancellation. That is a local lifetime assertion; it does not prove that an external media endpoint stopped playing or that a remote server acknowledged a protocol close.

A cleanup timeout needs its own disposition. The model uses unknown cleanup while keeping the phase draining and preserving task references. It does not mark the worker stopped until its owned task set has completed under the observed local boundary.

Business-intent idempotency protects a different risk. A submitted operation whose response is lost remains the same intent through drain and recovery. Canceling the conversation task must not generate a new booking key or cause another worker to assume the earlier write safely failed.

AWS's idempotent API design guidance explains why caller-supplied request identity is useful for safe recovery. The actual provider still needs documented support. Retain immutable operation values and permitted submission evidence outside an ephemeral session task when its outcome can outlive the conversation.

A CRM refresh waiting during drain also needs current admission and authorization checks before it can submit another action. The CRM failure policy rejects stale refresh success and preserves unknown writes. A late credential result should not bypass a worker's closed action gate.

Use explicit cleanup ownership for resources shared across sessions. Closing a shared model client because one session ended can disrupt other conversations. Conversely, retaining a shared connection forever because one task failed can leak resources. Declare whether the session, worker or supervisor owns each handle.

Do not promise that every cleanup path can run under forced process termination. A kill can interrupt Python before any outer ledger is exported. Durable state and external observations must support the recovery claim you want to make after such a failure; the in-memory program deliberately does not supply them.

The webhook evidence contract can support later provider reconciliation. Its signed callback remains evidence to verify and project under the provider's ordering semantics. It should not be treated as proof that a local close flag or a rollout controller knew the final state earlier.

Integrate the protocol with deployment routing

A production rollout needs a sequence that matches its actual routing and termination mechanics. Close the local admission gate, publish readiness changes through the supported integration, let existing work drain within its budget, then account for cleanup and unresolved external operations before process exit.

This is a protocol outline, not a copied universal Kubernetes manifest. Different load balancers, WebSocket transports, dispatch systems and carrier integrations can deliver different races. Test the exact deployed stack instead of assuming a sleep in a termination hook always covers propagation.

Kubernetes Deployment documentation describes rolling update controls. Replica availability settings do not independently migrate an established conversation's media state or transfer business-action ownership. Keep those application requirements explicit when selecting rollout settings.

The termination grace period includes time consumed by termination hooks under the documented Pod lifecycle. Budget drain and cleanup against the whole sequence rather than treating each stage as if it received a fresh full interval. An overlong hook can leave less time for the application to finish its own work.

LiveKit's self-hosted deployment guidance is a provider-specific example of agent deployment and drain considerations. Verify the SDK and deployment version you use. Its documented behavior does not establish that an unrelated voice stack will preserve every active call under the same configuration.

A historical SIGTERM shutdown report describes an agent becoming unresponsive during server termination. It motivates testing shutdown order across components. It is not a current failure-rate claim, and the local simulation does not reproduce or resolve that particular deployed incident.

For long-lived connections, identify who owns routing after admission. Removing a worker from new-request routing does not necessarily close an existing socket. Closing that socket may itself require a caller-safe handoff or explicit termination policy rather than an assumption that the replacement replica receives the same session automatically.

The network-resume guide explains why reconnecting needs generation and replay boundaries. A rollout-induced reconnect must not replay obsolete speech or execute old tool results merely because the new transport is healthy.

If your rollout transfers ownership, use an authoritative protocol. etcd's v3.6 API guarantees distinguish linearizable operations from weaker observation paths such as watches. A watcher noticing a draining worker is useful notification, not an atomic grant for a replacement to perform protected business effects.

Validate the conversation and business boundaries

Start with local state tests, then add routing, transport and provider evidence. The local program checks admission and cleanup accounting. A deployment test should also send new traffic during stale routing, hold active conversations through a rollout and inspect what each endpoint actually observes.

The bursty-call load-testing method helps construct workload arrival schedules. Retain every planned identity when the test stops early. A report that counts only completed calls can hide canceled-before-start and not-attempted work, making a disruptive rollout look artificially successful.

Use voice-agent observability to preserve the meanings of arrival, admission, drain-start, deadline, local close and provider acknowledgment timestamps. Keep clock domains explicit. Overlapping durations should not be added as independent phases, and a server-local timer is not a caller-side audio observation.

For media validation, record whether audio continued, stalled, repeated or ended at the endpoint under the permitted test setup. The SIP troubleshooting guide separates signaling success from working media. A connected call status alone cannot establish an uninterrupted conversation during rollout.

Business actions need their own checks. Verify that a submitted operation remains visible after session cancellation, that a replacement worker reuses its immutable identity, and that an unknown result is reconciled before another mutation. A clean audio handoff can still conceal a repeated booking if those checks are omitted.

Test user-driven interruption during drain as well. The interruption-safe output contract requires current generation and cancellation state at the final submission boundary. A worker trying to finish quickly before exit should not release stale speech or ignore a caller's correction.

Keep the test report scoped to observed evidence. A finite local fixture proves its assertions on its runtime. A controlled provider test adds evidence for the selected versions and paths. Neither justifies a universal statement that rolling deployments never lose a conversation across every network, process failure and carrier configuration.

When cleanup times out, preserve the pending owner and the next recovery step. Unknown cleanup must not vanish into a success count just because the rollout controller moved on. If the process is subsequently killed, the report should say which local evidence was durable and which remote outcomes require later observation.

Define what a handoff means before using it as the fallback for a draining conversation. A new worker receiving context is not automatically the owner of the old worker's media stream or submitted business actions. The warm-handoff guide separates useful transfer context from independently observed acceptance and continuing responsibility.

An authorized handoff should preserve the exact pending intent, current conversation generation and permitted diagnostic references. It should also state which actions were never submitted, which completed with evidence and which remain unknown. Without that distinction, the receiving actor can repeat a mutation while believing it is simply finishing abandoned work.

Drain can begin while a tool result is being processed rather than while audio is actively flowing. Closing only the listener socket leaves that tool path able to submit another business action after admission has ended. Define whether admission closure applies to new sessions only or also to new actions inside an existing session, and enforce the selected rule at the corresponding local gate.

Existing conversations may legitimately continue some work during their drain budget. If so, the action gate must still check current authorization, ownership, deadline and exact approved values. The word existing does not exempt a session from normal policy; it only identifies which conversations the worker already owns while new admission is closed.

The model deliberately avoids an automatic session transfer. It keeps each admitted identity attached to one local worker until its observed disposition is recorded. A production transfer protocol needs explicit acceptance by the new owner, a handoff boundary for in-flight work and fencing against late actions from the previous owner. Merely serializing a transcript does not provide those guarantees.

A rolling update can also change the code that interprets retained evidence. Version the session and operation schemas needed for recovery, and test compatibility between the old and new workers before a rollout depends on it. If the new version cannot interpret an unresolved operation safely, fail closed and preserve it for a supported recovery path rather than guessing its meaning.

Keep configuration changes within the same review boundary. A shorter termination grace period can invalidate a previously successful drain test even when application code is unchanged. A new load-balancer timeout can alter connection behavior, and a provider SDK update can change cleanup mechanics. The relevant proof belongs to the actual deployed combination, not merely to a familiar container image name.

A useful test captures both the intended workload and the infrastructure events that interrupt it. Record which identities were scheduled, admitted and started before the drain signal, then retain their final local and external dispositions. This allows a reviewer to recompute the denominator and distinguish an intentionally rejected new session from an admitted conversation that was interrupted.

Do not treat every rejection during drain as a service failure. New admission may correctly move to another ready worker under the routing protocol. The integration test must show where that request went and whether another worker accepted it, while preserving the fact that the draining worker did not acquire ownership. A local rejection response alone proves only its own gate behavior.

Test drain during both quiet and busy parts of a conversation. A session waiting for caller input, a session sending buffered audio and a session awaiting a remote write have different cleanup obligations. Use controlled synthetic cases for each local path, then collect permitted endpoint observations for the actual transport. A single idle-call rollout cannot establish the behavior of every active state.

Finally, assign an owner to the drain report itself. If the report exists only as a return value from a task that gets canceled or a process that gets killed, useful evidence can disappear at precisely the moment it is needed. The sample keeps an outer in-memory ledger; a deployment requiring recovery after process loss needs an appropriate durable sink and an honest account of any unflushed observations.

---

Frequently asked questions

Does readiness false stop all new sessions immediately?

It changes a routing signal, while external components can retain stale endpoint views. The application must reject new admission after its own gate closes. The local fixture tests that rejection separately from existing-session completion.

Should a draining worker fail its liveness probe?

A worker deliberately closing new admission can still be healthy enough to finish existing work. Readiness and liveness serve different purposes in Kubernetes. Test the selected probe integration so drain does not trigger an unintended restart before its cleanup budget is available.

When does a session become owned by the worker?

In the model, phase check, ledger registration and task-reference creation occur in one admission critical section. A session admitted before drain belongs to existing work even if its coroutine has not started. A request after closure is rejected without acquiring ownership.

What if admission races with the drain signal?

The shared lock determines the local linearization order. Either the session is registered before closure or it is rejected afterward. The fixture checks both coroutine submission orders and verifies that the ledger agrees with the observed admission result.

Why keep a ledger outside the session tasks?

A task can be canceled before it starts or before it returns its final report. An outer ledger preserves planned and admitted identities through those paths. Counting only results returned by a final gather would omit work that never reached its body.

Can a coroutine clean up before it starts?

Its internal finally block cannot run if execution never entered the body. The model records canceled-before-start and does not claim a resource-close acknowledgment. Resources allocated before task creation need a separate outer owner and cleanup contract.

How is the drain deadline represented?

The local model receives a trusted event that the deployment adapter must set at the configured deadline. The fixture controls it directly to exercise races quickly. Production timing requires an appropriate monotonic clock and an original budget that repeated signals cannot extend.

What happens when the deadline expires?

The controller requests cancellation of unfinished tasks and waits within a separate cleanup budget. Completed, canceled and unresolved cleanup outcomes remain distinct. A cancellation request alone does not prove that arbitrary adapters stopped or that remote effects were canceled.

What if cleanup exceeds its budget?

The model records unknown cleanup, retains task references and remains draining. An outer owner must continue supervising or apply the documented process policy. The fixture explicitly releases and reaps its delayed-close task so the test does not abandon background work.

Can repeated drain signals reopen admission?

The model's begin-drain transition is idempotent and never returns to ready. Repeating a completed drain returns the retained snapshot. It assumes one drain controller and does not implement a distributed supervisor or concurrent cleanup ownership protocol.

What if the drain controller is canceled?

Under the stated single cooperative cancellation assumption, its finally path cancels and accounts for owned session tasks. The outer worker ledger remains available after the controller raises cancellation. Repeated cancellation during cleanup and process kill require additional supervision and durability design.

Does local resource closure prove the caller heard the last audio?

No. A local adapter close, writer acceptance and endpoint playback are different evidence boundaries. Test actual media observations separately, and avoid treating a closed socket or completed coroutine as proof of acoustic audibility or comprehension.

Does a stopped conversation cancel a submitted booking?

Stopping local work does not retract an already submitted remote mutation. Preserve its immutable operation identity and unknown outcome until the provider supplies authoritative evidence. A replacement worker must reconcile rather than assume the old operation safely failed.

Does maxUnavailable protect active conversations?

It controls a Deployment availability dimension, not the application state of each established conversation. Session drain, transport continuity and business ownership need separate protocols. Validate the selected rollout configuration against those boundaries instead of inferring them from replica health.

Is a termination hook a fresh grace period?

The hook consumes time within the configured termination sequence described by Kubernetes. Budget application drain and cleanup against the complete window. Do not assume each stage receives an independent full interval or that a long hook leaves unlimited shutdown time afterward.

Can another region immediately take over the task?

Only under an explicit authoritative ownership protocol. A local drain notice or a watcher update does not grant exclusive business-action authority. The replacement needs current fencing, operation idempotency and any required session-transfer evidence at the protected boundary.

What does the asyncio fixture establish?

It establishes the asserted single-process admission, ledger and cooperative cleanup behavior on its runtime. It does not test Kubernetes routing, SIP teardown, physical audio, carrier delivery or process-kill durability. Those missing layers are explicit in the emitted synthetic report.

Why distinguish not attempted from canceled before start?

Not attempted means the planned identity was never admitted. Canceled before start means ownership was accepted and a task existed, but its body never ran. Those states imply different admission and recovery responsibilities even though neither performed session work.

How should rollout metrics count unknown sessions?

Keep them visible as unresolved rather than folding them into completed or failed counts. Define the denominator over all planned or admitted identities, according to the question. Local work completion and confirmed business success should remain separate outcome dimensions.

What should be tested before claiming a smooth rollout?

Test local races, stale-route admission, active media behavior, submitted-operation recovery and the exact deployed provider versions. Retain every workload identity and its evidence. Report the observed scope instead of claiming zero loss from a local simulation alone.

Deploy with an auditable drain result

Write the admission boundary and session ledger before tuning rollout delays. Confirm that starting and draining workers reject new work, while admitted identities remain owned until completion, explicit cancellation or an unresolved cleanup record. Preserve canceled-before-start and never-attempted states rather than forcing both into one failure bucket.

Run the full synthetic fence, then add deployment and provider observations for your stack. A rollout report should identify which resources closed, which business effects remain unknown and which evidence survives process termination. Those details let an operator recover safely instead of repeating work based on an empty active-task count.

For 60db Agent integrations, treat graceful drain as an auditable protocol with declared limits. Use the business-intent retry contract for remote effects and the regional ownership contract when another worker can act. Then make the deployment claim match the conversation and business evidence actually collected.