Regional failover without two agents owning the same task

TL;DR: Failover safety requires an authoritative current owner and a write-time fence at every protected mutation. A lease can expire while old code continues, so publish the new fence when granting ownership, reject expired grants even before another owner writes, and keep business-operation idempotency separate from owner generation.

---

A regional worker pauses long enough for its lease to expire. Another region acquires the task, but the old worker resumes with a completed tool result and its previous belief that it still owns the conversation. If the protected service trusts that belief, both workers can perform the same business action.

Moving traffic to a healthy region does not resolve this ownership race. The rolling-deployment drain contract separates routing from admitted work; failover must additionally decide who may mutate the shared business resource after ownership changes.

etcd's v3.6 API guarantees describe atomic operations and linearizable reads, with different guarantees for watches. Those mechanisms can support an authoritative ownership protocol, but they do not automatically fence an unrelated booking or CRM provider. The protected effect must participate in the enforcement boundary.

The program below models one local authority that issues grants and enforces protected writes in the same SQLite store. It tests the stale-owner race without implementing regional consensus, media migration or a production failover service for 60db Agent. Its strongest claims stop at that local authoritative transaction.

Table of contents

1. Separate availability from mutation ownership
2. Treat a lease as an expiring grant
3. Publish the fence when ownership changes
4. Keep operation identity independent of owner epoch
5. Run an authoritative local fencing model
6. Hold actions when authority is unavailable
7. Enforce the boundary at downstream effects
8. Test failover claims against their failure domain
9. Frequently asked questions
10. Recover with one auditable action owner

Separate availability from mutation ownership

Availability answers whether a component can serve some work. Ownership answers whether a particular actor may perform a protected action for a particular task now. A region can be reachable, authenticated and able to generate speech while lacking authority to update the task's business resource.

State that authority explicitly. The local model scopes a grant by authority realm, tenant, task, owner and epoch. The authority stores the current owner and expiry for that task. A protected commit checks that stored record instead of relying on the caller's cached grant alone.

An owner label such as region A describes an actor in the fixture, not a deployment topology recommendation. Production ownership may belong to a worker, shard, task coordinator or authoritative gateway. The important requirement is a precise identity that the protected service can compare against its current authority.

AWS's leader-election discussion describes both the usefulness and failure modes of leadership. Electing a leader simplifies work assignment, but it does not make a paused former leader stop executing. Protection must survive that actor's later return.

A traffic router can send new calls elsewhere while old work remains active. The admission-control guide concerns acceptance of new responsibility, while the ownership gate concerns current permission to perform effects for already-owned tasks. Keep those decisions separate in the controller and report.

Authentication is another independent condition. A former owner may still possess a valid service credential. The CRM authorization failure policy shows why refreshed credentials do not create missing permission; similarly, a valid credential does not establish current task ownership.

The local model trusts that business intent and user authorization have been checked by their owning workflow. Its grant establishes only the modeled task writer. Production integration must retain both boundaries: the current worker must own the task, and the exact principal/action/resource combination must still be authorized.

Do not identify the owner solely by a reusable hostname or regional label. A restarted process can inherit the same label while holding a different grant generation. The epoch and authority realm distinguish those admissions so a late result from the previous incarnation cannot masquerade as the current writer.

The per-call isolation contract applies to the values carried by each owner too. A correctly fenced task can still perform the wrong action if its approved payload aliases another call's mutable state. Own the intent snapshot independently of scheduling and worker lifecycle.

Treat a lease as an expiring grant

A lease grants authority only within its defined validity interval. The client can continue computing after that interval ends because process scheduling, network delays and application bugs do not automatically stop at the authority's deadline. An old local belief is insufficient at action time.

Redis's distributed-lock guidance discusses validity assumptions and fencing for long-running work. Read the complete selected lock protocol before using it. A lock recipe's assumptions should not be silently expanded into safety for every remote provider effect.

The model checks lease expiry inside the same authoritative transaction as the protected mutation. At the exact expiry boundary, the old grant is rejected even when no new owner has claimed the task. This is stronger than a receiver that merely remembers the highest token it has seen on previous writes.

Renewal also checks the current owner, epoch and expiry. It cannot revive an expired grant. If the same actor claims the task again after expiry, the authority issues a new epoch and treats it as a new ownership generation rather than retroactively extending the old grant.

The fixture uses trusted nondecreasing integer authority ticks. They are synthetic ordering inputs, not caller timestamps or a claim about clock synchronization between regions. A real issuer must define its clock and expiry semantics, including process restart, pause behavior and how the protected service obtains current validity evidence.

The model persists its last observed tick and rejects rollback. That is a local teaching guard against passing an earlier clock value to revive an expired lease. It does not solve distributed clock uncertainty or make a host-local monotonic timer durable across machine replacement.

An ownership API response can itself be delayed. A historical etcd lock issue asks about ownership returned after lease expiry. It motivates checking the action boundary, but it is not evidence that current etcd releases share that historical behavior or that this program tests their implementation.

After any await, assume relevant authority may have changed. The held-owner fixture pauses one actor, advances authority to a new owner and then releases the old task. Its commit must consult current authority and fail, even though its local computation completed successfully.

The stale tool-result boundary supplies the matching business rule. A result can be technically valid for an earlier request and still be inadmissible for the current owner or resource version. Preserve its original provenance without allowing it to become a new action automatically.

A lease does not cancel media already buffered elsewhere. The playback evidence guide distinguishes local submission from endpoint playback. Fencing business mutations cannot retract speech that a provider has already accepted or prove that the caller heard only one agent.

Publish the fence when ownership changes

A fencing token is useful only where its meaning is enforced. A receiver that rejects tokens below the highest token it has previously seen can still accept an old owner's write before the new owner makes its first write. The new grant exists, but the receiver's remembered floor has not advanced.

Close that gap by publishing the authoritative floor at ownership grant, or by checking the current owner and lease directly at every protected mutation. The sample combines both: grant issuance updates the current owner row and epoch atomically, and commits compare against that authoritative row before changing business state.

The test deliberately sends the old owner after the new grant but before any new-owner write. It must be rejected. A test that lets the new owner write first would miss the weaker highest-seen-token implementation and could report a false sense of protection.

etcd's v3.6 API documentation describes revisions, transactions and leases. Those are building blocks for an authority, not a complete application policy. You still need to specify which transaction publishes ownership and how every downstream action observes that decision.

Bind an epoch to its owner and task scope. Two actors claiming the same numeric token should not both pass merely because the number is current. The fixture constructs a different owner with the new epoch and requires rejection before any mutation or receipt disclosure.

An authority realm adds another identity dimension. A new independent authority instance must not accept grants issued by a different realm just because their counters happen to match. The sample persists a generated realm per database and tests mismatch against a newly initialized independent store.

Backup restoration is harder. Restoring an old database can restore both an old counter and the same realm, so the local model cannot prove monotonic safety across that rollback. The production recovery protocol must preserve or advance the authority generation and ensure every protected receiver recognizes the new boundary.

Do not hide this obligation behind a UUID field. A randomly generated label on initial setup does not magically detect that an operator restored an old copy later. Treat restore, cloning and authority replacement as explicit operational transitions with their own fencing and reconciliation evidence.

PostgreSQL's transaction isolation documentation explains guarantees available under selected isolation levels. A production relational authority must use the right locking, constraints and retry behavior for its schema. The sample's one-store SQLite serialization is not a proof of a different database or consensus implementation.

The comparison table uses ✓ for a check supplied by the named boundary under its stated assumptions and ✗ when it is absent. It measures no regional availability, failover duration or customer outcome.

Keep operation identity independent of owner epoch

Ownership can change while business intent remains the same. A replacement worker reconciling an already-submitted booking should retain the original operation identity and exact payload. Adding the new owner epoch to that key can turn a safe retry into a new logical operation.

The sample has a frozen Intent separate from its frozen Grant. The intent binds tenant, task, operation identity, value and expected business version. The grant binds current writer authority. Neither object is derived from or silently rewritten by the other.

AWS's idempotent API guidance explains why retries need caller-provided intent identity. Fencing answers which actor may write now; idempotency answers whether the same logical mutation already has a result. They protect different invariants.

The authority checks current ownership before returning a historical receipt. A stale or expired owner receives no receipt through this method, even for an exact prior operation. A current new owner can retrieve the exact historical result without performing another mutation.

An existing operation identity with a changed payload is a conflict. The artifact binds both the synthetic value and expected business version in its explicit schema-specific serialization. A newer owner cannot change those values under the old operation ID merely because failover interrupted the conversation.

A genuinely new approved operation needs its own identity. It must also use the current business version and current authorization. Failover is not permission to combine an old approval with a new resource value or to retry a canceled task under a convenient fresh key.

The booking contract separates operation uniqueness from resource capacity. A fenced current owner can still double-book if the authoritative resource service lacks the right capacity constraint. One writer per task does not imply one reservation per real-world slot across all tasks.

The confirmation repair method supplies another boundary. A caller's correction invalidates the old exact proposal. Preserve both operation history and the new reviewed intent rather than treating every late affirmative phrase as authorization to finish the old action.

Keep receipt meaning narrow. The sample receipt records a committed local business version and scoped operation identity. It does not establish a provider reservation, a sent message, caller comprehension or completed payment. The emitted report states that remote effects were not attempted.

Run an authoritative local fencing model

Save the complete fence as fenced_task_commit.py and run it with Python 3.12 or later. It uses temporary local SQLite databases and a controlled asynchronous barrier. It opens no network connections and implements no region discovery or consensus algorithm.

The trusted authority owns grant issuance, current owner state, expiry checks and protected local business writes in one store. Its input clock represents the authority's current synthetic tick. Clients must not select arbitrary past times; the local rollback guard rejects decreasing observations but is not a substitute for a trustworthy production clock adapter.

The immutable intent contains a narrow set of printable ASCII labels and a bounded integer version. It is not a serializer for arbitrary caller objects. The protected method checks owner scope and exact operation payload before version mutation, so historical retries remain identifiable after the business version advances.

import asyncio
from contextlib import closing,contextmanager
from dataclasses import dataclass
import json
from pathlib import Path
import sqlite3
import tempfile
import uuid

@dataclass(frozen=True)
class Grant:
    realm: str
    tenant: str
    task: str
    owner: str
    epoch: int

@dataclass(frozen=True)
class Intent:
    tenant: str
    task: str
    operation: str
    value: str
    expected_version: int


def label(value):
    return type(value) is str and 0 < len(value) <= 80 and value.isascii() and all(33 <= ord(c) <= 126 for c in value)


def integer(value):
    return type(value) is int and 0 <= value <= 2**63-1


class Authority:
    # One authoritative local store models issuer AND protected write gate.
    def __init__(self,path):
        self.path,self.available = str(path),True
        with closing(sqlite3.connect(self.path)) as db:
            db.executescript('''
                CREATE TABLE IF NOT EXISTS meta(id INTEGER PRIMARY KEY,realm TEXT,epoch INTEGER,clock INTEGER);
                CREATE TABLE IF NOT EXISTS owner(tenant TEXT,task TEXT,owner TEXT,epoch INTEGER,expires INTEGER,
                    PRIMARY KEY(tenant,task));
                CREATE TABLE IF NOT EXISTS business(tenant TEXT,task TEXT,version INTEGER,
                    PRIMARY KEY(tenant,task));
                CREATE TABLE IF NOT EXISTS operation(tenant TEXT,task TEXT,identity TEXT,payload TEXT,receipt TEXT,
                    PRIMARY KEY(tenant,task,identity));
            ''')
            db.execute('INSERT OR IGNORE INTO meta VALUES(1,?,0,0)',(str(uuid.uuid4()),))
            db.commit()

    @contextmanager
    def transaction(self,now):
        if not integer(now):
            raise ValueError('trusted authority clock')
        with closing(sqlite3.connect(self.path,timeout=5,isolation_level=None)) as db:
            try:
                db.execute('BEGIN IMMEDIATE')
                realm,epoch,last = db.execute('SELECT realm,epoch,clock FROM meta WHERE id=1').fetchone()
                if now < last:
                    raise ValueError('authority clock rollback')
                db.execute('UPDATE meta SET clock=? WHERE id=1',(now,))
                yield db,realm,epoch
                db.commit()
            finally:
                if db.in_transaction:
                    db.rollback()

    def claim(self,tenant,task,owner,now,duration):
        if not all(label(v) for v in (tenant,task,owner)) or not integer(duration) or duration == 0 or not integer(now) or now+duration > 2**63-1:
            raise ValueError('claim')
        if not self.available:
            return 'authority_unavailable',None
        with self.transaction(now) as (db,realm,epoch):
            old = db.execute('SELECT expires FROM owner WHERE tenant=? AND task=?',(tenant,task)).fetchone()
            if old and now < old[0]:
                return 'busy',None
            if epoch == 2**63-1:
                return 'epoch_exhausted',None
            epoch += 1
            # Grant and fence floor are published in this same transaction.
            db.execute('UPDATE meta SET epoch=? WHERE id=1',(epoch,))
            db.execute('INSERT INTO owner VALUES(?,?,?,?,?) ON CONFLICT(tenant,task) DO UPDATE SET '
                       'owner=excluded.owner,epoch=excluded.epoch,expires=excluded.expires',
                       (tenant,task,owner,epoch,now+duration))
            db.execute('INSERT OR IGNORE INTO business VALUES(?,?,0)',(tenant,task))
            return 'granted',Grant(realm,tenant,task,owner,epoch)

    @staticmethod
    def current(db,realm,grant,now):
        if type(grant) is not Grant or not all(label(v) for v in (grant.tenant,grant.task,grant.owner)) or not integer(grant.epoch):
            return 'invalid_grant'
        if grant.realm != realm:
            return 'authority_generation_mismatch'
        row = db.execute('SELECT owner,epoch,expires FROM owner WHERE tenant=? AND task=?',
                         (grant.tenant,grant.task)).fetchone()
        if row is None or (grant.owner,grant.epoch) != row[:2]:
            return 'stale_owner'
        if now >= row[2]:
            return 'expired_owner'
        return 'current_owner'

    def renew(self,grant,now,duration):
        if not integer(duration) or duration == 0 or not integer(now) or now+duration > 2**63-1:
            raise ValueError('renewal')
        if not self.available:
            return 'authority_unavailable'
        with self.transaction(now) as (db,realm,_):
            status = self.current(db,realm,grant,now)
            if status != 'current_owner':
                return status
            # A valid renewal cannot shorten the existing lease.
            db.execute('UPDATE owner SET expires=MAX(expires,?) WHERE tenant=? AND task=?',
                       (now+duration,grant.tenant,grant.task))
            return 'renewed'

    def commit(self,grant,intent,now):
        if type(intent) is not Intent or not all(label(v) for v in (intent.tenant,intent.task,intent.operation,intent.value)) or not integer(intent.expected_version):
            raise ValueError('intent')
        payload = json.dumps([intent.value,intent.expected_version],separators=(',',':'))
        if not self.available:
            return 'authority_unavailable',None
        with self.transaction(now) as (db,realm,_):
            status = self.current(db,realm,grant,now)
            if status != 'current_owner':
                return status,None
            if (grant.tenant,grant.task) != (intent.tenant,intent.task):
                return 'scope_mismatch',None
            key = (intent.tenant,intent.task,intent.operation)
            prior = db.execute('SELECT payload,receipt FROM operation WHERE tenant=? AND task=? AND identity=?',key).fetchone()
            if prior:
                return ('historical_retry',prior[1]) if prior[0] == payload else ('operation_conflict',None)
            version = db.execute('SELECT version FROM business WHERE tenant=? AND task=?',key[:2]).fetchone()[0]
            if version != intent.expected_version:
                return 'version_conflict',None
            if version == 2**63-1:
                return 'version_exhausted',None
            receipt = json.dumps(dict(tenant=intent.tenant,task=intent.task,operation=intent.operation,
                                      committed_version=version+1),sort_keys=True,separators=(',',':'))
            db.execute('UPDATE business SET version=? WHERE tenant=? AND task=?',(version+1,)+key[:2])
            db.execute('INSERT INTO operation VALUES(?,?,?,?,?)',key+(payload,receipt))
            return 'committed',receipt


async def main():
    with tempfile.TemporaryDirectory() as directory:
        authority = Authority(Path(directory)/'authority.db')
        status,old = authority.claim('tenant-a','task-a','region-a',0,10)
        assert status == 'granted'
        intent = Intent('tenant-a','task-a','operation-a','synthetic-value',0)
        assert authority.commit(old,intent,10) == ('expired_owner',None)
        assert authority.renew(old,10,10) == 'expired_owner'
        status,new = authority.claim('tenant-a','task-a','region-b',10,10)
        assert status == 'granted' and new.epoch > old.epoch
        # Old owner is rejected BEFORE the new owner has performed any write.
        assert authority.commit(old,intent,11) == ('stale_owner',None)
        impostor = Grant(new.realm,new.tenant,new.task,'another-owner',new.epoch)
        assert authority.commit(impostor,intent,11) == ('stale_owner',None)
        assert authority.renew(new,11,2) == 'renewed'
        assert authority.commit(new,intent,12)[0] == 'committed'
        assert authority.commit(new,intent,12)[0] == 'historical_retry'
        changed = Intent('tenant-a','task-a','operation-a','different-value',0)
        assert authority.commit(new,changed,12) == ('operation_conflict',None)
        assert authority.commit(new,Intent('tenant-a','task-a','operation-b','synthetic-value',0),12)[0] == 'version_conflict'
        authority.available = False
        assert authority.commit(new,intent,12) == ('authority_unavailable',None)
        authority.available = True
        entered,release = asyncio.Event(),asyncio.Event()
        clock = [13]
        async def held_owner():
            entered.set()
            await release.wait()
            return authority.commit(new,Intent('tenant-a','task-a','operation-b','synthetic-next',1),clock[0])
        held = asyncio.create_task(held_owner())
        await entered.wait()
        clock[0] = 20
        _,latest = authority.claim('tenant-a','task-a','region-c',20,10)
        release.set()
        assert await held == ('stale_owner',None)
        assert authority.commit(latest,intent,21)[0] == 'historical_retry'
        assert authority.commit(latest,Intent('tenant-a','other-task','operation-a','synthetic-value',0),21)[0] == 'scope_mismatch'
        fresh = Authority(Path(directory)/'new-authority-generation.db')
        fresh.claim('tenant-a','task-a','region-c',21,10)
        assert fresh.commit(latest,intent,21)[0] == 'authority_generation_mismatch'
        _,same_owner = authority.claim('tenant-a','task-b','region-c',30,1)
        assert authority.renew(same_owner,31,10) == 'expired_owner'
        _,reclaimed = authority.claim('tenant-a','task-b','region-c',31,10)
        assert reclaimed.epoch > same_owner.epoch
        try:
            authority.commit(latest,intent,19)
            raise AssertionError('clock rollback admitted')
        except ValueError:
            pass
        try:
            authority.claim('tenant-a','task-a','region-z',True,10)
            raise AssertionError('boolean clock admitted')
        except ValueError:
            pass
        print(json.dumps({'fixture_class':'Synthetic example','expired_owner':'rejected_before_regrant',
                          'old_owner_before_new_write':'rejected','held_owner_after_wait':'rejected',
                          'new_owner_exact_operation':'historical_retry',
                          'regional_consensus':'not_tested','remote_effect':'not_attempted',
                          'backup_restore_epoch_safety':'external_obligation'},indent=2))
    print('All local issuer, lease, fencing and operation fixtures passed')


if __name__ == '__main__':
    asyncio.run(main())

The first assertion rejects the old owner at expiry before any new owner exists. The next ownership claim publishes its floor, and another assertion rejects the old grant before the new owner has written anything. Together those cases distinguish the intended contract from a weaker highest-seen-token receiver.

The asynchronous fixture holds a current owner's planned commit while a newer grant is issued. When the old task resumes, it uses the current authority clock and is rejected. The fixture does not rely on a sleep or assume that losing a lease automatically cancels a coroutine.

The new owner retrieves the historical result for the same operation without creating another mutation. A changed value conflicts, an incorrect expected version conflicts for a new operation, and a cross-task intent is rejected. These checks keep current ownership from becoming permission to rewrite operation history.

Python's sqlite3 documentation distinguishes transaction control from connection lifetime. The context manager explicitly closes its owned connection on success and exceptions, rolls back an unfinished transaction and commits the scope, owner and business decisions together locally.

The fresh-database realm test demonstrates rejection of a grant from a different authority instance. The report labels backup-restore epoch safety as an external obligation because restoring the same old realm and counter is not solved by that test. Keep that limitation when adapting the example.

Hold actions when authority is unavailable

A worker unable to reach current authority cannot safely replace the missing answer with its cached belief. The model's unavailable branch rejects new commit and receipt access through that boundary. It makes no mutation and does not classify the operation as a confirmed business failure.

This choice trades action availability for the declared single-authority safety rule. The conversation may continue in a limited mode, but protected writes wait for an authoritative result or an explicitly authorized recovery path. Do not turn an outage into an undocumented bypass that allows both regions to act.

The authority's availability flag is a synthetic fixture control. It does not simulate a real partition, quorum protocol or network timeout. A production adapter must distinguish unavailable authority from an authoritative denial and retain enough evidence to recover without issuing duplicate operations.

etcd's guarantee documentation matters when selecting read paths. A stale serializable read or a watch notification should not be presented as the same current authority observation as the linearizable protocol your action gate requires. Select the appropriate operation and document its failure behavior.

An action already submitted elsewhere can remain unknown while authority is unreachable. Preserve its operation identity and submission evidence. Restoring connectivity should trigger reconciliation of that original request before a new owner decides whether another mutation is necessary.

The CRM unknown-outcome policy applies here too. A timeout does not prove nonexecution, and a refreshed credential does not prove task ownership. Recovery must satisfy the provider's operation contract and current authority together.

Keep user-facing waiting language truthful. The slow-tool speech guide supports a concise explanation that an existing request is being checked. Avoid announcing a successful failover or a safely canceled action solely because another region became reachable.

A human override, if the business permits one, should be a separately authorized operation with explicit consequences and evidence. It should not silently reuse a stale grant or erase the original unknown result. The local teaching program provides no administrative bypass.

Enforce the boundary at downstream effects

A local fence cannot protect an unrelated service that ignores it. If a CRM, booking provider or message service accepts a request directly from an old worker without current ownership enforcement, your local authority's rejection of another code path does not cover that effect.

One design is a downstream API that enforces the authority-issued fence and operation identity itself. Another is an authoritative gateway through which every protected mutation must pass. Both require a precise transaction and trust boundary; simply forwarding a token header does not create enforcement.

The gateway must also close the gap between checking authority and performing the effect. If it reads current ownership, waits on an unrelated network call and then mutates without another valid boundary, ownership can change during the wait. Define the protected service's actual atomic or conditional operation rather than assuming a preflight read is enough.

PostgreSQL's constraint documentation describes structural guards that can support the resource's own invariants. For a database-owned effect, current owner, expected version and operation uniqueness can be checked within the correct transaction. An external provider still needs its documented idempotency and conditional behavior.

If the provider cannot enforce fencing, a gateway can serialize authorized submissions, but unknown remote outcomes remain. It must retain exact operation identity and reconcile rather than blindly resubmit after failover. The gateway's local ledger cannot claim an atomic commit with a provider that does not participate in the protocol.

Authenticated webhook projection can supply later outcome evidence. Preserve the provider's exact signed representation, event identity and documented ordering semantics before connecting that evidence to a business operation. A callback's authenticity is not an independent grant for another action.

Media ownership has a different effect surface. Switching the business-action owner does not necessarily stop an old audio stream or clear a provider buffer. The interruption-safe output method needs generation and cancellation checks at final speech submission, with provider-specific evidence for anything already sent.

The network-resume contract similarly prevents reconnection from replaying obsolete output. Keep business fencing and media replay protection distinct even when the same failover event triggers both workflows.

A complete integration inventory should list every route that can mutate the protected resource. Scheduled jobs, webhook handlers, operator tools and recovery workers need the same authoritative boundary. One well-fenced primary path does not compensate for an alternate client holding a credential that can bypass it.

Test failover claims against their failure domain

Begin with deterministic local negatives. Test expiry before regrant, old-owner action before the new owner's first write, equal-token different-owner claims, stale completion after an await and exact operation recovery under a new owner. These cases expose different gaps and should remain separate assertions.

The regression-testing guide helps preserve the exact fixture inputs and expected dispositions. Do not summarize every rejected action as a generic failure. Stale owner, expired owner, scope mismatch, operation conflict and unavailable authority imply different recovery paths.

Then test the actual authority service and protected adapter under controlled faults. A local SQLite model does not establish quorum behavior, regional latency, storage durability or clock assumptions. Use permitted synthetic tasks and record the selected versions and topology rather than presenting the article's fixture as a deployment benchmark.

Use voice-agent observability to retain grant issuance, lease expiry, action submission and provider acknowledgment as distinct observations. Keep their clock domains explicit. A client timestamp saying it believed itself owner is not the authoritative grant record.

An application can remain responsive during a safety hold. Measure that conversation behavior separately from protected action completion. The outcome-metrics guide distinguishes local events from confirmed business success, which is essential when a failover demonstration continues talking while writes are intentionally blocked.

A bursty-arrival test can add workload pressure after local correctness is established. Preserve all planned identities and unresolved operations when the test stops. Throughput under a healthy authority does not prove safe behavior during authority loss or an old owner's delayed return.

Finally, validate backup and restore as authority transitions. The local realm mismatch fixture is useful but incomplete for restored copies retaining old state. Your operational protocol must prevent epoch reuse from authorizing stale clients, and the report should name any unresolved receiver or provider gap plainly.

Map each failover state to an allowed next action before introducing automatic recovery. A reachable replacement can inspect permitted context, but it should not submit a mutation until its current grant and the exact business intent pass the owning boundaries. The workflow-enforcement guide explains why conversational progress and authorized action transitions need independent control.

A grant can be current while its business proposal is obsolete. If the caller changed the requested date during a regional pause, a new worker must not execute the earlier values simply because it now owns the task. Ownership transfers responsibility for evaluating the preserved evidence; it does not certify that every queued command remains appropriate.

Similarly, a business operation can have a historical result while the present owner is unable to disclose it. The model checks current writer ownership before returning history, but a real service also needs current user and resource access checks. Keep the receipt as controlled evidence rather than turning possession of its identifier into a bearer capability.

A protected write gate needs a documented linearization point. In the local program, the owner comparison and business mutation share the same serialized transaction, so a competing grant cannot interleave between them inside that store. In a distributed design, a separate authority read followed by an unrelated service call does not automatically preserve that property, even if both individual services are reliable.

If an adapter uses an atomic fence-floor advance at grant, define what happens when that publication is incomplete. A coordinator must not announce the new owner as able to act while some required receivers still accept the old floor. The protocol can wait, hold affected actions or use a single authoritative gateway, but it must expose the partial-publication state instead of assuming notifications completed everywhere.

An unavailable protected receiver also changes the recovery path. Ownership can be successfully granted while the business service remains unreachable. That does not justify making another owner or issuing a fresh operation identity. Retain the current grant context, exact intent and unresolved submission state, then follow the receiver's documented retry and reconciliation contract when connectivity returns.

Be explicit about read behavior too. Some diagnostic reads can tolerate stale data, while an action-admission read may require current authoritative state. Using the same cache or observation path for both can hide a safety dependency. Label the purpose of each read and keep an operational dashboard's approximate owner display separate from the protected service's actual decision.

A recovery drill should retain the old owner's attempted action as evidence without letting the attempt alter the protected record. Record its original grant identity, intended operation and rejection category in a permitted diagnostic form. Do not rewrite the attempt to the newest epoch after rejection, because that would erase the distinction between an obsolete request and a newly admitted action.

The model's transaction scope also limits its durability claim. It closes local connections and rolls back unfinished local transactions, but it does not exercise filesystem failure, replicated recovery or disaster restoration. A production runbook needs evidence for those layers, particularly when an epoch counter and operation ledger must survive longer than any individual worker process.

For a controlled regional test, pause the old actor rather than terminating it in every scenario. Killing it permanently can make the unsafe path disappear from the experiment. A useful negative lets the actor resume after ownership changes and proves that the receiver still rejects its stale request, including the case before the replacement has performed a first write.

Finally, preserve the reason for a safety hold in the handoff record. Authority unavailable, expired owner, changed business state and unknown remote completion are different situations. An operator who sees only failover failed may bypass the wrong boundary. A precise reason code and immutable evidence reference make recovery possible without broadening a stale actor's authority or inventing a completed outcome.

---

Frequently asked questions

Does a lease guarantee the old worker stopped?

No. A paused or disconnected process can resume after its lease expires. The protected mutation must check current authority or a correctly published fence at write time. The fixture deliberately lets old code continue and verifies that its commit is rejected.

Why reject an expired owner before another owner exists?

The grant's validity interval ended, so the old actor no longer has authority under this contract. Waiting for a new owner to write would leave an unauthorized interval. The model checks expiry against current authoritative time even before any regrant occurs.

Is highest token seen enough for fencing?

It can miss an old write arriving after regrant but before the new owner's first write. The receiver has not yet seen the new token. Publish the floor at grant or consult current authoritative ownership at every protected mutation, as the local model does.

When does the new owner become able to act?

In the model, grant issuance and publication of the new owner/fence occur in one transaction. Only after that transaction completes can the new grant pass the protected check. Production adapters need an equally explicit authoritative boundary rather than relying on notification timing.

Can two workers use the same epoch?

The epoch must also bind the exact owner and task scope. The fixture gives another owner the current numeric epoch and requires rejection. A bare number without owner and authority identity is insufficient for the modeled permission check.

Can renewal revive an expired grant?

The model rejects renewal at or after expiry. The same actor can claim again only through a new grant with a newer epoch. That new generation does not retroactively authorize work submitted under the expired grant.

Should the operation key include the owner epoch?

Keep business-operation identity independent of changing ownership metadata. A new owner recovering the same exact intent should reuse its original operation identity under the provider's contract. Including the epoch can cause failover to create another logical mutation instead of a safe retry.

Can the new owner retrieve an old result?

A current owner can receive the historical receipt for an exact matching operation without another mutation. The method checks current ownership first. Stale or expired owners receive no receipt through that boundary, and changed payloads remain conflicts.

Does current ownership establish user authorization?

No. It identifies the permitted task writer in this model. The exact principal, action, resource and caller-approved values still need their owning authorization workflow. A service credential or leadership grant must not silently substitute for those business checks.

What happens if authority is unreachable?

The model holds the action and returns authority unavailable without mutation or receipt access. A production adapter must retain the unresolved operation and recover through its declared protocol. Cached ownership belief is not a safe substitute for the required current check.

Does the unavailable fixture simulate a regional partition?

It is a local control flag that tests the fail-closed branch. It does not exercise quorum, network delay or a real coordination service. Regional partition behavior needs separate controlled infrastructure and protected-adapter evidence.

Can a local fence protect a provider that ignores it?

No. The downstream service must enforce the fence or every protected submission must pass through an authoritative gateway with a documented effect protocol. A token header or local log entry alone cannot constrain an unrelated provider's mutation path.

Does a gateway create distributed exactly-once behavior?

It can enforce a local submission boundary, but an external provider can still commit while its response is lost. Preserve exact operation identity and reconcile using documented provider semantics. The local model makes no distributed exactly-once claim.

What if the old owner returns after an await?

It must recheck current ownership, lease validity and relevant business state immediately before the protected action. The fixture pauses a held owner, grants a newer epoch and rejects the old commit after release. Successful computation does not preserve authority through waiting.

Does fencing retract audio already buffered remotely?

No. Business write fencing and media playback are different boundaries. Apply current generation and cancellation checks before new speech submission, then use provider-specific evidence for audio already accepted. A local ownership transition does not prove that buffered speech stopped.

Why include an authority realm?

An independent authority instance can issue similar numeric epochs, so grants need an authority identity too. The fixture rejects a grant from another newly initialized database realm. This does not solve restoring an old database with the same realm and counter.

Is backup restoration safe in the local model?

The model does not prove epoch monotonicity across restoring old authority state. The production protocol must preserve or advance the authority generation and update protected receivers appropriately. The emitted report labels restore safety as an external obligation.

What does the SQLite transaction establish?

It atomically combines current local ownership, expiry, operation history and business-version mutation in one authoritative store. Connections close on success and exceptions. It does not establish regional consensus, remote provider completion or another database's isolation behavior.

How should failover metrics count held actions?

Keep them distinct from confirmed business failures and successes. A safety hold means authority was unavailable or invalid, while a previously submitted effect can still be unknown. Define the denominator over admitted intents and retain unresolved outcomes rather than treating continued conversation as task completion.

Which negative test should I add first?

Reject the expired owner before any regrant, then reject the old owner after regrant but before the new owner's first write. Those two cases expose lease-belief and highest-seen-token gaps directly. Add held-after-await and exact-operation recovery fixtures before testing deployment-scale behavior.

Recover with one auditable action owner

Write down which authority issues ownership, when its fence becomes effective and where every protected mutation checks it. Keep expiry, renewal and new grants explicit. An old process can continue running, so safety must come from the action boundary rather than an assumption that the process stopped.

Run the complete local fixture, then test the actual issuer and downstream adapter under controlled faults. Preserve exact business intent across owner changes, reject changed payloads under old operation identities and retain unknown remote outcomes for reconciliation. Treat backup restoration and unsupported provider fencing as concrete integration obligations.

For 60db Agent integrations, separate a reachable replacement region from a verified current action owner. The booking idempotency contract and shutdown accounting contract supply the adjacent recovery rules. Make the final failover claim match the authority, media and business evidence actually observed.