Out-of-order and duplicate voice webhooks

TL;DR: Verify a webhook using the provider's exact signed representation, then project validated events under a documented entity-revision contract. Delivery identity, causal ordering, terminal state and business authorization are separate checks; a valid signature or a short replay cache does not replace them.

---

A call-ended event arrives before a delayed call-active event. Your handler accepts both requests, updates one row twice, and leaves a finished conversation looking active. A retry can also create a second follow-up task if the handler treats every delivery as a new business instruction.

Start with the state you must preserve. Retain authenticated delivery evidence, distinguish event identity from entity revision, and make the projection decision atomic with its delivery ledger. Put business effects behind their own authorization and operation identity, as explained in the business-state conflict boundary.

Stripe's webhook guidance explicitly warns about duplicate deliveries and unordered events. That is a provider contract, not permission to invent revision numbers for a different voice API. The reference program below accepts synthetic full-state events with an explicitly defined revision field; real integrations need an adapter that can establish the equivalent semantics.

The useful output is a durable local projection plus classified dispositions. It is not a Twilio validator, Stripe SDK replacement, global ordering service or authorization engine for 60db Agent. No live provider delivery or remote business action is performed by the example.

Table of contents

1. Separate delivery identity from entity state
2. Verify the representation the provider signs
3. Define causal ordering before comparing revisions
4. Make deduplication and projection one transaction
5. Preserve terminal state and conflict evidence
6. Run a bounded webhook projector
7. Connect acknowledgment to durable ownership
8. Diagnose failures without replaying business actions
9. Frequently asked questions
10. Validate the boundary you actually control

Separate delivery identity from entity state

A delivery is an attempt to carry an event to your endpoint. An event describes something that happened or a state observed elsewhere. An entity is the call, recording, task or other object whose local projection you maintain. These identities answer different questions and should remain separate in storage and traces.

For example, a retry of one event should retain its event identity even if a transport request identifier changes. Several distinct events can refer to the same call. One call can also produce several kinds of evidence whose ordering and authority differ, such as signaling state and recording status.

The CloudEvents specification defines uniqueness using the combination of source and event ID. It does not make that ID an entity revision or promise a total order over business actions. A lexicographically larger event ID is not automatically a newer state.

Bind your deduplication key to a trustworthy namespace. A provider account, registered endpoint or documented event source may form part of that namespace, but it must come from the verified adapter contract. Letting untrusted body text choose another tenant's namespace defeats the purpose of scoped identities.

The teaching artifact has one trusted endpoint and one synthetic source namespace. Its signed body carries that source because the fixture owns both sides of the protocol. A production adapter must derive or validate source/account membership against the verified endpoint configuration before accepting the normalized event.

Retain the original event identity and admitted raw-body digest. A changed payload under an existing identity is a conflict, even when its new state happens to look reasonable. Silently replacing the original evidence makes a later investigation unable to distinguish a genuine retry from inconsistent source data.

Task identities across redials introduce another boundary. A call ID can help trace the conversation, while a booking intent must remain stable across several calls. Neither webhook event identity nor a new call attempt should create a fresh business operation automatically.

Treat account, entity and operation identifiers as opaque values within their declared domains. Avoid lowercasing, trimming or numeric conversion without the owner's documented rules. The same preservation principle appears in order-ID and email capture, where presentation convenience must not rewrite the canonical object.

Write these roles into the schema before choosing a queue. A queue can preserve delivery order for the records it sees and still carry events whose source causality differs from arrival order. Explicit identity and state semantics make that distinction reviewable before a retry storm exposes it.

Verify the representation the provider signs

Signature verification must operate on the representation named by the provider. Parsing JSON, changing whitespace, rebuilding a URL or normalizing form fields can alter that representation. Preserve the necessary original bytes and request metadata before generic middleware transforms them.

Twilio's webhook security documentation describes validation involving the documented URL and request parameters, with different details for payload forms. Use its supported validator for the selected webhook type. A generic HMAC over a JSON body is not a drop-in Twilio implementation.

Stripe's signed webhook verification requires the unmodified request body. That requirement does not mean every other provider signs the same input or uses the same algorithm. Keep provider selection explicit at the endpoint or adapter boundary rather than trying several validators until one happens to accept.

The example uses a deliberately synthetic envelope. Its message combines an endpoint label, an integer timestamp and the raw body with fixed delimiters. That format exists only to demonstrate the sequencing rule: authenticate the supplied bytes before decoding the event and making projection decisions.

The fixed endpoint label contains printable ASCII without whitespace, so the delimiter cannot be smuggled through that component. The raw body is bounded separately. A real adapter needs its own documented limits for header size, body size, encoding, content type, URL reconstruction and trusted proxy handling.

Python's HMAC documentation supplies compare_digest for digest comparison. That function does not choose the right signed message, authorize a key, rotate credentials or prevent replay on its own. Those obligations remain outside a comparison helper.

Keep timestamp validation distinct from signature validity. A correctly signed old message can still be outside your admitted freshness window. Conversely, a current timestamp attached to a tampered message must not make its payload trustworthy. The sample reports these failures separately so a test can identify which boundary rejected the envelope.

The synthetic window accepts its exact lower boundary and rejects future timestamps. Those are teaching-policy choices, not copied provider defaults. Production tolerances must follow the provider's signature contract and your clock-management assumptions; do not transfer the sample's number into a deployment configuration by habit.

A signature also says nothing about the caller's permission to modify a customer record. Keep the caller authentication boundary and per-action CRM authorization independent. An authentic service notification can update evidence without authorizing a new financial, scheduling or disclosure action.

Store keys in the deployment's credential mechanism and keep them out of voice responses and diagnostic payloads. The hardcoded key in the program is visibly synthetic fixture material. It cannot authenticate a real provider, and its presence should never become a pattern for embedding production credentials in article-derived code.

Define causal ordering before comparing revisions

A revision comparison is useful only when the source defines what the revision orders. The sample assumes a strictly increasing integer revision for full-state snapshots of one source/entity pair. It does not infer causality from receipt time, a random event identifier or an arbitrary timestamp field.

Full-state snapshots matter. If an event says the complete current state is ended, a later-arriving older active snapshot can be ignored safely under this contract. If events instead describe incremental changes, skipping an older delta may lose required information. A delta stream needs its own gap, dependency and reconciliation policy.

The accepted states here are ringing, active, ended and failed. The program does not attempt to encode every carrier or provider call state. An adapter must map supported source states deliberately, preserve unmapped evidence and avoid treating an unknown source status as active merely to keep a pipeline moving.

There is no requirement that every intermediate revision arrive. A higher full snapshot can replace a lower full snapshot. That simplification is declared and tested. If your application requires a complete timeline, the latest projection is insufficient; retain an append-only evidence stream and define how incomplete history affects downstream decisions.

A missing revision takes the reconciliation path. The program records that delivery without changing the projection. An absent field cannot be converted into zero, arrival position or a locally invented increment, because doing so would present transport timing as source authority.

The stale-result article applies the same principle to business reads: freshness must be meaningful at the mutation boundary. A timestamp that describes when a callback was sent does not necessarily identify the version of the business object you plan to update.

Do not compare revisions across unrelated source namespaces or entity kinds. A recording revision and a call revision may advance independently even when they share a conversation. Keep their projections separate unless the public provider contract defines a causal relationship you can enforce.

Validate the revision's type and supported range before touching storage. Python booleans are integers for some operations, but true is not revision one in this schema. The sample also rejects integers outside the SQLite signed-integer boundary instead of discovering the problem after opening a transaction.

The timestamp precision boundary explains another tempting shortcut. Even accurately retained timestamps do not establish an order that the source never promised. Preserve their original meaning while using the documented revision or reconciliation mechanism for state decisions.

Make deduplication and projection one transaction

Deduplication performed before a separate state update leaves a crash window. The process can mark an event seen and stop before changing the projection. Reversing the order creates another window in which the projection changes but a retry is treated as unseen.

Put both decisions under the same authoritative local transaction. In the sample, BEGIN IMMEDIATE serializes writers to one SQLite database while the handler checks delivery identity, inspects the current entity revision and records the resulting disposition. A commit makes those local decisions visible together.

Python's sqlite3 documentation distinguishes transaction behavior from connection lifetime. The example uses explicit commits and rollbacks and wraps each owned connection with closing. Exiting a transaction context alone should not be described as closing the database handle.

The delivery table stores a source/event key, raw-body digest and original disposition. The projection table stores one source/entity snapshot. These records answer different audit questions: which bytes were admitted under an event identity, and which entity snapshot currently wins under the declared ordering rule? A separate conflict table retains the original and incoming digests plus disposition without replacing the first delivery record.

An exact delivery retry returns duplicate. It does not repeat the first mutation. The original disposition remains in the ledger, so a duplicate of a previously stale or pending event cannot be mistaken for a newly accepted projection simply because its current response says duplicate.

A raw-body digest is a compact equality check for admitted bytes. It is not a replacement for storing permitted forensic evidence where required, nor a promise that hashes anonymize customer data. Apply the telemetry redaction boundary before exporting identifiers, bodies or derived fingerprints to broader sinks.

The model accepts only a small fixed JSON object. Duplicate keys, unknown fields, unsupported states and nonfinite constants are rejected before storage. A decoder nesting failure is also classified as invalid schema; a byte-size limit alone does not guarantee that every supported decoder can traverse the document. This prevents a parser's last-key-wins behavior from silently selecting different identities or revisions in different components of the same pipeline.

For a production relational adapter, read the actual isolation and retry contract. PostgreSQL's transaction-isolation documentation describes concurrent transaction behavior and serialization failures. Copying SQLite syntax into another database does not transfer its writer serialization or eliminate the need to retry an entire failed transaction correctly.

A database transaction protects only the state it owns. It cannot atomically commit an unrelated CRM write or carrier operation unless the external system participates in an appropriate protocol. Regional ownership and fencing addresses that additional boundary; a local delivery ledger alone does not decide which regional worker may act remotely.

The table below uses ✓ for a check enforced by the named mechanism under its stated assumptions, and ✗ for a check that mechanism does not supply. These are contract properties, not measured production reliability scores.

Preserve terminal state and conflict evidence

Terminal state is a lifecycle rule, not just a revision comparison. Once this model records ended or failed for an entity, a higher nonterminal event cannot reopen it. A provider that legitimately reuses object identities or supports reopening needs a different explicit lifecycle contract.

Ordering checks still come first. Revision two arriving after terminal revision three is stale, not a new attempt to alter the terminal state. Classifying it as a terminal conflict would obscure ordinary delayed delivery and inflate the category of inconsistent source behavior.

An equal revision with the same normalized state is a duplicate revision, even if it has a different event ID. An equal revision with a different state is a revision conflict. Both outcomes retain delivery evidence without mutating the established projection.

A higher revision after a terminal state is conservatively classified as a terminal conflict in this model, including another terminal label. That choice keeps the teaching lifecycle small. Production code must document whether terminal metadata can advance, whether a correction event is authoritative, and how a genuinely new call obtains a new identity.

The shutdown semantics contract provides a related rule: historical success or cached work must not restore canceled activity. Here, a retried callback can return historical evidence while the current terminal projection remains unchanged. A duplicate response is never a command to resume a call.

Conflicts deserve a distinct reconciliation queue or operator workflow. Do not repair them by rewriting the old payload under the same event key. Preserve the first admitted evidence, the conflicting evidence permitted by your retention policy, the adapter version and the specific comparison that failed. The artifact durably records each distinct conflicting body digest in the same transaction and exposes it through conflicts() and the emitted report. Repeating the same conflicting bytes retains one evidence row; this is not an arrival-count audit.

A short in-memory replay cache cannot supply that history after eviction or restart. It may reduce repeated verification work under a carefully scoped policy, but business deduplication must last for the relevant operation lifetime and survive the failure modes named in the design.

Avoid a blanket claim that every duplicate is harmless. A handler that sends an email, books a slot or starts a recording on each delivery can create repeated effects even if its final call-state row looks correct. Give each authorized business intent its own stable operation identity and durable outcome evidence.

The recording-consent state boundary is especially important when a callback updates capture evidence. A signed notification about a recording leg does not independently establish continuing permission for another purpose, and a delayed start acknowledgment must not override a newer stop decision.

Run a bounded webhook projector

Save the complete fence as voice_event_projector.py and run it with Python 3.12 or later. It creates temporary local databases, runs its assertions and emits a synthetic report. The fixtures contain no customer bodies, real endpoint URLs, provider credentials or business identifiers.

The artifact trusts its configured key and endpoint, the test clock and the declared source revision semantics. Its JSON vocabulary is intentionally narrow. It does not admit arbitrary nested payloads, validate every possible Unicode identifier or implement a production HTTP server.

The envelope is frozen and carries immutable bytes. Decoding creates a fresh local dictionary; none of that mutable object escapes into approved state. SQLite stores the selected scalar values, so later mutation of a caller's event-building dictionary cannot change an already committed projection.

The output explicitly states that business permission was not granted, provider signature validation was not tested and remote effects were not attempted. These fields keep a reusable fixture report from looking like a live integration receipt when someone copies it into an incident note.

from contextlib import closing
from dataclasses import dataclass
import hashlib
import hmac
import itertools
import json
from pathlib import Path
import sqlite3
import tempfile

# Synthetic teaching protocol, not a provider signature implementation.
@dataclass(frozen=True)
class Evidence:
    endpoint: str
    timestamp: int
    raw: bytes
    signature: str


def signed(key, endpoint, timestamp, raw):
    message = endpoint.encode('ascii') + b'\n' + str(timestamp).encode('ascii') + b'\n' + raw
    return Evidence(endpoint, timestamp, raw,
                    hmac.new(key, message, hashlib.sha256).hexdigest())


def object_pairs(pairs):
    result = {}
    for key, value in pairs:
        if key in result:
            raise ValueError('duplicate key')
        result[key] = value
    return result


def text(value):
    return type(value) is str and 0 < len(value) <= 80 and value.isascii() and all(33 <= ord(c) <= 126 for c in value)


class Projector:
    def __init__(self, path, key, endpoint):
        if type(key) is not bytes or len(key) < 16 or not text(endpoint):
            raise ValueError('trusted configuration')
        self.path, self.key, self.endpoint = str(path), bytes(key), endpoint
        with closing(sqlite3.connect(self.path)) as db:
            db.executescript('''
                CREATE TABLE IF NOT EXISTS delivery(
                    source TEXT, event_id TEXT, raw_digest TEXT,
                    disposition TEXT, PRIMARY KEY(source,event_id));
                CREATE TABLE IF NOT EXISTS identity_conflict(
                    source TEXT, event_id TEXT, original_digest TEXT,
                    conflicting_digest TEXT, disposition TEXT,
                    PRIMARY KEY(source,event_id,conflicting_digest));
                CREATE TABLE IF NOT EXISTS projection(
                    source TEXT, entity TEXT, revision INTEGER,
                    state TEXT, PRIMARY KEY(source,entity));
            ''')
            db.commit()

    def admit(self, evidence, now):
        if type(evidence) is not Evidence or type(now) is not int or now < 0:
            return 'invalid_envelope'
        if evidence.endpoint != self.endpoint or type(evidence.timestamp) is not int or evidence.timestamp < 0:
            return 'invalid_envelope'
        if type(evidence.raw) is not bytes or not 0 < len(evidence.raw) <= 4096 or type(evidence.signature) is not str:
            return 'invalid_envelope'
        expected = signed(self.key, self.endpoint, evidence.timestamp, evidence.raw).signature
        if not evidence.signature.isascii() or not hmac.compare_digest(expected, evidence.signature):
            return 'invalid_signature'
        if not 0 <= now - evidence.timestamp <= 30:
            return 'invalid_timestamp'
        try:
            event = json.loads(evidence.raw.decode('utf-8'), object_pairs_hook=object_pairs,
                               parse_constant=lambda _: (_ for _ in ()).throw(ValueError('nonfinite')))
            if type(event) is not dict or set(event) != {'source','id','entity','revision','state','schema'}:
                raise ValueError('schema')
            if not all(text(event[k]) for k in ('source','id','entity')):
                raise ValueError('identity')
            if type(event['schema']) is not int or event['schema'] != 1:
                raise ValueError('version')
            rev = event['revision']
            if rev is not None and (type(rev) is not int or not 1 <= rev <= 2**63-1):
                raise ValueError('revision')
            if type(event['state']) is not str or event['state'] not in {'ringing','active','ended','failed'}:
                raise ValueError('state')
        except (ValueError, UnicodeError, TypeError, RecursionError):
            return 'invalid_schema'
        source, identity, entity, state = (event[k] for k in ('source','id','entity','state'))
        digest = hashlib.sha256(evidence.raw).hexdigest()
        with closing(sqlite3.connect(self.path, timeout=5, isolation_level=None)) as db:
            try:
                db.execute('BEGIN IMMEDIATE')
                prior = db.execute('SELECT raw_digest FROM delivery WHERE source=? AND event_id=?',
                                   (source, identity)).fetchone()
                if prior:
                    result = 'duplicate' if prior[0] == digest else 'identity_conflict'
                    if result == 'identity_conflict':
                        db.execute('INSERT OR IGNORE INTO identity_conflict VALUES(?,?,?,?,?)',
                                   (source, identity, prior[0], digest, result))
                else:
                    current = db.execute('SELECT revision,state FROM projection WHERE source=? AND entity=?',
                                         (source, entity)).fetchone()
                    if rev is None:
                        result = 'pending_reconciliation'
                    elif current and rev < current[0]:
                        result = 'stale'
                    elif current and rev == current[0]:
                        result = 'duplicate_revision' if state == current[1] else 'revision_conflict'
                    elif current and current[1] in {'ended','failed'}:
                        result = 'terminal_conflict'
                    else:
                        result = 'accepted'
                        db.execute('INSERT INTO projection VALUES(?,?,?,?) ON CONFLICT(source,entity) '
                                   'DO UPDATE SET revision=excluded.revision,state=excluded.state',
                                   (source, entity, rev, state))
                    db.execute('INSERT INTO delivery VALUES(?,?,?,?)', (source, identity, digest, result))
                db.commit()
                return result
            finally:
                if db.in_transaction:
                    db.rollback()

    def conflicts(self):
        with closing(sqlite3.connect(self.path)) as db:
            return tuple(db.execute('SELECT source,event_id,original_digest,conflicting_digest,disposition '
                                    'FROM identity_conflict ORDER BY source,event_id,conflicting_digest'))

    def snapshot(self):
        with closing(sqlite3.connect(self.path)) as db:
            return tuple(db.execute('SELECT source,entity,revision,state FROM projection ORDER BY source,entity'))


def body(identity='event-a', revision=1, state='ringing', **updates):
    event = dict(source='synthetic-source', id=identity, entity='synthetic-call',
                 revision=revision, state=state, schema=1)
    event.update(updates)
    return json.dumps(event, separators=(',', ':'), allow_nan=False).encode()


def main():
    key = b'synthetic-test-key-only'
    endpoint = 'synthetic-endpoint'
    def envelope(raw, timestamp=100):
        return signed(key, endpoint, timestamp, raw)
    with tempfile.TemporaryDirectory() as directory:
        model = Projector(Path(directory)/'events.db', key, endpoint)
        raw = body()
        good = envelope(raw)
        assert model.admit(good, 100) == 'accepted'
        assert model.admit(good, 100) == 'duplicate'
        changed = body(state='active')
        assert model.admit(envelope(changed), 100) == 'identity_conflict'
        expected_conflicts = (('synthetic-source','event-a',hashlib.sha256(raw).hexdigest(),
                               hashlib.sha256(changed).hexdigest(),'identity_conflict'),)
        assert model.conflicts() == expected_conflicts
        assert model.admit(envelope(changed), 100) == 'identity_conflict'
        assert model.conflicts() == expected_conflicts
        altered = Evidence(endpoint,100,raw+b' ',good.signature)
        assert model.admit(altered,100) == 'invalid_signature'
        assert model.admit(envelope(raw,69),100) == 'invalid_timestamp'
        assert model.admit(envelope(raw,101),100) == 'invalid_timestamp'
        assert model.admit(envelope(raw,70),100) == 'duplicate'
        assert model.admit(Evidence('wrong-endpoint',100,raw,good.signature),100) == 'invalid_envelope'
        assert model.admit(envelope(b'{"schema":1,"schema":1}'),100) == 'invalid_schema'
        nested = b'[' * 1500 + b']' * 1500
        assert len(nested) < 4096
        assert model.admit(envelope(nested),100) == 'invalid_schema'
        assert model.admit(envelope(body(revision=True)),100) == 'invalid_schema'
        assert model.admit(envelope(body(revision=2**63)),100) == 'invalid_schema'
        assert model.admit(envelope(body(schema=True)),100) == 'invalid_schema'
        assert model.admit(envelope(body(extra='unadmitted')),100) == 'invalid_schema'
        assert model.admit(envelope(body('missing',None)),100) == 'pending_reconciliation'
        assert model.admit(envelope(body('end',3,'ended')),100) == 'accepted'
        assert model.admit(envelope(body('older',2,'active')),100) == 'stale'
        assert model.admit(envelope(body('equal',3,'active')),100) == 'revision_conflict'
        assert model.admit(envelope(body('terminal-copy',3,'ended')),100) == 'duplicate_revision'
        assert model.admit(envelope(body('resurrection',4,'active')),100) == 'terminal_conflict'
        restarted = Projector(Path(directory)/'events.db',key,endpoint)
        assert restarted.snapshot() == model.snapshot()
        assert restarted.conflicts() == expected_conflicts
        permutations = 0
        events = [body('one',1,'ringing'),body('two',2,'active'),body('three',3,'ended')]
        for index, order in enumerate(itertools.permutations(events)):
            p = Projector(Path(directory)/f'permutation-{index}.db',key,endpoint)
            for value in order:
                p.admit(envelope(value),100)
                assert p.admit(envelope(value),100) == 'duplicate'
            assert p.snapshot() == (('synthetic-source','synthetic-call',3,'ended'),)
            permutations += 1
        print(json.dumps({'fixture_class':'Synthetic example', 'permutations_checked':permutations,
                          'projection':model.snapshot(), 'identity_conflicts':model.conflicts(),
                          'business_permission':'not_granted',
                          'provider_signature_validation':'not_tested', 'remote_effect':'not_attempted'},indent=2))
    print('All local webhook projection fixtures passed')


if __name__ == '__main__':
    main()

The permutation fixture uses all orderings of three synthetic full-state events and repeats each delivery. It checks the final projection under the stated revision semantics. This is stronger than demonstrating one convenient arrival sequence, but it is still a finite local state test rather than a claim about every provider event type.

The timestamp assertions cover the admitted boundary, an expired message and a future message. The altered-whitespace assertion reuses the original signature over changed bytes and must fail verification. Re-signing changed bytes is a different case, tested separately as changed identity reuse.

A reopened Projector reads the same committed local state from disk. That demonstrates persistence across object reconstruction within the fixture. It does not simulate power failure, filesystem corruption, database replication, backup restoration or regional failover; those require their own environment and evidence.

The regression-testing method helps organize additional fixtures. Keep parser failures, signature failures, projection conflicts and external-action outcomes in separate categories so a green count cannot conceal that an entire adapter boundary was never exercised.

Connect acknowledgment to durable ownership

An HTTP response acknowledges something specific. Decide whether it means verified receipt, durable enqueue, committed projection or completed business action. Using the same success response for all these stages can be valid transport behavior, but the application must retain their distinct internal evidence.

If you acknowledge before durable ownership, a crash can lose work that the sender believes was accepted. If you wait for every remote action before acknowledging, delivery retries can pile up behind a slow or unavailable dependency. The right boundary depends on the provider's retry policy and your durable processing design.

A common contract is to validate, durably record or enqueue admitted evidence, then acknowledge transport acceptance. A separate worker projects that evidence and performs independently authorized operations. This is a design option, not an implementation provided by the local projector.

The admission-control guide explains why capacity decisions belong before accepting responsibility you cannot retain. During overload, distinguish rejecting a request from accepting it into durable storage. Dropping work after returning success should not be described as a successful bounded queue.

A queue acknowledgment and a database commit are also separate operations unless the chosen infrastructure supplies a shared transaction boundary. Design crash recovery for the interval between them. Idempotent projection makes repeated processing safe locally; it does not make every acknowledgment ordering correct automatically.

During deployment, close new-work admission while allowing already-owned work to finish or leave an explicit unresolved record. Rolling conversation drains apply the same ownership accounting to active sessions. A process exit is not evidence that every callback or external action completed.

Do not use an event's receipt time as a substitute for business deadlines. A signed callback can arrive after a user canceled an intent or after an authorization epoch changed. The action gate must enforce the current policy even if the event was authentic when originally produced.

Retain enough provenance to reconcile unresolved work: source namespace, event identity, entity identity, schema/adapter version, original disposition and relevant authoritative revision. Keep secrets and unnecessary body content out of broad logs. Evidence completeness and privacy are separate requirements that should be designed together.

Diagnose failures without replaying business actions

Begin with the failed boundary rather than the visible symptom. A signature failure points toward signed-input preservation, key selection or provider configuration. An invalid schema points toward adapter assumptions. A stale event describes an ordering result; it does not necessarily indicate a provider outage.

A developer's historical Stripe CLI ordering report asks why events arrive in a different order. It is useful question evidence, while current provider documentation owns the delivery guarantee. The thread is not a measurement of current voice-webhook disorder or a benchmark for this program.

Record state before and after admission, together with the returned disposition. If the projection changed on a duplicate, inspect whether a side effect lives outside the transaction or whether another code path bypasses the projector. A final database row alone can hide repeated notifications or external writes.

Use synthetic identifiers and permitted payload captures to reproduce parser and ordering behavior. Do not replay production business actions just to confirm a suspected duplicate bug. A diagnostic replay should have an explicit effect boundary and return an unambiguous not-attempted status for remote mutations.

The component-regression method separates correlation from cause. A rise in repeated callbacks after a deployment does not prove the provider changed behavior; altered acknowledgment timing, proxy rewriting or handler errors can also change what your receiver observes.

For conversation diagnostics, keep signaling state separate from media evidence. The audio-boundary guide concerns actual audio framing, and the turn-detection guide concerns conversational timing. Neither can be inferred from an ended webhook alone.

Create explicit runbook branches for invalid signature, invalid schema, stale, identity conflict, revision conflict and pending reconciliation. Each branch should state whether automated retry is useful, what evidence is retained and which authoritative system can resolve the uncertainty. Avoid a generic retry button that silently creates another business intent.

A reconciliation read introduces its own race. Suppose a worker notices a missing revision, fetches the authoritative object, and waits while a newer callback is projected. When the read returns, it must not blindly overwrite the newer local state. Its adapter needs an authoritative version, a conditional update or a documented merge rule that remains valid at commit time.

Do not attach the missing event's old identity to a newly fetched snapshot as if the original event contained those values. Keep the original delivery evidence and the reconciliation observation separate. Record which authoritative read supplied the replacement evidence, what version it observed, and which local projection version was present when the repair was admitted.

If the source cannot expose an orderable version, choose a weaker explicit policy. You might keep the projection unresolved until a designated owner supplies an authoritative decision. You might maintain several independently observed facts instead of forcing them into one status field. What matters is that the uncertainty remains visible rather than being hidden by a convenient last-write-wins assignment.

Schema evolution also needs a deliberate boundary. A new optional provider field may be harmless to a supported SDK but invalid in the teaching schema, which intentionally accepts exactly six fields. A production adapter can preserve additional raw evidence while normalizing only its supported projection vocabulary. Version that adapter and test the conversion before extending the local model's accepted shape.

Separate malformed evidence from unsupported evidence in an operational implementation when the recovery paths differ. A broken JSON document may require transport diagnosis, while a valid provider event of an unhandled type may need a new adapter capability. The sample groups several schema failures under one safe disposition to keep the model bounded; it does not claim that one reason code is enough for every production runbook.

A useful trace records the local transaction's identity and outcome without confusing its duration with provider delivery delay. The latency measurement guide shows why an interval needs explicit endpoints. Time spent waiting for a receiver transaction, time since a provider timestamp and time until a caller hears a response are different measurements with different clock assumptions.

Count duplicate deliveries against an explicit denominator. Repeated events divided by total received deliveries answers a transport question; repeated business mutations divided by admitted business intents answers a correctness question. A system can receive many repeated deliveries while making no repeated mutation, or receive few repeats while one unsafe handler creates a serious business error.

The cost-per-success method is useful when retries consume verification, storage or worker capacity. Retain the unit of work behind each cost estimate and keep unknown business outcomes outside the confirmed-success denominator. Synthetic fixture counts should never be promoted into a claim about provider traffic, customer conversion or production reliability.

Keep recovery tools subject to the same authorization rules as normal handlers. An operator choosing to reconcile an event should not gain an unrestricted ability to execute its suggested business action. The tool can display permitted evidence, fetch current state and propose a bounded correction while the action service still checks current principal, resource and immutable operation values.

Finally, test retention changes against real recovery requirements. Removing old delivery evidence may be valid under a declared data policy, but it changes which retries can be recognized locally. Document whether the business operation ledger remains authoritative after transport evidence is purged, and ensure an old event cannot create a fresh operation simply because one diagnostic table has been emptied.

---

Frequently asked questions

Why can voice webhooks arrive out of order?

Independent delivery attempts, retries and processing paths can produce an arrival order different from source causality. Use the selected provider's documented ordering contract. The reference model requires explicit full-state entity revisions and does not infer order from event IDs or receipt times.

Is an event ID a revision number?

Only when the provider explicitly defines it that way. CloudEvents source-plus-ID identifies an event, while this artifact uses a separate entity revision to choose the projected snapshot. Random or lexical IDs must not be treated as causal sequence numbers.

Should I verify a signature before parsing JSON?

Preserve the signed representation and run the provider's required verification before trusting decoded event fields. The exact signed input varies by provider and payload form. The synthetic fence demonstrates this sequence without claiming compatibility with Twilio or Stripe.

Can I rebuild JSON before verifying it?

Rebuilding can change whitespace, encoding or property order and therefore change signed bytes. Preserve the original body wherever the provider requires it. Parsing and schema validation belong after the appropriate verification boundary, with middleware behavior checked in the actual deployed stack.

Does Twilio use the same signature input as Stripe?

No universal shared input should be assumed. Twilio's documented validation includes request URL and parameters according to payload form, while Stripe requires its unmodified body for webhook verification. Use the correct supported adapter and test its exact deployed configuration.

What does a valid webhook signature authorize?

It authenticates the documented signed message under the configured verification contract. It does not establish the caller's identity, current CRM permission or approval for another business action. Keep event admission and resource authorization as independent checks with their own evidence.

How should an exact duplicate be handled?

Bind source and event ID to the original admitted-body digest, then return a duplicate disposition without repeating projection mutation. Preserve the first ledger disposition for audit. Any business effect still needs a stable operation identity and its own current authorization.

What if the same event ID carries different data?

Treat it as identity conflict under this contract. Preserve the original record and route permitted conflicting evidence to reconciliation. The model stores the original digest, incoming digest and identity-conflict disposition separately, and its getter retains that evidence after reopening the database. Overwriting the first body would erase the distinction between an exact retry and an inconsistent identity reuse.

What if two event IDs share one revision?

The model compares their normalized state for the same source/entity pair. Identical state is a duplicate revision; different state is revision conflict. A richer production snapshot must compare every field that participates in its declared projection, not merely a convenient status label.

Why check stale revisions before terminal conflicts?

An older active snapshot after a newer ended snapshot is ordinary stale evidence under the model's ordering rule. It should not be reported as a new terminal-state violation. The projection remains terminal in either case, but distinct classifications support better diagnosis.

Can a terminal call reopen with a higher revision?

This teaching lifecycle rejects every higher revision after ended or failed. A provider supporting authoritative terminal corrections or reopening needs a separately documented rule and fixtures. Do not silently add such behavior because a delayed event appears to suggest activity.

What if the provider supplies no revision?

Record the delivery as pending reconciliation and avoid guessing its place in source history. Use an authoritative read or another documented mechanism to establish the state. Arrival order and local counters cannot manufacture a missing causal guarantee.

Is a replay-cache TTL enough for deduplication?

A short cache can reduce repeated work, but eviction or restart removes its memory. Durable business deduplication must cover the operation's actual lifetime and recovery requirements. The sample's signed-message freshness window is separate from its persistent delivery ledger.

Does the SQLite transaction provide exactly-once delivery?

It provides a local atomic decision for delivery evidence and projection state in one database. It does not control network delivery or unrelated remote effects. A sender may deliver again, and a remote provider may require its own idempotency and reconciliation protocol.

Why reject duplicate JSON keys?

Different parsers or stages can choose different values when an object repeats a key. The fixed schema rejects that ambiguity before storage. It also rejects unknown fields, nonfinite constants and booleans masquerading as integer revisions so the admitted record shape remains explicit.

When should the receiver acknowledge the request?

Choose and document the durable ownership boundary required by the provider integration. Verified receipt, durable queueing, projection commit and remote completion are different stages. The offline artifact provides no HTTP acknowledgment policy or live retry timing evidence.

Can I replay production callbacks to debug a failure?

Use a diagnostic path that explicitly disables business effects and respects the approved data policy. Reproducing projection behavior should not resend bookings, messages or recording commands. Preserve original identities and report remote outcomes as not attempted rather than inventing success.

What should be logged for reconciliation?

Retain permitted source/event/entity identities, original disposition, adapter/schema version and relevant authoritative revision. Keep credentials and unnecessary body content out of broad telemetry. A digest helps equality checks but does not automatically make sensitive evidence anonymous.

What does the permutation test establish?

It checks every ordering of the three synthetic full-state events and repeated copies against the declared revision and terminal rules. It does not establish real-provider delivery behavior or complete coverage of delta events. Extend fixtures when the admitted schema or lifecycle changes.

How should these callbacks trigger a booking?

First project authenticated evidence, then separately authorize an exact business intent against current resource state. Reuse its immutable operation identity across retries and redials. A callback ID, signature or call-active state alone must not create another booking or restore canceled work.

Validate the boundary you actually control

Start by documenting the provider's signed-input representation and the adapter's normalized schema. Then state whether events are complete snapshots, ordered deltas or unordered evidence requiring reconciliation. These decisions determine which revision and terminal checks are legitimate.

Run the complete fixture before adding a web framework. Keep the exact-duplicate, changed-identity, stale-after-terminal, equal-revision conflict and missing-revision cases visible in the report. A future change that weakens one of those dispositions should require a contract explanation and a corresponding negative test.

Next test the real adapter with provider-supplied verification fixtures and your deployed proxy configuration. Inspect acknowledgment timing, durable ownership and restart behavior independently. The local pass is useful evidence for the projector; it does not establish provider acceptance, regional durability or a completed external action.

For 60db Agent integrations, use this separation to make incident statements precise: authenticated delivery received, entity projection accepted, action permission checked, or remote outcome unresolved. Each statement should name the evidence supporting it. Move to the business-intent booking contract when the next requirement is preventing repeated authorized effects across calls.