Make CDC Consumers Safe to Replay

A CDC consumer crashes after updating its destination but before acknowledging the input. On restart, the same change arrives again. If the handler increments a revenue counter, the report is now wrong even though the pipeline recovered exactly as designed.

Replay is not an exceptional corner case. It is a normal recovery operation. Debezium's PostgreSQL connector documentation explicitly warns that failure recovery can emit duplicate changes. A consumer should therefore prove its behavior under repetition rather than assuming the delivery path eliminates it. Debezium PostgreSQL connector failure handling

Decide what is being replayed

Distinguish a transport record from a business event. A topic, partition, and offset identifies a record within that topic incarnation. It does not necessarily identify the same source change republished into another topic or emitted during a resnapshot. Include a stream generation when topics can be recreated. For business side effects, prefer a stable business event identifier generated by the producer.

A row-state projection has a different job from an event ledger. Setting order 42's status to shipped twice may be harmless. Adding the order's amount to a total twice is not. Replaying an older state after a newer one can also be harmful, even if each individual assignment looks idempotent.

Document the identity and ordering scope before implementing deduplication. An LSN is useful source metadata, but do not assume one raw position uniquely identifies every row change in a transaction or orders unrelated sources. Use the connector's documented envelope and a tested tie-breaker where necessary.

Make the destination transaction the boundary

For a relational sink, put the deduplication claim and the projection mutation in the same transaction. The following pseudocode is illustrative:

begin destination transaction
  insert delivery_key into processed_records
    on conflict do nothing
  if insert succeeded:
    apply change to projection
commit destination transaction
acknowledge input record

The unique constraint is important. Two workers checking for a key and then inserting it can race. A transactional insert lets the database arbitrate ownership. If the mutation fails, the deduplication claim must roll back with it. Otherwise the retry sees a processed marker for work that never completed.

Acknowledge only after commit. When the commit result is uncertain because the connection drops, replay through the same deduplication path. Do not infer failure and invent a second key. The database may have committed successfully before the response was lost.

Protect against stale updates and resurrection

Deduplication prevents repeating the same record, but it does not prevent an older, distinct record from overwriting newer state. Preserve per-entity order or compare a trustworthy source version within each entity's stream. A conditional update should accept only a version newer than the stored one. Define how equal versions with different payloads are investigated rather than arbitrarily accepted.

Deletes require version memory too. Physically removing an order and its last-seen version allows a delayed insert to recreate it. Keep a tombstone with the deletion version for as long as older changes can return, or use an equivalent durable version ledger. Coordinate tombstone expiry with retention, recovery, and backfill policy.

Snapshots need explicit treatment. A snapshot describes observed state, not necessarily a new business action. Do not send a customer notification merely because an old order appeared during a rebuild. Keep projection reconstruction separate from external side effects. If side effects are needed, write an outbox record transactionally and give the downstream action a stable idempotency key.

Test the dangerous boundaries

Build a deterministic fixture containing an insert, two updates, a delete, and a later recreation with an explicit new version. Apply it once and save the expected state. Apply it twice. Restart between every destination commit and input acknowledgment. Shuffle only the records your actual transport permits to arrive out of order.

Then test a rebuilt topic, a resnapshot, an expired deduplication entry, and two concurrent workers claiming the same delivery. Assert final state, tombstones, processed markers, and side-effect counts. Counting consumed messages is not a correctness assertion.

Storage for deduplication is a deliberate cost. Define the supported replay horizon and measure its size. If records may return after the ledger expires, either rebuild into an empty destination or use business-level state/version guarantees that survive that expiry.

Recovery should preserve meaning

A replay-safe consumer has a written answer to four questions: what identifies a change, what orders changes, what commits atomically, and how long those guarantees survive. Once those answers exist, replay becomes a routine operational tool rather than a gamble that today's dashboard will resemble yesterday's.