DUTY LOG · 2026-08-20 · PHASE B1–B4 · LOCAL LAB

Ten failure classes hit the same pipeline. Snapshot diff never leaves zero.

A dual-path pipeline: MySQL through Kafka and Flink into Iceberg. Every line below is copied from the real run recorded on 2026-08-20 — it is a log, not a diagram. Open any entry to replay that recovery.

10FAILURE CLASSES0SNAPSHOT DIFFS1,791EVENTS / S

Throughput is a measured end-to-end figure: 55.814s from the first MySQL write to zero Iceberg backlog, including the injected broker outage.

DUTY LOG — 20 AUG 2026 · 10 ENTRIES · ALL PASSED
PINNED · FEATURED RUN12:53:51Z · PHASE B3 · BROKER-RESTART · RUN 0B79472B · GIT F27AC1D1

Kill the Kafka broker container outright. It's back to running in 5.8s; Flink resumes from checkpoint chk-1 and finishes recovery in 47.688s — reconciliation is 120 rows against 120 rows, snapshot diff 0.

47.688 sDIFF 0
  1. T+0.0sINJECT docker compose kill kafka · exit 137 · partitions 0/1/2 lag 3·0·2
  2. T+5.8sRECOVER container running again · started 12:54:24Z
  3. RESUME checkpoint chk-1 · da52d7f2ccb2 · connector p1-orders-connector RUNNING
  4. T+47.7sVERIFY mysql 120 = iceberg 120 · sha256 ab3a445b8ee0… = ab3a445b8ee0… · diff 0 · 10/10 checks
Entry 5 of 10
2026-05-27 15:55:18Z · PHASE 2.2 · SMALL-FILE-REWRITE

48 small data files get compacted by rewrite_data_files down to 2; manifests shrink from 24 to 1, query-planning latency drops from 54.92ms to 44.57ms, table state unchanged.

—DIFF 0
  1. T+0.0sRUN before-state 48 data files · 24 manifests · table cdc_lab.orders_current
  2. REWRITE rewrite_data_files + rewrite_manifests maintenance
  3. VERIFY after-state 2 data files (-46 delta) · 1 manifests · planning latency -10.35ms delta
Entry 1 of 10
2026-07-11 03:47:48Z · PHASE 2.1 · EO-RECONCILIATION

Flink is put through 5 internal failure scenarios in a row (task-crash, checkpoint-restore, jobmanager-restart, savepoint-restore, sink-commit-fault), reconciled after each one — every snapshot diff comes back 0.

—DIFF 0
  1. RUN task-crash: one-shot Flink operator exception on event_id=1303 · snapshot_diff_count 0 · passed true
  2. RUN checkpoint-restore: cancel running job and restore from latest completed checkpoint · snapshot_diff_count 0 · passed true
  3. RUN jobmanager-restart: restart the Flink JobManager container after a completed checkpoint · snapshot_diff_count 0 · passed true
  4. RUN savepoint-restore: create a Flink savepoint, cancel the job, and restore from that savepoint · snapshot_diff_count 0 · passed true
  5. RUN sink-commit-fault: test-only checkpoint-complete fault after event_id=9303 · snapshot_diff_count 0 · passed true
Entry 2 of 10
10:21:56Z · SEED 17 · BROKER-PARITY

Two fully independent delivery paths — MySQL binlog and MySQL GTID — each push 1,000 rows into the same Iceberg table; the two snapshot digests match byte-for-byte, row-level diff 0.

—DIFF 0
  1. T+0.0sRUN path A (1000 rows, MySQL binlog -> embedded Debezium -> Flink -> Iceberg) and path B (1000 rows, MySQL GTID -> standalone Debezium at-least-once -> Kafka offsets -> Flink checkpoint -> Iceberg keyed upsert) in parallel
  2. VERIFY iceberg_snapshot(path_A) == iceberg_snapshot(path_B): true · row_level_diff_count 0
Entry 3 of 10
11:59:57Z · SEED 211 · SCHEMA-CONTRACT

Submit an incompatible Avro schema (a field removed) to the Schema Registry; registration is rejected with HTTP 409. The main pipeline keeps running — 3 more events land cleanly after the rejection, source-to-Iceberg diff 0.

—DIFF 0
  1. T+0.0sINJECT incompatible Avro schema (removed field) · candidate_schema_sha256 2e4885c50a…
  2. DETECT registration rejected · http_status 409
  3. RECOVER old-schema pipeline continues · connector_running_after_rejection true
  4. VERIFY iceberg_row_count 3 · post_rejection_source_iceberg_diff_zero true
Entry 4 of 10
13:07:00Z · SEED 302 · DUPLICATE-REDELIVERY

Rewind the consumer offsets to earliest and redeliver 36 events as one batch; the pipeline recognizes all 36 redeliveries as duplicates — final state gains not one extra row.

50.696 sDIFF 0
  1. T+0.0sINJECT kafka-consumer-groups.sh --reset-offsets --to-earliest --execute
  2. DETECT 36 duplicate deliveries · distinct_event_ids 36
  3. RECOVER restart Path B from rewound committed offsets
  4. VERIFY snapshot_diff_count 0 · redelivery_job 9fa6b5152416
Entry 6 of 10
13:11:03Z · SEED 303 · MIS-KEYING

A mis-keyed record is quarantined into the probe topic — the main path never receives it; the ordering ceiling is proven, not papered over.

24.456 sDIFF 0
  1. T+0.0sINJECT mis-keyed batch (wrong partition key) · order_id 9200303
  2. DETECT audit gate: rejected before main-pipeline admission
  3. RECOVER audit gate rejects mis-keyed batch; retain PK-keyed main path
  4. VERIFY non_monotonic_transition_count 1 · snapshot_diff_count 0
Entry 7 of 10
13:17:52Z · SEED 304 · POISON-DLQ

A poison message drops into the DLQ; the main pipeline never stops. Repaired and replayed, the final state matches the source row for row.

52.414 sDIFF 0
  1. T+0.0sINJECT malformed UTF-8 JSON bytes on an Avro order topic · topic broker.cdc_lab.orders
  2. DETECT connector PAUSED · dlq offset 0 partition 0
  3. RECOVER decode intended row, re-encode registered Avro, replay with PK key
  4. VERIFY iceberg_snapshot_row_count 13 · snapshot_diff_count 0
Entry 8 of 10
13:52:36Z · SEED 305 · OFFSET-REPLAY

Rebuild the whole table by timestamp: timestamp diff 0, offset diff 0 — the replay itself is a reconciliation.

38.008 sDIFF 0
  1. T+0.0sINJECT consumer group rewound to offset 0 · group p1-b3-replay-offset-zero-305
  2. RECOVER fresh-table rebuild from offset 0 and chosen timestamp · job 3cb3b30705ac
  3. VERIFY offset_zero_diff_count 0 · timestamp_diff_count 0 · snapshot_diff_count 0
Entry 9 of 10
16:23:53Z · SEED 401 · BROKER-SLO

Under load with all 5 fault classes stacked, throughput holds at 1,791 events/s, end-to-end freshness p50 15.2s / p95 20.6s, final snapshot diff 0.

—DIFF 0
  1. RUN broker-restart: recovered in 47.688s under 100,000 events
  2. RUN duplicate-redelivery: recovered in 50.696s under 100,000 events
  3. RUN mis-keying: recovered in 24.456s under 100,000 events
  4. RUN poison-dlq: recovered in 52.414s under 100,002 events
  5. RUN offset-replay: recovered in 38.008s under 100,002 events
  6. VERIFY sustained 1791.665 events/s · freshness p50 15.2s p95 20.6s · snapshot_diff_count 0
Entry 10 of 10

RECONCILIATION · SNAPSHOT DIGEST EQUALITY

Exactly-once ends
at reconciliation.

∀ drill ∈ 10 faults: iceberg_snapshot(path_A) ≡ iceberg_snapshot(path_B)

drilldiffthresholdverdict
duplicate-redelivery0snapshot_diff_count<=0PASS
ordering-miskey0snapshot_diff_count<=0PASS
poison-dlq0snapshot_diff_count<=0PASS
broker-restart0snapshot_diff_count<=0PASS
offset-replay0snapshot_diff_count<=0PASS
schema-contract0snapshot_diff_count<=0PASS
small-file-rewrite0snapshot_diff_count<=0PASS
eo-reconciliation0snapshot_diff_count<=0PASS
broker-parity0snapshot_diff_count<=0PASS
broker-slo0snapshot_diff_count<=0PASS

No JavaScript required: every drill below is a static table server-rendered from the same summary, with a link to its full recorded run.

DUPdiff = 0
INJECTDETECTRECOVERVERIFYDIFFhash (sha256)
——50.7s—0d62685a0bdf3168d…

Download raw JSON (28,683 bytes)

ORDdiff = 0
INJECTDETECTRECOVERVERIFYDIFFhash (sha256)
——24.5s—038ca7d6d0ded8fbe…

Download raw JSON (18,348 bytes)

POISONdiff = 0
INJECTDETECTRECOVERVERIFYDIFFhash (sha256)
——52.4s—08920a86ab10a09f6…

Download raw JSON (25,519 bytes)

RESTARTdiff = 0
INJECTDETECTRECOVERVERIFYDIFFhash (sha256)
——47.7s—078896060dfa5b32a…

Download raw JSON (79,102 bytes)

OFFSETdiff = 0
INJECTDETECTRECOVERVERIFYDIFFhash (sha256)
——38.0s—0444d5a19ed576f15…

Download raw JSON (62,845 bytes)

SCHEMAdiff = 0
INJECTDETECTRECOVERVERIFYDIFFhash (sha256)
————03c65511960885761…

Download raw JSON (22,235 bytes)

SMALLFILEdiff = 0
INJECTDETECTRECOVERVERIFYDIFFhash (sha256)
————03ed9659a57bdd116…

Download raw JSON (4,052 bytes)

RECONdiff = 0
INJECTDETECTRECOVERVERIFYDIFFhash (sha256)
————0e6c613bc1dc0be55…

Download raw JSON (22,821 bytes)

PARITYdiff = 0
INJECTDETECTRECOVERVERIFYDIFFhash (sha256)
————048be70bbe064112a…

Download raw JSON (4,699 bytes)

SLOdiff = 0
INJECTDETECTRECOVERVERIFYDIFFhash (sha256)
————05176dd0ac38d9778…

Download raw JSON (105,652 bytes)

PATH A · PATH B · 1000 EVENTS

Two delivery paths
must land on the same state.

PATH A

MySQL binlog -> embedded Debezium -> Flink -> Iceberg

1,000 rows5780efc5f0fb5005…

PATH B

MySQL GTID -> standalone Debezium at-least-once -> Kafka offsets -> Flink checkpoint -> Iceberg keyed upsert

1,000 rows5780efc5f0fb5005…

PASS

Snapshot digests match: true. Row-level diff count: 0.

CHECKPOINT DURATION · ICEBERG COMMIT LAG

Recovery has
ten different failure shapes.

Checkpoint pressure recorded-run trace10 ms100 ms1k ms10k msCHECKPOINT DURATION · LOG SCALEckpt 3ckpt 13ckpt 21ckpt 27ckpt 27ckpt 28ckpt 28ckpt 39ckpt 5255 mscheckpoint 2819,022 ms0160320ICEBERG COMMIT LAG · EVENTS3200BACKPRESSURE INDICATOR0.649baselinebackpressurerecovery
9 recorded samples show the checkpoint spike, commit-lag backlog and recovery in one shared timeline.
Maximum checkpoint duration55 ms → 19,022 msbaseline → induced backpressure
Maximum Iceberg commit lag320 events → 0recovered in the recorded run
Checkpoint failures1captured in the run record
SamplePhaseCheckpointDurationCommit lagBackpressure
0baseline355 ms01.000
1baseline1344 ms00.000
2baseline2140 ms00.000
3backpressure2753 ms3200.000
4backpressure2753 ms3200.140
5backpressure2819,022 ms1600.397
6recovery2819,022 ms00.498
7recovery3954 ms00.649
8recovery5243 ms00.601

HOW THIS WAS VERIFIED

Every drill opens
the same raw file.

What was verified
The recorded fault drills reconcile recovered snapshots with zero differences in the captured runs.
Evidence class
Recorded drill JSON, with historical May artifacts and the July U6 local-Mac reproduction kept separate.
Boundary
The replay covers the recorded fault classes and environments. It does not establish every possible failure, a live cluster or general hardware compatibility.
Files, hashes and methods
index.summary.jsonLucisZhang/portfolio-site · acf05ae78859
sha256:c008972eaf6da3e53c1b5419177b608e46b6dd4f31dd1629ca1a0af2308008ee
broker_slo.jsonLucisZhang/exactly-once-drills · 50d88bb4cc9b
sha256:5176dd0ac38d977813693de85fa2a6fc3fcfb9307048cc149b0408237488e200
Result manifestLucisZhang/portfolio-site · acf05ae78859
sha256:7867fb1ec1f91787ada2bb9141426b29273324f8d84b3eac6968572085120d32
Generated
2026-08-27

Every number on this page is read from the JSON files linked above at build time; the chessboard's summary is generated from the ten full drill records, never typed in.

BOUNDARY

This page proves that these ten fault classes recover with zero snapshot diff. It does not prove that no other fault class exists, or that every possible failure in a MySQL → Kafka → Flink → Iceberg pipeline has been drilled.

Architecture

I built all of it: the deterministic event generator, the failure harness, the reconciliation checks, the incident runbook, and the recorded-run dashboard.

  1. Source

    Deterministic inserts, updates, and deletes in MySQL.

  2. CDC

    Flink captures and checkpoints the changelog.

  3. Table

    Iceberg stores the current state and change history.

  4. Failure

    The harness triggers one recorded failure type at a time.

  5. Reconcile

    Check source, snapshot, and event-ID sets after recovery.

Results & negatives

Ten ways to break the same pipeline: MySQL CDC on one path, Debezium → Avro contracts → Kafka on the other, both landing in Flink → Iceberg. After every induced failure, source state, table snapshots, and event IDs are reconciled — all ten recoveries came back with zero diffs. Sustained throughput measured at 1,791 events/s in the B4 SLO run.

  1. The task-crash drill fires from a one-shot marker file. A marker left behind by an earlier local run can swallow the next run's induced failure, and the drill then reports a clean recovery it never performed. The note I left in the runbook is to keep the marker path unique per run.

  2. Restoring from a retained checkpoint looks like reading a path and handing it to flink run -s. It is not. The checkpoint the REST API reports can be superseded by a later retained checkpoint during shutdown, so the path has to be resolved after cancellation, not before it.

  3. After I restarted the JobManager container the session job was simply gone, and waiting for a registered worker got me nothing. The TaskManager has to be brought back first; only then does the restore from chk-7 take. Reconciliation still came out at zero snapshot differences, but that ordering belongs to this single-node Compose setup, not to Flink.

Limitations

  1. The U6 result is environment-specific: Apple Silicon macOS with 16 GiB RAM and a Docker VM reporting 10 CPUs and about 7.65 GiB memory.

  2. One captured run does not establish universal hardware compatibility or one-command reproducibility. The historical dashboard proves only the run it captured.