XGZXiangguo Zhang
Back to discipline

MySQL CDC → Flink → Iceberg · failure injection

Streaming Reliability Lab

I break a MySQL-to-Flink-to-Iceberg pipeline on purpose, then check that source state, table snapshots, and event IDs still agree after recovery.

MySQL CDCApache FlinkApache IcebergMinIODocker ComposePythonJava
Problem
A happy-path demo cannot show what happens when a task, coordinator, checkpoint, savepoint, or sink commit fails.
Audience
Data and platform engineers who need to inspect recovery and reconciliation before trusting a streaming pipeline.
What I built
I built all of it: the deterministic event generator, the failure harness, the reconciliation checks, the incident runbook, and the recorded-run dashboard.
Result
The July U6 run completed all five induced failure classes with zero snapshot differences and consistent event-ID audits after recovery.
Links
GitHub repositoryOpens external site

How it works

  1. 01
    Source

    Deterministic inserts, updates, and deletes in MySQL.

  2. 02
    CDC

    Flink captures and checkpoints the changelog.

  3. 03
    Table

    Iceberg stores the current state and change history.

  4. 04
    Failure

    The harness triggers one recorded failure type at a time.

  5. 05
    Reconcile

    Check source, snapshot, and event-ID sets after recovery.

Try it

Original benchmark runMay capture

Result JSON, charts, dashboard capture, and incident runbook at commit 47b4268.

Later workstation reproduction20260711T034018Z-local-mac

Commit 7eab9c3 records the full command, timing, and reconciliation result.

Loading recorded evidence...
Phase / artifactRun IDGit SHACommand
1.1Toolchain and core-stack smoke20260527T093100Z-phase-1-1-smokeb5668d9make doctor / up-core / gen1.2CDC correctness smoke20260527T133945Z-58e25cd1b2434d1make test-cdc2.1Five-failure reconciliation20260527T151754Z-ef73a5a5b2434d1make eo-verify2.2Iceberg small-file rewrite20260527T155713Z-a88e18f6ad7ad52make small-file-rewrite2.3Checkpoint pressure capture20260527T233135Z-0b65b846ce084a5make ckpt-metrics

Recorded historical evidence

Checkpoint pressure under induced backpressure

Open source JSON

Maximum checkpoint duration55 ms → 19,022 msbaseline → induced backpressure
Maximum commit lag320 events → 0recovered in the recorded run
Checkpoint failure1captured in the run record
Historical evidence dashboard for the captured p1 run
The dashboard summarizes reconciliation status, failure-class outcomes, source counts, and Iceberg snapshot differences in one operational view.
Historical Iceberg small-file rewrite evidence
The maintenance view shows how Iceberg data files were compacted while preserving table state.

How this was verified

  • The May public run files come from commit 47b4268 and include result JSON, charts, a dashboard capture, and incident notes.
  • The later U6 run is 20260711T034018Z-local-mac at evidence commit 7eab9c3; its exported summary and reconciliation JSON are the authority for the local-Mac result.

What this does not prove

  • The U6 result is environment-specific: Apple Silicon macOS with 16 GiB RAM and a Docker VM reporting 10 CPUs and about 7.65 GiB memory.
  • One captured run does not establish universal hardware compatibility or one-command reproducibility. The historical dashboard proves only the run it captured.