Incident Lab · SEV1 · Airflow
Units shipped doubled for May 1-3 after a backfill
Ops: the shipments dashboard now shows almost twice the units for May 1-3. Carrier invoices didn't change.
[2026-05-06 11:42:19] INFO - Backfill dag_run load_shipments ds=2026-05-01..2026-05-03 mode=append: 3 tasks SUCCESS
Someone backfilled load shipments for May 1-3 to pick up a corrected feed. The task writes with INSERT (append) instead of overwriting the partition, so each of those days now holds two complete batches : the original one and the backfill, each with its own batch id . The backfill is the correct one (May 2 was fixed in it). Deduplicating rows won't help here: inside one batch the same warehouse can legitimately ship the same number of units twice. The unit of duplication is the whole partition ( ds ). Investigate: Fix daily units so each ds uses only its latest batch (the highest batch id ). Columns: ds , units (the total), ordered by ds .