Incident Lab: on-call training for data engineers

• Revenue doubled overnight (SEV2, Airflow): Finance: yesterday's revenue on the dashboard is twice what the payment provider settled.
• Conversion Error: 'N/A' is not a number (SEV2, Snowflake): The stg_payments model has failed three runs in a row. Downstream dashboards are stale.
• Revenue by segment is 40% too high (SEV2, dbt): Sales: the segment report shows more revenue than we have orders.
• The churn report is suddenly empty (SEV3, SQL): Customer success: 'customers with no order in 90 days' returned 0 rows this morning. It had 2 yesterday.
• Binder Error: column "amount" not found (SEV1, Fivetran): Every model downstream of fct_payments failed at 06:00. Executive dashboard is blank.
• Yesterday's late orders are in today's report (SEV2, Snowflake): Ops: Saturday's sales report is short about $95, and Sunday has orders our New York store says came in Saturday night.
• Mobile purchases missing from fct_events (SEV2, dbt): Product: a purchase and a signup from yesterday are in the app logs but not in fct_events.
• Revenue jumped 2,000% after the Japan launch (SEV1, Snowflake): CFO: the board deck shows $15k of revenue for October 1. Settlements say about $430. Which one is right?
• Conversion rate is 0% every day (SEV3, Postgres): Growth: the funnel dashboard says nobody has converted since we moved it to Postgres. Orders look normal.
• Midnight renewals missing from every daily load (SEV2, Airflow): Billing: every subscription renewal is missing from the warehouse. Card payments during the day look fine.
• Products with no March sales vanished from the report (SEV2, dbt): Merchandising: the March product report only lists 3 products. We have 6 in the catalog, and the ones that didn't sell are exactly the ones we need to see.
• Last year's segment revenue changed overnight (SEV2, dbt): Finance: 2024 business-segment revenue dropped by 400 since last week's board deck. Nobody reloaded 2024 orders.
• Average order value fell by half with no change in sales (SEV3, SQL): E-commerce lead: the dashboard says AOV dropped from ~$90 to ~$45 since the line-item migration. Revenue and order counts look normal.
• Delivered shipments showing twice in the tracking table (SEV2, Snowflake): Support: the customer portal shows shipment 1 as both 'out_for_delivery' and 'delivered'. The ops dashboard counts it twice.
• Revenue per customer short after the "duplicate fix" (SEV2, SQL): Billing: customer 101 paid for three orders (218.00) but the account report shows 169.00.
• Units shipped doubled for May 1-3 after a backfill (SEV1, Airflow): Ops: the shipments dashboard now shows almost twice the units for May 1-3. Carrier invoices didn't change.
• Closed accounts showing as suspended (SEV2, Kafka): Support: account 2 was closed on Monday, but the admin panel says it's suspended and billing tried to charge it.
• Freshness check is green, CRM data is three days old (SEV2, Fivetran): Sales ops: pipeline numbers haven't moved since Sunday, but the data health dashboard is all green.
• Deleted customers are back in the CRM sync (SEV2, Postgres): Privacy: two customers who had their accounts deleted got this week's marketing email.
• GMV down 62% today (SEV2, Databricks): Finance: today's GMV tile dropped 62% versus yesterday. Is something wrong with sales or with the data?
• The Oct 7 revenue figure is above what the payment provider reports (SEV2, Airflow): Finance: the Oct 7 revenue figure on the dashboard is well above the payment provider. The days before match.
• The daily revenue report doesn't match the New York team's numbers (SEV2, Airflow): Finance (New York): our daily revenue figures never match the dashboard. Totals over the three days agree, single days don't.
• The segment breakdown of revenue adds up to more than the total (SEV2, Airflow): Finance: the segment breakdown of revenue sums to more than the company total since dim_customers started keeping history.
• The inactive-customer report is suddenly empty (SEV3, Airflow): Finance: "customers with no orders this week" returned 0 rows this morning. It listed 4 yesterday.
• stg_orders fails: "Could not convert string 'N/A' to DOUBLE" (SEV2, Airflow): Finance: the revenue dashboard is stale. stg_orders has failed three runs in a row, everything downstream is skipped.
• Binder Error: column "amount" not found in daily_revenue (SEV1, Airflow): Finance: every revenue report failed at 06:00. Nothing was deployed on our side.
• Retried batches are doubling orders, and the pipeline is still running (SEV1, Airflow): Finance: the hourly revenue dashboard is above the payment provider and climbing. load_orders retried a batch and keeps running every 2 minutes.
• The Oct 7 gross bookings figure is above what the driver payout file reports (SEV2, Databricks): Ops: the Oct 7 gross bookings figure on the dashboard is well above the driver payout file. The days before match.
• The daily gross bookings report doesn't match the Mumbai team's numbers (SEV2, Databricks): Ops (Mumbai): our daily gross bookings figures never match the dashboard. Totals over the three days agree, single days don't.
• The city breakdown of gross bookings adds up to more than the total (SEV2, Databricks): Ops: the city breakdown of gross bookings sums to more than the company total since dim_drivers started keeping history.
• The inactive-driver report is suddenly empty (SEV3, Databricks): Ops: "drivers with no trips this week" returned 0 rows this morning. It listed 4 yesterday.
• stg_trips fails: "Could not convert string 'N/A' to DOUBLE" (SEV2, Databricks): Ops: the gross bookings dashboard is stale. stg_trips has failed three runs in a row, everything downstream is skipped.
• Binder Error: column "fare" not found in daily_bookings (SEV1, Databricks): Ops: every gross bookings report failed at 06:00. Nothing was deployed on our side.
• Retried batches are doubling trips, and the pipeline is still running (SEV1, Databricks): Ops: the hourly gross bookings dashboard is above the driver payout file and climbing. load_trips retried a batch and keeps running every 2 minutes.
• The Oct 7 collected revenue figure is above what Stripe reports (SEV2, dbt): RevOps: the Oct 7 collected revenue figure on the dashboard is well above Stripe. The days before match.
• The daily collected revenue report doesn't match the Berlin team's numbers (SEV2, dbt): RevOps (Berlin): our daily collected revenue figures never match the dashboard. Totals over the three days agree, single days don't.
• The plan breakdown of collected revenue adds up to more than the total (SEV2, dbt): RevOps: the plan breakdown of collected revenue sums to more than the company total since dim_accounts started keeping history.
• The inactive-account report is suddenly empty (SEV3, dbt): RevOps: "accounts with no payments this week" returned 0 rows this morning. It listed 4 yesterday.
• stg_payments fails: "Could not convert string 'N/A' to DOUBLE" (SEV2, dbt): RevOps: the collected revenue dashboard is stale. stg_payments has failed three runs in a row, everything downstream is skipped.
• Binder Error: column "amount" not found in daily_collections (SEV1, dbt): RevOps: every collected revenue report failed at 06:00. Nothing was deployed on our side.
• Retried batches are doubling payments, and the pipeline is still running (SEV1, dbt): RevOps: the hourly collected revenue dashboard is above Stripe and climbing. stripe_payments_sync retried a batch and keeps running every 2 minutes.