Incident Lab · SEV2 · Kafka
Closed accounts showing as suspended
Support: account 2 was closed on Monday, but the admin panel says it's suspended and billing tried to charge it.
WARN consumer-group account-status: partition 3 lag recovered after broker failover, 2 events redelivered late
current account status reads the account-events topic and keeps the last event received per account, ordering by kafka offset . That works only if events arrive in the order they happened. After a broker failover, producers retried, and some older events landed after newer ones, so they got higher offsets. The model now reports the late event as the current status. Investigate: Look for rows where a higher offset has an earlier event time . Fix current account status so each account shows the status from its latest event time . If two events share the same event time , the one with the higher kafka offset wins. Columns: account id , status , status since (that event's event time ), ordered by account id .