Lead · Scheduled Tasks
Not every failure deserves to wake someone at 3 a.m. Fail to tell transient from structural, and you just train your team to ignore alerts.
Derek Finch
·
July 30, 2026
At 3 a.m., a scheduled month-end reconciliation task fails because the vendor's API timed out briefly. The system has two options: retry silently once, which usually succeeds the second time, or fire off a notification and wake up whoever is on call. Most teams designing scheduled tasks only think as far as 'notify on failure' and never work out whether to try saving it once before notifying...