Three A.M. Truths: What Your Alert History Says About Your System Design
Photo by Photo by Greg Rosenke on Unsplash on Unsplash
It is 3:14 a.m. and the pager goes off. A service is timing out. Someone rolls out of bed, opens a laptop, checks a dashboard, retries a job, and goes back to sleep. In the morning, the team files it under on-call pain. Add another engineer to the rotation. Raise the threshold. Mute the alert. Move on.
That is the wrong file.
A pager is not just a staffing artifact. It is a feedback loop from your architecture. The alert history is a compressed record of where boundaries are fuzzy, where ownership is missing, and where integrations are held together with retries and hope. If the same service wakes someone up every week, the problem is probably not the rotation. It is the system.
The alert log is a diagram in disguise
Architecture diagrams show what you intended. Alert logs show what actually happens. A page for checkout latency might really be a database connection pool problem. A page for a failed background job might be a shared queue with no clear owner. A page for a third-party API timeout might be a retry storm that your own code created.
Read the log as a graph. Group pages by service, time, and dependency. Look for clusters. Are three services paging at the same minute because one shared dependency is failing? Is one team getting every alert because it is the default route for anything unowned? Does the same cron job page every night but never during business hours? These patterns are architecture review inputs.
Four patterns worth naming
The first is boundary blur. An alert fires for a system that no single team owns. The runbook says ask the platform team. The platform team says ask the data team. The data team says the queue is shared. Everyone is technically right, which is the problem.
The second is integration fragility. A third-party API slows down and your system pages. Maybe that is unavoidable. But if the integration has no timeout, no circuit breaker, and no fallback, the page is not telling you about the vendor. It is telling you about your own design.
The third is ownership gaps. Shared queues, cron jobs, and internal tools are classic orphans. They page the nearest team because nobody else will take them. That is not an on-call problem. It is an ownership problem with a pager attached.
The fourth is feedback latency. Alerts fire after users are already affected because the system lacks service-level objectives or meaningful health checks. The pager is doing the work that better instrumentation should have done earlier.
Turn pages into review inputs
Once a week, look at the top ten alerts by volume and the top five by pain. For each one, ask three questions: What boundary failed? What contract was missing? What would make this alert unnecessary? Not who missed it. Not who was slow. Those questions are for a different meeting.
Then make small fixes. Add owner metadata to alerts. Set timeouts and circuit breakers on fragile integrations. Give shared queues a real owner and a dashboard. Replace raw error alerts with SLO-based alerts where it makes sense. Delete noisy alerts that nobody acts on. A page that is always ignored is worse than no page at all.
This is not an argument against on-call. Someone has to carry the pager. But the schedule should not be a permanent apology for architecture that nobody has time to fix. If a rotation is consistently miserable, the team has data. Use it before the next handoff.
Staffing is not the fix
Adding people to a bad architecture does not fix the architecture. It spreads the pain across more calendars. If every rotation burns out, the schedule is not the root cause. The schedule is the symptom.
The pager is a mirror. It reflects the system back at the team that built it. If you do not like what you see, do not just change the rotation. Change the system.
