The four phases
- DetectLearn something is wrong. The goal is to hear it from your monitoring, not from a customer.
- TriageEstablish scope and severity, and get the right people involved. Resist the urge to start fixing before you know what is broken.
- CommunicateTell affected users what is happening, in parallel with the fix rather than after it.
- LearnUnderstand why it happened and what would prevent a recurrence. This is the phase most teams skip.
Detection sets the floor
Total incident duration is detection plus response plus recovery. Teams invest heavily in the last two and then leave a five-minute check interval in place, which guarantees a multi-minute floor no amount of heroics can beat.
Detection is also usually the cheapest of the three to improve, which makes it the obvious place to start.
Assign a single coordinator
The most common failure mode in a small-team incident is five people independently investigating the same thing while nobody talks to customers. One person should coordinate: track what is known, decide who does what, and own communication.
For a small incident this can be the same person doing the fixing. As soon as more than two people are involved, it should not be.
Deciding whether something is a Sev1 during the incident wastes time and produces inconsistent responses. Write the definitions down in advance, tied to user impact rather than to which system is broken.
Communicate earlier than feels comfortable
The instinct is to wait until you understand the problem so you can give a complete answer. That instinct is wrong. Users do not need a root cause analysis; they need to know that you know.
A status page update saying "we are investigating reports of errors on checkout" posted within five minutes prevents far more support tickets than a detailed explanation posted an hour later.
- Post within minutes of confirming impact, even with almost no detail
- Say what is affected and, importantly, what is not
- Give a time for the next update rather than a time for resolution
- Never promise a fix time you are not confident about
Blameless postmortems
The purpose is to find the systemic conditions that let a mistake become an outage, not to identify who made the mistake. Once a postmortem becomes about attribution, people withhold detail, and you lose the information that would have prevented the next one.
The productive question is never "who deployed this" — it is "why was it possible to deploy this without anything catching it".
Action items with owners
A postmortem that ends in a shared understanding and no scheduled work has accomplished very little. Every action item needs a specific owner and a date, and should be tracked in the same system as normal work rather than in a document nobody reopens.
Two or three items that actually ship beat fifteen that do not.