Incident response
Also known as Incident management, Production incident response
Definition
Incident response is the coordinated process of detecting, assessing, containing, communicating about, and recovering from an event that threatens a service or users. It includes the operational actions during the event and the learning work that follows.
Response has several phases
An incident begins when a service behaves outside an acceptable condition or a user reports a serious problem. Responders establish what is happening, assign roles, reduce impact, and communicate with people who depend on the service.
Recovery can involve rollback, disabling a feature, shifting traffic, or applying a targeted fix. The safest immediate action is not always the permanent engineering solution. Keeping those goals separate helps responders restore service without attempting a risky redesign under pressure.
A practical example
Imagine an illustrative release causes elevated checkout errors. The incident lead coordinates the response, an engineer checks traces and recent deployments, and a communicator updates stakeholders. Rolling back reduces impact while the team preserves evidence for later analysis.
The response record should capture decisions and times, not only the final cause. A failed hypothesis is useful when it explains why the team took a particular action.
Learn without blame
After recovery, review the conditions that made the incident possible and the factors that shaped the response. Connect the timeline to code changes, review decisions, testing, and service telemetry.
Avoid treating an incident count or recovery time as an individual score. Reliable response is a property of systems, training, tools, and team coordination. Improvements should reduce risk or make the next response clearer and faster.
How this relates to Weave
Weave helps teams trace quality and delivery signals around changes that precede an incident. That supports the learning part of incident response, while detection, paging, and operational coordination remain responsibilities of the service's reliability process.
Explore Engineering intelligence