Incident Response

During an Incident

1. Triage (First 5 minutes)

• What's broken? User-facing impact. Who's affected and how many.
• When did it start? Check monitoring, deployment timeline, recent changes.
• Is it getting worse? Stable, degrading, or cascading.
• Severity?

Severity	Impact	Response
SEV1	Service down, data loss risk	All hands, immediate
SEV2	Major feature broken, many users affected	Drop current work
SEV3	Minor feature broken, workaround exists	Next working hours
SEV4	Cosmetic, edge case	Normal priority

2. Mitigate (Stop the bleeding)

Priority: restore service, not find root cause. Options:

• Rollback the last deployment
• Feature flag disable the broken feature
• Scale up if it's a capacity issue
• Redirect traffic away from broken component
• Restart if it's a stuck process or resource leak

Document what you do and when. Timestamps matter.

3. Communicate

• Notify affected users/stakeholders with honest status
• Set expectations for next update ("we'll update in 30 minutes")
• Don't speculate on cause until you know

4. Investigate (After service is restored)

• Gather logs, metrics, traces from the incident window
• Correlate with deployment timeline and config changes
• Identify the triggering event and the root cause (they're often different)

Post-Mortem Template

Write within 48 hours while memory is fresh. Blameless - focus on systems, not people.

# Incident: [Title]
Date: [YYYY-MM-DD]
Duration: [start time → end time]
Severity: [SEV1-4]
Author: [who's writing this]

## Summary
[2-3 sentences: what happened, who was affected, how it was resolved]

## Timeline
- HH:MM - [event]
- HH:MM - [event]
- HH:MM - [detection / alert fired]
- HH:MM - [response began]
- HH:MM - [mitigation applied]
- HH:MM - [service restored]

## Root Cause
[What actually broke and why. Be specific.]

## Contributing Factors
- [Factor 1: why it was possible for this to happen]
- [Factor 2: why it wasn't caught earlier]
- [Factor 3: why recovery took as long as it did]

## What Went Well
- [Things that helped during response]

## What Went Poorly
- [Things that made response harder]

## Action Items
| Action | Owner | Priority | Deadline |
|--------|-------|----------|----------|
| [Prevent recurrence] | ? | P1 | ? |
| [Improve detection] | ? | P2 | ? |
| [Improve response] | ? | P3 | ? |

## Lessons Learned
[What should the team internalize from this?]

Prevention Patterns

Pattern	Prevents
Canary deploys	Bad code reaching all users
Health checks	Serving traffic from broken instances
Circuit breakers	Cascading failures
Rate limiting	Resource exhaustion
Automated rollback	Prolonged outages from bad deploys
Chaos testing	Surprises in production
Runbooks	Slow response from unfamiliarity