Turn incident chaos into a blameless postmortem. Feed raw timeline notes, get structured root cause analysis and systemic fixes—no finger-pointing.
You are a blameless postmortem facilitator. Your job is to transform raw incident notes into a structured postmortem that surfaces systemic causes and actionable improvements.
Incident details:
- Type: {{incident_type}}
- Duration: {{incident_duration}}
- Severity: {{severity_level}}
- Raw timeline notes: {{timeline_notes}}
- Team involved: {{team_names}}
- Systems affected: {{affected_systems}}
Structure the postmortem as follows:
1. **Incident Summary**
- What: One-sentence description of what broke
- When: Start and end time (UTC)
- Impact: Users affected, SLA cost, revenue impact if known
- Detection: How was it found? How long until detection?
2. **Timeline**
- List events in chronological order
- Include detection, escalation, mitigation, and resolution
- Use timestamps
- Avoid blame language; focus on observable facts
3. **Root Cause Analysis**
- What system design, process gap, or missing safeguard enabled this?
- Why did the normal controls fail?
- This is NOT 'operator did X wrong'—it's 'the system allowed X to happen undetected'
4. **Contributing Factors**
- List conditions that made failure more likely
- Examples: on-call fatigue, undocumented runbook, missing monitoring, recent deploy, config drift
- Each factor should be systemic, not personal
5. **Action Items**
- Rank by effort (1–5 days) and blast radius (how many incidents this prevents)
- Include owner and target completion date
- Prefer prevention over response (e.g., 'add alert' over 'train team')
6. **Lessons Learned**
- What did we learn about our system, process, or team?
- What should we do differently next time?
Tone: Curious, not accusatory. Assume everyone involved was doing their best with the information they had. Focus on system gaps, not human gaps.1. Collect raw incident notes: Slack thread, PagerDuty timeline, on-call notes, whatever you have. Messy is fine. 2. Fill in the {{placeholders}}: incident type, duration, severity, team names, affected systems. 3. Paste into your AI tool (Claude, GPT-4, or your internal LLM). 4. Copy the output into a Google Doc or Notion. 5. Share with the team 24–48 hours after incident resolution. Use it in a 30-min sync, not an email. 6. Track action items in your project management tool. Link each one back to the postmortem. 7. Revisit in 30 days: Did we ship the fixes? Did they work? Update the postmortem with outcomes.
Incident type: Database connection pool exhaustion Incident duration: 47 minutes (2024-01-15 14:22 UTC to 15:09 UTC) Severity level: P1 (all API requests failing) Timeline notes: - 14:22: Customer reports 'service down' in Slack - 14:25: On-call (Sarah) paged, checks dashboard, sees 100% error rate - 14:28: Sarah escalates to backend team lead (Marcus) - 14:35: Marcus identifies connection pool at max (500 connections, all in use) - 14:40: Team checks recent deploys—nothing in last 2 hours - 14:45: Marcus restarts the connection pool service - 14:50: Error rate drops to <1%, customers report recovery - 15:09: All systems green, incident declared resolved Team involved: Sarah (on-call), Marcus (backend lead), Priya (infra) Systems affected: API gateway, user service, payment service
# Postmortem: Database Connection Pool Exhaustion ## January 15, 2024 **Incident Summary** - **What:** All API requests failed due to database connection pool exhaustion. - **When:** 2024-01-15 14:22–15:09 UTC (47 minutes) - **Impact:** 100% of API traffic affected; estimated 8,000 failed requests; ~$2,400 in failed transactions (based on typical transaction value) - **Detection:** Customer report via Slack; no internal alert fired **Timeline** | Time (UTC) | Event | Owner | |---|---|---| | 14:22 | Customer reports service unavailable | External | | 14:25 | On-call (Sarah) paged; dashboard shows 100% error rate | Sarah | | 14:28 | Escalated to backend lead (Marcus) | Sarah | | 14:35 | Root cause identified: connection pool at 500/500 (max capacity) | Marcus | | 14:40 | Checked recent deploys; none in last 2 hours | Marcus | | 14:45 | Connection pool service restarted | Marcus | | 14:50 | Error rate <1%; customer recovery confirmed | Marcus | | 15:09 | All systems nominal; incident closed | Team | **Root Cause Analysis** The connection pool was configured with a hard limit of 500 connections. A slow database query (running for >10 minutes) was holding connections open, preventing new requests from acquiring a connection. The system had no mechanism to: 1. Detect when the pool was nearing capacity 2. Kill long-running queries 3. Alert on-call before customer impact The slow query was introduced in a deploy 6 days ago (not recent, so not caught in the 2-hour deploy window). No query performance regression testing exists in CI/CD. **Contributing Factors** - No alert on connection pool utilization (alert threshold was never set after infra migration 3 months ago) - No query timeout or kill policy in the database configuration - No performance regression testing in CI/CD pipeline - On-call had to wait for customer report to know about the incident (no internal monitoring) **Action Items** | Action | Owner | Effort | Blast Radius | Target Date | |---|---|---|---|---| | Add alert: connection pool >80% utilization | Priya | 1 day | Prevents 90% of pool exhaustion incidents | Jan 22 | | Implement query timeout (60s) in DB config | Marcus | 2 days | Prevents runaway queries from holding connections | Jan 24 | | Add query performance regression test to CI/CD | Priya | 3 days | Catches slow queries before production | Jan 29 | | Document connection pool limits and tuning in runbook | Sarah | 1 day | Reduces MTTR for future incidents | Jan 18 | **Lessons Learned** - We rely on customers to tell us about incidents. Internal monitoring is a gap. - Configuration changes (like pool limits) need to be re-validated after migrations. - Slow queries are a class of incident we're not testing for. **Follow-up (30 days later)** - Alert implemented Jan 22; caught 2 minor pool spikes before they became incidents - Query timeout deployed Jan 24; no regressions - CI/CD regression test added Jan 29; already caught 1 slow query in a feature branch - Runbook updated Jan 18; new on-call found it helpful