Open-Source AI Incident Commander Hits 88% on 15 Real Outages

During live incidents, the real bottleneck is often not technical diagnosis but documentation overload — juggling a timeline, stakeholder updates, and postmortem drafting while trying to make decisions. ai-incident-commander, an MIT-licensed CLI tool, targets exactly this problem by running entirely on a local LLM with no API keys, cloud dependency, or Docker required. It generates timelines, Slack-ready stakeholder updates, cited remediation suggestions, and blameless postmortem drafts, but pauses for human approval at three critical points and never executes any action itself.

The author validated it against 15 real-world incidents drawn from public postmortems (Cloudflare, GitLab, GitHub, AWS) using a cheap small model (DeepSeek V4 Flash), for a total run cost of about $0.02. The tool passed 106 of 120 checks (88.3%), scoring 100% on six criteria including blameless framing, resilience to missing logs, absence of hallucination, and cost tracking.

The most useful finding was methodological: an initial test run scored only 61% because LLM-generated timelines were being compared word-for-word against human-written postmortems. Switching the evaluation to check coherence and quality — does the timeline stay chronological, does each action item have an owner and priority — pushed those scores to 100%. The takeaway: treat LLM output like work from a junior engineer to review, not a search engine expected to match ground truth. The author is transparent about remaining gaps — RAG quality depends on runbook freshness, small models struggle with precise root-cause chains, and only 15 of 124 planned test fixtures have been run so far.