How to Build Incident AI That Engineers Actually Trust
An incident-detection AI lost engineer trust by reasoning before context. Adding deploy tracking and a context graph raised hypothesis accuracy from 35% to 70%.
A team's first attempt at an AI-driven incident-detection system ran into serious trust problems by prioritizing reasoning over system context. Without topology, deploy history, or past incident records, it produced plausible but frequently wrong explanations, often blaming whichever service was most recently deployed. Engineers agreed with its top hypothesis only about 35% of the time, and several stopped using it within weeks, preferring to review raw trace data themselves.
Rebuilding from that failure, the team found that adding deploy and config-change awareness made the single biggest difference — roughly 60% of incidents involved a change from the prior two hours, and incorporating that context pushed hypothesis agreement from 35% to 70%. Layering in service ownership, dependency graphs, and structured postmortem history let the system behave more like institutional memory rather than just summarizing current data.
The team settled on a six-layer architecture: normalization, a context graph, targeted evidence retrieval, structured reasoning output, human approval, and a learning loop. Small issues, like inconsistent service naming, were found to silently break correlation across every layer above them, while feeding raw, unstructured postmortems into the learning loop actually made retrieval worse until a strict write-back format was enforced. Automated remediation was deliberately left as the last feature to add, only after human-approved trust was established.
The broader lesson is that useful incident AI isn't about generating text — it's about quickly connecting the right context. Confident but wrong answers, produced without solid system grounding, are more damaging than having no automated answer at all.