How AI Pinpointed the Root Cause of a Multi-Region K8s Outage

Trendyol engineers investigated an outage that made ~63 Kubernetes nodes across seven regions unreachable — no ping, no SSH, no gateway access — with no deployment or config change to explain it. The only clues left after reboots were rotated logs totaling over 200MB, with an unknown failure window, making manual review impractical.

The team used a large language model (Claude, served through an internal ML platform) as a reasoning partner to direct the log forensics. The model first spotted a 32-hour gap in kernel logging to orient the timeline, then bucketed error counts by minute to pinpoint the exact failure second (06:21). Reading only the critical 50-line slice, it traced the cascade to an unattended systemd upgrade triggering a reexec that caused systemd-networkd to reclaim Cilium-managed network interfaces — all within under an hour.

The case illustrates how LLMs can compress time-to-signal on large unstructured logs when every command is human-reviewed and every conclusion verified. As a fix, the team marked Cilium interfaces as unmanaged in systemd-networkd and tightened their unattended-upgrade policy.