Kubernetes DNS on AWS, Part 2: The Hidden ztunnel Layer
EKS DNS timeouts traced to Istio ambient mode's ztunnel DNS capture, a hidden layer that mimics NodeLocal DNS failures — fixed with one annotation.
Following up on a common EKS DNS-timeout investigation, an engineer measured the three usual suspects — ndots amplification, NodeLocal DNS health, and the per-ENI 1024 pps cap — and found all three clean, yet pods kept logging identical i/o timeout errors on the NodeLocal DNS address.
The actual cause turned out to be Istio's ambient mode: a per-node ztunnel DaemonSet was transparently intercepting pod DNS syscalls (DNS capture) before they reached NodeLocal DNS. Because the interception is invisible to the pod and ztunnel's own drops surface as the exact same 169.254.20.10 timeout, every downstream metric looked healthy while the real failure happened one hop earlier.
The fix was a single annotation (ambient.istio.io/dns-capture: false) disabling capture per workload, which eliminated the timeouts entirely. The broader takeaway for anyone running a service mesh alongside Kubernetes DNS: enumerate every DaemonSet-scoped component in the request path — not just the well-known three layers — before trusting that your pod talks directly to the local resolver.