When logs stayed silent but metrics caught the event
A memory spike in a network lab reveals why gNMI reads leave no log trace, and how time-series metrics caught what logs never recorded.
An engineer running a containerlab-based Arista cEOS network lab noticed an unexpected memory spike on a Grafana dashboard. Checking four separate log layers—EOS agent logs, ConfigAgent, Sysdb, and system logs—turned up nothing. Early theories proved wrong: a spike in systemctl process counts was actually a metric-expiry artifact, and along the way the investigation stumbled on an unrelated but striking bug—rsyslog silently crash-looping roughly 2.4 million times.
The real cause emerged only after examining full time-series data instead of two-point snapshots: a sharp, simultaneous jump across four routers followed by a slow decay. That signature pointed to a read operation rather than a write, since configuration writes get logged but reads do not. Controlled experiments confirmed it—CLI-based reads left memory untouched, while a bulk gNMI read through Sysdb/ConfigAgent filled internal memory pools that were released only gradually.
The story offers concrete lessons for network engineers and observability practitioners: relying on logs alone can leave an entire class of read-based activity invisible. Two-point comparisons can mislead; full time-series analysis is essential. Understanding exactly what a metric measures matters, and any hypothesis remains unproven until validated through a reproducible experiment.