Fixing False Positives in Prometheus Predictive Alerting
Why Prometheus-based predictive alerting models that ace offline tests turn noisy in production, and how query alignment, rate() handling and schema checks fix it.
An anomaly detection model that scored 94% precision offline started firing false predictive alerts in production the moment it was wired to live Prometheus query_range calls, triggering on pod restarts and scrape gaps. The root cause traced back not to the classification threshold but to a mismatch between the training and inference queries: differing step/resolution settings, raw counter metrics fed directly into the model instead of being wrapped in rate(), unhandled NaNs from Prometheus staleness marking, and cardinality explosions from open labels like pod_name during rolling deploys.
The fix has two parts. First, make the live query mathematically equivalent to the training query — pin the step value as a named constant, apply rate()/increase() at the PromQL level rather than post-processing counters in pandas, and tolerate only short gaps via limited forward-fill while hard-failing on longer ones instead of scoring garbage. Second, freeze the feature contract — move feature extraction behind stable recording rules using a level:metric:operation naming convention, hash the ordered feature list at training time and re-verify it on every inference call, and version model artifacts with both the feature hash and the scikit-learn version to catch silent schema drift.
The takeaway for engineers running predictive alerting on Prometheus is that offline accuracy metrics can't catch query semantic drift or silent schema changes — only an explicit, versioned feature contract and cardinality/staleness guards in the pipeline can. This makes the ingestion layer, not the model, the real reliability boundary for production ML on observability data.