New method flags risky tool calls in AI agents before they happen
Researchers built a sparse-autoencoder and probe-based toolkit that reads AI agent internals to flag risky or unnecessary tool calls before execution happens.
AI agents deployed in enterprise workflows can misfire on tool-use decisions—skipping necessary calls, invoking unneeded tools, or triggering effects that only become visible after execution. Existing observability approaches like prompt inspection, output scoring, and logging operate after the fact, which is costly in long-horizon tasks where an early mistake can cascade through the rest of the execution trajectory, inflate token usage, and introduce safety risks.
The paper introduces a mechanistic-interpretability toolkit that reads a model's internal state before it acts. Sparse autoencoders decompose activations into interpretable sparse features, while lightweight linear probes read signals from those features to predict whether a tool call is needed and how risky the next action is likely to be. The researchers pinpoint which layers and features are most tied to tool-use decisions and confirm their functional relevance through feature ablation. Probes were trained on multi-step agent traces from the NVIDIA Nemotron function-calling dataset and then applied to GPT-OSS 20B and Gemma 3 27B.
Rather than replacing external evaluation, the framework adds a missing layer of insight: what the model internally signaled before taking action. This is especially useful for diagnosing failures in long-horizon agent runs, where early mistakes compound over time, and it demonstrates a broader path for using mechanistic interpretability as an internal observability tool for monitoring tool calls and risk in agentic systems.