Audit Finds LLM Agent Credit Signals No Better Than Random Chance

A new study audited the step-level credit assignment signals commonly used to train LLM agents—LLM-judge scores, outcome-conditioned logprob ratios, and the policy's own confidence—against real causal ground truth obtained by resampling and replaying decisions in the ALFWorld tool-use environment. None of these signals identified which steps causally mattered better than random guessing.

The key distinction is between step 'correctness,' which prior evaluations measured, and step 'contribution,' which requires testing what actually changes when the policy's alternative choices are resampled and rolled forward. Causal contribution turned out to be sparse—only about 30% of decision points showed a measurable effect—and this varied by a factor of two between similarly-sized models.

Existing credit signals largely tracked the policy's fluency rather than genuine causal impact, and conditioning on outcomes added essentially no causal information. A follow-up training experiment across seven configurations found that once training dose (example count, optimizer steps) was controlled for, no arm reliably beat the untrained baseline—apparent differences were explained by dose, not by the credit signal's actual content.

For teams building RL-style training pipelines for agentic LLMs, the findings suggest that judge- or confidence-derived step rewards may not reliably capture which decisions matter, and that comparing credit assignment methods requires matching effective sample size to avoid confusing training dose with credit quality.