Sparse Policy Selection in RL for LLM Reasoning, Not Capability Learning
Reinforcement learning enhances LLM reasoning by focusing on sparse policy selection rather than teaching new capabilities.
Reinforcement learning (RL) has become the standard for enhancing reasoning in large language models (LLMs), yet it appears to redistribute existing strategies rather than teach new ones. This study questions the necessity of the RL optimization loop, revealing that its benefits are concentrated at high-entropy decision points. The findings suggest that reasoning improvement is more about sparse policy selection than capability acquisition, leading to the development of ReasonMaxxer, a method that matches RL performance with significantly reduced training costs.