Final Token Preference Optimization Tackles Reasoning Model Doom Loops
Antidoom uses Final Token Preference Optimization to fix repetitive doom loops in reasoning models, cutting loop rates sharply in LFM2.5 and Qwen3.5 without broad model degradation.
Small reasoning models often fall into 'doom loops,' repeating a span like 'Wait, let me reconsider' until the context window is exhausted, especially on hard math and coding prompts. The analysis traces this to three compounding mechanisms: overtrained high-probability tokens (such as Wait, So, Alternatively), prior context reinforcing the same repeated sequence, and greedy or low-temperature sampling that traps the model with no exit. Standard fixes like repetition_penalty are blunt band-aids, while RL-based corrections require costly rollouts and careful reward tuning.
The proposed Antidoom method instead applies Final Token Preference Optimization (FTPO), an Antislop-derived variant of DPO, to make a surgical fix: it identifies the exact token that kicks off a loop, trains the model to prefer coherent alternatives only at that position, and leaves the rest of the distribution largely untouched. FTPO trains only the trailing token mid-generation, spreads probability across multiple chosen alternatives, and uses a logit-space KL-like regularizer to avoid disturbing unrelated tokens.
On an early LFM2.5-2.6B checkpoint, the doom-loop rate fell from 10.2% to 1.4%, and on Qwen3.5-4B it dropped from 22.9% to 1%, with evaluation scores improving substantially in both cases. Training uses LoRA (rank 128-256) for a single epoch with early stopping at a chosen_win threshold of 0.35, since longer training tends to introduce new degeneration issues.
For engineers, the significance lies in removing a failure mode—not adding new capability—so models reach answers they could already produce, using a cheap, targeted intervention rather than expensive RL loops or generic penalty heuristics.