World models remember, actors forget: fixing catastrophic forgetting in RL

DreamerV3-family model-based reinforcement learning agents forget catastrophically when trained on sequential tasks — even when an unbounded replay buffer preserves every past experience. This work asks a question the continual-RL literature has long assumed but never actually measured: which component is responsible for the forgetting?

Pre-registered, component-level probes show the world model retains nearly everything measurable about old tasks — reward discrimination, value estimates, termination structure — while the actor's behavior collapses. The failure is a channel problem, not a memory problem.

Intervention experiments confirm this: with the world model frozen, reinforcement learning inside imagined rollouts fails to recover a lost skill, but supervised self-imitation on the world model's own graded 'dreams' recovers it fully with zero environment interaction. Interleaved during training, this dream rehearsal yields a task-label-free, parameter-constant continual learner that outperforms plain replay and matched real-episode cloning across four- and eight-task chains.

The study also shows the dream-grading step is load-bearing, characterizing two failure modes and an offline selection gauge that catches them early. All experiments were pre-registered and run on a single GPU, with every refuted hypothesis reported.