One encoder, seven heads: lessons from training a unified security classifier

In recent months, we consolidated seven separate sequence classifiers into a single multi-head model. This model utilizes a shared mmBERT-small encoder with seven task heads. Our training rows only carry labels for a subset of tasks, leading us to mask absent tasks from the loss entirely. We developed a self-test to ensure absent-task gradients are zero, which helped identify subtle bugs. The results indicate that the unified model operates more efficiently than seven dedicated models, although some tasks show slight performance drops.