Cross-tokenizer distillation can silently erase 85% of teacher signal
Naive logit projection across mismatched tokenizers can erase 85% of teacher information; chain-rule conditioning restores retention to 86%.
An engineer distilling from a large-vocabulary teacher (Qwen2.5-Coder, 151K tokens) into a small-vocabulary student (BPE 1-4K) uncovered a subtle but severe information-loss problem. The standard approach projects the teacher's top-K logits onto the student's first token at each segment boundary. Because this mapping is many-to-one — words like 'the', 'then', and 'they' collapse onto the same initial student token — most of the teacher's uncertainty gets summed away. Measured entropy dropped from 2.09 bits to 0.32 bits, retaining only about 15% of the teacher's information. Counterintuitively, expanding the student vocabulary to 4096 tokens made retention slightly worse (13%).
Using this degraded target, a preliminary knowledge distillation (KD) run underperformed a plain cross-entropy baseline (2.729 vs 2.081). Left unexamined, this could easily be misread as evidence that distillation simply doesn't work for this vocabulary mismatch.
The fix leverages the chain rule: the teacher's uncertainty factorizes across the sequence of student tokens, and the ambiguity between 'the/then/they' is actually resolved at later tokens in the span, not the first one. By conditioning the stored top-K rows on the bytes already emitted within the segment, retention jumps to 83-86% — with no extra compute or storage overhead. It's a practical, low-cost correction for anyone doing distillation across mismatched tokenizers.