Not All Tokens Are What You Need for Pretraining

Questions whether every pretraining token deserves equal weight. Selective Language Modeling scores tokens with a reference model and focuses training loss on tokens aligned with the target distribution.

Questions whether every pretraining token deserves equal weight. Selective Language Modeling scores tokens with a reference model and focuses training loss on tokens aligned with the target distribution.

When reading the original paper, identify its problem, inputs, outputs, and methodological assumptions.

Questions whether every pretraining token deserves equal weight. Selective Language Modeling scores tokens with a reference model and focuses training loss on tokens aligned with the target distribution.

Check the baselines, datasets, evaluation protocol, and ablations in the original paper. This directory overview does not provide an independent evaluation or experimental results table.