Paper of the Week — Which Tokens Should SFT Actually Learn? A Token-Trimming Perspective on Mathematical Reasoning
Token-level SFT masking on math reasoning: a small-team paper showing which tokens actually hurt training, with released analysis tools.
Which Tokens Should SFT Actually Learn? A Token-Trimming Perspective on Mathematical Reasoning
Yaning Jia, Chunhui Zhang, Wenxuan Xu, Xingjian Diao, Xiaoyuan Wang, Soroush Vosoughi. Published 2026-09-09. arXiv:2609.09707
One sentence summary
Uniform cross-entropy over all tokens during SFT actively hurts math reasoning — selectively masking high-confidence and low-confidence tokens yields measurable accuracy gains without any architecture changes.
Why this paper
With RL-based post-training dominating the frontier conversation, SFT is quietly becoming the workhorse for teams who can’t afford GRPO or PPO runs on every fine-tune. A paper that squeezes more out of the SFT recipe with zero inference overhead is immediately deployable.
What they did
Standard SFT applies the same cross-entropy loss to every target token in a reasoning trace — whether it’s a trivial connector word the model already predicts with 99% confidence, or a genuinely uncertain mathematical step where the model is flailing. The authors analyze this mismatch and propose a token-trimming strategy: identify tokens that are either already mastered (near-zero loss, no gradient signal) or pathologically uncertain (noisy, potentially wrong label), then exclude or downweight them during training. The result is a cleaner gradient signal focused on the tokens that actually teach reasoning.
Key findings
- Uniform SFT over-trains on “easy” tokens, compressing the model’s distribution on steps it already handles correctly, which the authors call “over-sharpening”
- Low-confidence tokens (the model assigns high uncertainty) carry noisy learning signal and disproportionately inflate training loss without improving downstream accuracy
- Trimming both extremes of the per-token confidence distribution — keeping only the middle band — consistently improves math benchmark accuracy vs. full-token SFT baselines
- Gains are additive on top of chain-of-thought data and do not require any changes to the optimizer, learning rate schedule, or model architecture
- The approach works across model scales tested, suggesting the token-confidence mismatch is a general SFT pathology rather than a small-model artifact
Why it matters for practitioners
If you’re fine-tuning any model — Qwen3.6, Mistral Large 3, or a smaller distilled checkpoint — on reasoning traces, you’re almost certainly over-fitting to tokens that teach nothing and under-weighting the tokens that matter. This gives you a concrete pre-processing and loss-masking step you can add to an existing SFT pipeline in an afternoon. The payoff is particularly relevant for teams generating synthetic CoT data at scale, where label noise at uncertain steps is a real and underappreciated problem.
What you can use today
- Compute per-token model confidence (log-prob under your base or reference model) on your SFT dataset before training, and build a histogram to identify the over-represented high-confidence tail
- Implement a token-level loss mask in your training loop that zeros gradients on tokens above a confidence threshold (already mastered) and below a floor threshold (too noisy) — both thresholds can be set with a small held-out math eval
- Apply this to any chain-of-thought or scratchpad dataset, not just math: the same over-sharpening dynamic appears wherever some reasoning steps are formulaic and others are genuinely hard