Anytime Training with Schedule-Free Spectral Optimization
A new paper introduces SF-NorMuon, a schedule-free spectral optimizer designed to improve neural network training. This optimizer aims to close the performance gap with well-tuned AdamW methods, allowing for high-quality checkpoints at any training point without a fixed horizon. The research demonstrates that SF-NorMuon can match or exceed the performance of tuned baselines on large language models.
- ▪SF-NorMuon is a schedule-free spectral optimizer proposed to enhance neural network training.
- ▪It matches or exceeds the performance of tuned AdamW optimizers on large language models.
- ▪The optimizer allows practitioners to obtain high-quality checkpoints without committing to a training horizon.
Opening excerpt (first ~120 words) tap to expand
Computer Science > Machine Learning arXiv:2605.23061 (cs) [Submitted on 21 May 2026] Title:Anytime Training with Schedule-Free Spectral Optimization Authors:Anuj Apte, Pranav Deshpande, Niraj Kumar, Shouvanik Chakrabarti, Junhyung Lyle Kim View a PDF of the paper titled Anytime Training with Schedule-Free Spectral Optimization, by Anuj Apte and 4 other authors View PDF HTML (experimental) Abstract:Standard neural network training relies on learning-rate schedules tied to a fixed horizon, leading to strong path dependence and costly re-tuning as data availability changes. Schedule-Free (SF) methods address this by removing explicit schedules, yet SF-AdamW, the current state-of-the-art anytime optimizer, consistently underperforms well-tuned AdamW baselines.
…
Excerpt limited to ~120 words for fair-use compliance. The full article is at arXiv cs.AI.
Discussion
0 commentsMore from arXiv cs.AI
