AVSD: Adaptive-View Self-Distillation by Balancing Consensus and Teacher-Specific Privileged Signals
The paper introduces AVSD, a novel method for self-distillation in language models. AVSD addresses challenges related to using multiple privileged information views by balancing consensus and view-specific signals. Experiments demonstrate that AVSD outperforms existing self-distillation methods on various benchmarks.
- ▪AVSD stands for Adaptive-View Self-Distillation, which utilizes multiple privileged information views.
- ▪The method reconstructs token-level supervision by separating stable cross-view consensus from view-specific residual signals.
- ▪Experiments show that AVSD achieves significant performance gains over single-view self-distillation baselines.
Opening excerpt (first ~120 words) tap to expand
Computer Science > Machine Learning arXiv:2605.20643 (cs) [Submitted on 20 May 2026] Title:AVSD: Adaptive-View Self-Distillation by Balancing Consensus and Teacher-Specific Privileged Signals Authors:Duy Nguyen, Hanqi Xiao, Archiki Prasad, Zaid Khan, Anirban Das, Austin Zhang, Sambit Sahu, Hyunji Lee, Elias Stengel-Eskin, Mohit Bansal View a PDF of the paper titled AVSD: Adaptive-View Self-Distillation by Balancing Consensus and Teacher-Specific Privileged Signals, by Duy Nguyen and 9 other authors View PDF HTML (experimental) Abstract:Self-distillation enables language models to learn on-policy from their own trajectories by using the same model as both student and teacher, with the teacher being conditioned on privileged information unavailable to the student.
…
Excerpt limited to ~120 words for fair-use compliance. The full article is at arXiv cs.AI.
Discussion
0 commentsMore from arXiv cs.AI
