Data Scaling as Progressive Coverage of a Predictive Contribution Spectrum
The paper explores the hypothesis that real-data scaling laws are influenced by a latent predictive contribution spectrum. It presents a method using a suffix-automaton representation to analyze text corpora and defines a global-KL predictive contribution spectrum. The findings indicate a strong correlation between the tail slope of this spectrum and the empirical data-scaling exponent of a small GPT learner.
- ▪The research investigates how real-data scaling laws are governed by a predictive contribution spectrum.
- ▪A suffix-automaton representation of text corpora is utilized to define a global-KL predictive contribution spectrum.
- ▪The study finds a strong correlation between the tail slope of the spectrum and the data-scaling exponent of a GPT learner.
Opening excerpt (first ~120 words) tap to expand
Computer Science > Computation and Language arXiv:2605.20196 (cs) [Submitted on 5 Apr 2026] Title:Data Scaling as Progressive Coverage of a Predictive Contribution Spectrum Authors:Zihui Song, Shihao Ji, Hongxi Li, Shuaizhi Cheng, Chunlin Huang View a PDF of the paper titled Data Scaling as Progressive Coverage of a Predictive Contribution Spectrum, by Zihui Song and 4 other authors View PDF HTML (experimental) Abstract:We investigate the hypothesis that real-data scaling laws are governed by progressive coverage of a latent predictive contribution spectrum rather than by token-frequency tails alone.
…
Excerpt limited to ~120 words for fair-use compliance. The full article is at arXiv cs.AI.
Discussion
0 commentsMore from arXiv cs.AI
