When Mean CE Fails: Median CE Can Better Track Language Model Quality
The paper discusses the limitations of mean cross-entropy (CE) as a metric for evaluating language model quality. It highlights scenarios where mean CE fails to accurately reflect model performance, suggesting that median CE may be a better alternative. The authors recommend using percentile CE summaries alongside mean CE for a more comprehensive assessment of model quality during training.
- ▪Mean cross-entropy is commonly used to validate language models but can misrepresent model quality.
- ▪In experiments, median CE showed a stronger correlation with task performance than mean CE.
- ▪The authors propose reporting percentile CE summaries to better track distribution changes during training.
Opening excerpt (first ~120 words) tap to expand
Computer Science > Artificial Intelligence arXiv:2605.24667 (cs) [Submitted on 23 May 2026] Title:When Mean CE Fails: Median CE Can Better Track Language Model Quality Authors:Hao Guo, Simon Dennis, Rivaan Patil, Kevin Shabahang View a PDF of the paper titled When Mean CE Fails: Median CE Can Better Track Language Model Quality, by Hao Guo and 3 other authors View PDF HTML (experimental) Abstract:Mean cross-entropy is the standard validation metric for language models, but it can fail to track model quality during training. We examine this in two common scenarios. First, in Qwen2.5-1.5B SFT on synthetic fact-learning, we find that mean CE rises substantially after the initial learning phase while held-out fact-recall accuracy remains near its peak.
…
Excerpt limited to ~120 words for fair-use compliance. The full article is at arXiv cs.AI.
Discussion
0 commentsMore from arXiv cs.AI
