Representation Without Reward: A JEPA Audit for LLM Fine-Tuning
The article discusses a study on Joint-Embedding Predictive Architectures (JEPAs) for fine-tuning language models. It examines the relationship between hidden-state representation and decoded-task accuracy, revealing a weak coupling in the current regime. The findings suggest that while some training auxiliaries show promise, none meet the stringent statistical thresholds required for significant improvements.
- ▪The study tests the effectiveness of JEPAs in improving language model fine-tuning.
- ▪Twenty-two training-time auxiliaries were compared across various metrics.
- ▪The results indicate that hidden-state representation and decoded-task accuracy are weakly coupled.
Opening excerpt (first ~120 words) tap to expand
Computer Science > Machine Learning arXiv:2605.15394 (cs) [Submitted on 14 May 2026] Title:Representation Without Reward: A JEPA Audit for LLM Fine-Tuning Authors:Biswa Sengupta View a PDF of the paper titled Representation Without Reward: A JEPA Audit for LLM Fine-Tuning, by Biswa Sengupta View PDF HTML (experimental) Abstract:Joint-embedding predictive architectures (JEPAs) propose that a model should learn more useful abstractions when trained to predict latent representations rather than observed outputs. For autoregressive language-model fine-tuning the principle entails a stricter requirement: the induced hidden-state geometry must reach the language-model head \emph{and} improve the decoded task metric.
…
Excerpt limited to ~120 words for fair-use compliance. The full article is at arXiv cs.AI.
Discussion
0 commentsMore from arXiv cs.AI
