Alignment pretraining: AI discourse creates self-fulfilling (mis)alignment
A recent study explores the impact of AI discourse on alignment in large language models (LLMs). The research indicates that negative discussions about AI can lead to self-fulfilling misalignment in model behavior. Conversely, positive discourse can significantly reduce misalignment, suggesting that pretraining data plays a crucial role in shaping alignment outcomes.
- ▪The study involved pretraining 6.9B-parameter LLMs with varying amounts of alignment discourse.
- ▪Upsampling documents about AI misalignment increased misaligned behavior, while upsampling aligned behavior documents reduced misalignment scores from 45% to 9%.
- ▪The findings highlight the importance of considering pretraining for alignment in addition to post-training adjustments.
Opening excerpt (first ~120 words) tap to expand
Computer Science > Computation and Language arXiv:2601.10160 (cs) [Submitted on 15 Jan 2026 (v1), last revised 19 Feb 2026 (this version, v2)] Title:Alignment Pretraining: AI Discourse Causes Self-Fulfilling (Mis)alignment Authors:Cameron Tice, Puria Radmard, Samuel Ratnam, Andy Kim, David Africa, Kyle O'Brien View a PDF of the paper titled Alignment Pretraining: AI Discourse Causes Self-Fulfilling (Mis)alignment, by Cameron Tice and 5 other authors View PDF HTML (experimental) Abstract:Pretraining corpora contain extensive discourse about AI systems, yet the causal influence of this discourse on downstream alignment remains poorly understood.
…
Excerpt limited to ~120 words for fair-use compliance. The full article is at arXiv.org.
Discussion
0 commentsMore from arXiv.org
