A Primer on LLM Post-Training
Post-training is a crucial phase in developing Large Language Models (LLMs) that enables them to engage in human-like conversation and perform complex tasks like reasoning and tool use. Unlike pre-training, which focuses on next-word prediction, post-training teaches models conversational rules and alignment with human preferences. This phase uses structured data formats and system prompts to guide model behavior, making interactions more coherent and controlled.
- ▪Post-training, also known as alignment, teaches LLMs how to converse and reason in ways that align with human expectations.
- ▪Pre-trained models often fail to stop generating text or follow conversational turn-taking, which post-training helps correct.
- ▪The post-training process uses structured data formats with special tokens to indicate speakers and ends of turns, ensuring proper dialogue flow.
- ▪System prompts, Supervised Fine Tuning (SFT), and reward shaping are used during post-training to enforce behavioral rules.
- ▪LLMs remain fundamentally text completion systems and rely on external plumbing to manage conversational structure during inference.
Opening excerpt (first ~120 words) tap to expand
Large Language Models (LLMs) have revolutionized how we write and consume documents. In the past year or so, we have started to see them a lot more than just rephrasing docs: LLMs can now think before they act, they can plan, they can call tools like a browser, they can write code and check that it works, and a lot more – indeed, the list is growing quickly! What do all these skills have in common? The answer is that they are all developed in what we call the post-training phase of LLM training. Despite post-training unlocking capabilities that would have looked magical to us a few years ago, it surprisingly gets little coverage compared to the basics of Transformer architectures and pre-training.
…
Excerpt limited to ~120 words for fair-use compliance. The full article is at Pytorch.
Discussion
0 commentsMore from Pytorch
