Introduction
Course introduction
()
1. Foundations of Alignment
Key topics
()
Alignment goals and failure modes
()
Framing the alignment problem for LLMs
()
Post-training overview: SFT, preference optimization, and RLHF
()
2. Reinforcement Learning Basics for LLMs
Key topics
()
Policies, rewards, and exploration
()
Common algorithms used in post-training
()
PPO in action using Flan-T5
()
3. Building a Reward Model
Topics
()
Foundations of reward modeling
()
Training a reward model
()
4. Modern Post-Training
Topics
()
Why RLHF evolved
()
Direct preference optimization (DPO)
()
REINFORCE leave-one-out (RLOO)
()
5. Advanced Post-Training for Reasoning
Key topics
()
Why reasoning needs a different post-training method
()
Group relative policy optimization (GRPO)
()
Case study: Inducing reasoning in LLMs with GRPO
()
6. Future Directions, Safety, and Governance
Key topics
()
Practical safety and robustness benchmarks
()
Building a simple safety evaluation harness
()
Governance and risk frameworks
()
Joint safety evaluations and red teaming
()
Case study: Constitutional AI plus safety evaluation
()
Conclusion
Course summary and next steps
()