791: Reinforcement Learning from Human Feedback (RLHF) — with Dr. Nathan Lambert

Reward Alignment

Aligning reward models with human preferences presents significant challenges, particularly due to objective mismatches. Nathan Lambert explains that in reinforcement learning from human feedback (RLHF), the reward model acts as a filter that must be tuned to reflect human desires accurately. This complexity is compounded by the fact that companies often provide extensive guidelines for data labeling, yet the final model may not align with these instructions.

We don't know if our methods ever could actually be perfectly aligned with what our expectations are, because we're doing this all in different modules.

--- Nathan Lambert

This alignment ceiling highlights the difficulty in achieving perfect alignment between model outputs and human expectations 1.

   

Safety

Safety in AI systems is a critical concern, especially when fine-tuning models. Nathan Lambert notes that removing safety layers during fine-tuning can compromise model integrity, as seen with models like GPT-4 and Llama 2. He argues that safety is not just about the model itself but involves a comprehensive system that includes pre-training and preference models.

Safety is much more about a complete system than a model.

--- Nathan Lambert

This holistic approach ensures that AI systems remain robust and reliable, even as they evolve 2.

   

Constitutional AI

Constitutional AI offers an alternative approach to AI alignment by using AI feedback instead of human input. Nathan Lambert describes it as a two-stage process involving instruction revision and preference labeling based on principles. This method generates synthetic data, providing a new way to refine AI models.

It's just one more way of kind of adding synthetic data.

--- Nathan Lambert

Despite its complexity, constitutional AI represents a promising direction for improving AI alignment 3.