791: Reinforcement Learning from Human Feedback (RLHF) — with Dr. Nathan Lambert

Historical Roots

Reinforcement Learning from Human Feedback (RLHF) has deep historical roots, drawing from ancient philosophy and modern economics. Nathan Lambert highlights the influence of Aristotle and the von Neumann Morgenstern utility theorem, which allows for expressing behaviors as probability distributions 1. This historical context raises questions about the alignment of AI models with human preferences, especially when models are trained on internet data rather than curated datasets.

It's really just a super long list of questions of like, why we should look at other social sciences if we're making grand claims about human preferences.

--- Nathan Lambert

Understanding these foundations is crucial for developing AI systems that truly reflect human values and preferences.

   

Bias Correction

RLHF plays a significant role in correcting biases inherent in AI models trained on common data sources like Reddit. Nathan Lambert explains that fine-tuning, even with less computational power, can present information compellingly, akin to rewriting history in a captivating way 2. This process helps in creating models that resonate with users by addressing biases and enhancing the model's style and output.

RLHF is doing that at a small scale, which is like all these base models have similar things in them.

--- Nathan Lambert

By refining the model's presentation, RLHF ensures that AI outputs are more aligned with human expectations and less biased by the data's original context.

   

Model Steering

OpenAI utilizes RLHF to steer AI models towards desired behaviors, as detailed in their model specification documents. Nathan Lambert notes that RLHF introduces a human element into AI models, allowing them to exhibit a degree of humanness that is difficult to achieve through data alone 3. This technique is a popular fine-tuning method, offering flexibility in adjusting models to better reflect human feedback and societal values.

RLHF is a tool for doing this.

--- Nathan Lambert

The openness in AI development, as advocated by Lambert, aims to democratize AI, preventing corporate monopolies and encouraging broader participation in AI innovation.