#114 - Secrets of Deep Reinforcement Learning (Minqi Jiang)

RLHF Effects

Reinforcement Learning from Human Feedback (RLHF) plays a crucial role in shaping language models by introducing human preferences into their outputs. Minqi Jiang explains that RLHF fine-tunes models on a reward signal derived from human feedback, effectively biasing the model towards preferred outputs 1. This process can enhance reliability but also reduces diversity, as it aligns the model's responses with specific human values 2. However, Jiang notes that while RLHF is beneficial for tasks like search engines, it may limit creativity in applications like creative writing, where diversity is essential 3.

RLHF is basically sticking essentially like a smiley face on top of this, where it's essentially giving you, it's basically hiding this mess.

--- Minqi Jiang

This balance between reliability and creativity highlights the nuanced impact of RLHF on language models.

   

Intelligence Evolution

The potential for language models to evolve and self-improve raises intriguing questions about their future capabilities. Minqi Jiang discusses the possibility of models running out of novel training data by 2026, prompting a need for self-improvement and open-ended evolution 4. This evolution could lead to models generating their own training data, though Jiang emphasizes the necessity of external feedback to filter valuable outputs from the generated data 5. The concept of recursive training, where models train on their own outputs, suggests a potential plateau in development, yet reaching this point could take considerable time 6.

If we run out of data, the language model will basically end up training on largely its own outputs or variations thereof.

--- Minqi Jiang

This self-improvement process could redefine the boundaries of artificial intelligence.