#49 - Meta-Gradients in RL - Dr. Tom Zahavy (DeepMind)

Exploration

Tom Zahavy highlights the complexities of reinforcement learning (RL), emphasizing the balance between exploration and exploitation. He notes that while RL has potential, it faces challenges like non-stable training and sample inefficiency. Zahavy shares his experience at DeepMind, where access to resources allowed for deeper understanding and communication of RL methods to the community 1.

Reinforcement learning works. Every problem that I try to solve, eventually I managed to get learning, or I managed to do something in it.

Despite these challenges, he believes RL can solve complex problems, though it requires expertise and resources 2.

   

Non-stationarity

Addressing non-stationarity in RL environments, Zahavy discusses how meta gradients can stabilize learning. He explains that RL extends supervised learning by incorporating exploration and credit assignment, allowing for more complex problem-solving 3. Meta gradients, he argues, enable agents to adapt dynamically to changing environments, enhancing their ability to solve non-stationary problems.

Reinforcement learning is basically building on everything that was done in the bandit literature and later in the theoretical community on mdps.

This approach, he suggests, can lead to more efficient learning and better performance in diverse environments 4.