Tim & Heinrich — Democraticizing Reinforcement Learning Research

Engineering Challenges

The engineering difficulties in reinforcement learning (RL) are highlighted through the comparison of NetHack to other complex games like StarCraft. Heinrich Kuttler explains that while StarCraft requires immense computational resources, NetHack offers a challenging yet accessible environment for RL research 1. This accessibility allows researchers to explore long-range dependencies and strategic decision-making without the need for extensive resources. Tim Rocktäschel notes that current RL agents struggle with high-level planning and often optimize for short-term gains without considering past experiences 2.

Our agents optimize the current situation without any regard for the past.

--- Heinrich Kuttler

These challenges underscore the need for advancements in RL strategies to improve memory and planning capabilities.

   

Training Complexities

Training complexities in reinforcement learning are compounded by high variance in results. Tim Rocktäschel emphasizes the importance of multiple training runs to ensure reliable outcomes, as single runs can yield vastly different results 3. Despite these challenges, vanilla agents have shown surprising progress in NetHack, achieving scores that rival novice human players. Rocktäschel describes how agents learn through procedural generation, encountering simpler scenarios that help them develop skills for more complex tasks 4.

Our agents right now, just by optimizing for score, they average at a score of, I think, 750 ish, roughly.

--- Tim Rocktäschel

This progress is encouraging for the development of more sophisticated models.