Standard reinforcement learning methods like Q learning can struggle in practice, particularly when relying on off-policy data. The key issue lies in the counterfactual nature of Q learning, where the policy exploits the Q function’s inaccuracies, leading to potentially harmful actions. Recognizing this out-of-distribution problem opens the door to exploring effective solutions.
