Ashley Edwards - Genie Paper (DeepMind/Runway)

Action Mapping

Mapping AI-generated latent action spaces to real-world robotics applications presents significant challenges. Ashley Edwards explains that while increasing the action space can improve the fidelity of AI-generated actions, translating these into continuous real-world robotics actions is complex 1. The process involves mapping latent actions to real actions, often requiring expert data to create effective mappings 2.

How do you map this giant latent action space to the real actions that you can take? Robotics actions are continuous, so that's another thing that we would have to figure out.

--- Ashley Edwards

This complexity is compounded by the need to discretize continuous actions, highlighting the intricate nature of integrating AI with robotics.

   

Foundation Models

The development of foundational models for robotics involves integrating large-scale video datasets, which poses unique challenges. Ashley Edwards notes that while video data from the internet can be useful, it often lacks the specific information needed for robotics, such as how to manipulate objects 1. Collaborative efforts, like those from Chelsea Finn's lab, are crucial in creating comprehensive datasets that include diverse sources beyond just internet videos.

Chelsea Finn's lab is also learning. Like they have this giant database coming from many different labs, which I think is quite cool.

--- Ashley Edwards

These models must also incorporate sensor data to capture elements like haptics, which are not visible in video data.