Google DeepMind Researchers Propose RT-Affordance: A Hierarchical Method that Uses Affordances as an Intermediate Representation for Policies

Recent Advances in Robot Policy Representation **Understanding Policy Representation** Robots have made significant strides in learning how to make decisions. "Policy representation" refers to the ways robots choose their actions, allowing them to adapt to new tasks and environments. **Introducing Vision-Language-Action Models** Vision-language-action (VLA) models are trained with a lot of robot data. They merge visual understanding, language skills, and decision-making to help robots perform various tasks. While these models show potential, they are not yet reliable for everyday use outside of labs. **Challenges with Current Representations** Existing methods like language descriptions, goal images, and trajectory sketches have their limitations: - **Language-based guidance** often lacks detail, making tasks difficult. - **Goal images** provide spatial data but can be too complex for robots to learn from. - **Trajectory sketches** offer action guidance but may lack specifics for precise movements. **Introducing RT-Affordance** Researchers from Google DeepMind created a new model called RT-Affordance. This model generates an "affordance plan" based on task language, guiding the robot's actions. - **Affordances** describe how robots can interact with objects based on their design. - The RT-Affordance model uses data from various sources, including web datasets and robot movements. **How RT-Affordance Works** 1. An affordance plan is made using task language and an initial image. 2. This plan is combined with language instructions to guide the robot. 3. The robot uses this plan to effectively interact with objects. This approach uses extensive data, improving the model's ability to handle new tasks and environments. **Experimental Success** The research team tested the model, focusing on improved robotic grasping, especially with complex items like kettles and pots: - The RT-Affordance model achieved success rates of 68%-76%, compared to just 24%-28% for previous models. - In object placement tasks, it reached a success rate of 70%. While the model performs well, it slightly struggles with entirely new objects. **Conclusion** The RT-Affordance method improves the reliability and adaptability of robot policies, making it a valuable tool for a range of tasks. Although it has some challenges with completely new actions, it outperforms traditional methods and sets the stage for future advancements in robotics. **Transform Your Business with AI** Enhance your company’s competitiveness with AI by following these steps: - **Identify Automation Opportunities**: Find areas for AI in customer interactions. - **Define KPIs**: Measure AI's impact on your business. - **Select AI Solutions**: Choose tools that fit your needs. - **Implement Gradually**: Start small, learn, and expand carefully. For AI management advice, contact us at hello@itinai.com. Stay updated on leveraging AI by following us on Telegram or Twitter!