ScreenSpot-Pro: The First Benchmark Driving Multi-Modal LLMs into High-Resolution Professional GUI-Agent and Computer-Use Environments

**Challenges Faced by GUI Agents in Professional Environments** GUI agents face three main challenges in professional settings: 1. **Complex Applications**: Professional software is complicated, requiring a good understanding of complex layouts. 2. **High Resolution**: These tools often have high resolutions, making targets smaller and interactions less accurate. 3. **Additional Tools**: The use of extra tools and documents complicates workflows. These challenges show the need for better solutions to improve GUI agent performance. **Limitations of Current GUI Grounding Models** Current GUI grounding models do not meet the needs of professional environments: - Tools like ScreenSpot are designed for low-resolution tasks and do not reflect real-world situations well. - Models such as OS-Atlas and UGround struggle with small targets and icon-heavy interfaces. - Lack of multilingual support limits their usability in global contexts. These issues highlight the need for more realistic benchmarks in the field. **Introducing ScreenSpot-Pro** A team from various universities has created ScreenSpot-Pro, a framework for high-resolution professional environments. Key features include: - A dataset with 1,581 tasks across 23 applications in different industries. - High-resolution visuals and expert annotations for accuracy. - Multilingual guidelines in English and Chinese. ScreenSpot-Pro documents real workflows, making it a valuable tool for improving GUI grounding models. **Realistic Dataset Characteristics** ScreenSpot-Pro captures challenging scenarios with: - High-resolution images where target areas are only 0.07% of the total screen. - Data collected by professionals using specialized tools for accurate annotations. - Support for bilingual functionality and various workflows. This dataset is essential for enhancing the accuracy and flexibility of GUI agents. **Performance Analysis of GUI Grounding Models** Analysis using ScreenSpot-Pro shows significant shortcomings in current models: - OS-Atlas-7B achieved only 18.9% accuracy. - Iterative methods like ReGround improved performance to 40.2% through fine-tuning. - Small components and bilingual tasks posed challenges for these models. These results emphasize the need for improved techniques to enhance understanding in complex GUI environments. **Transformative Impact of ScreenSpot-Pro** ScreenSpot-Pro sets a new standard for evaluating GUI agents in high-resolution professional settings. It addresses complex workflow challenges and provides a precise dataset to drive innovation. This leads to smarter, more efficient agents that boost productivity across all industries. **Get Involved** Explore the paper and data for more insights. Follow us on Twitter, join our Telegram channel, and connect with our LinkedIn group. Join our 60k+ ML SubReddit for ongoing discussions. **Webinar Invitation** Join our webinar for practical insights on improving LLM model performance while ensuring data privacy. **Leverage AI for Your Business** Stay competitive by using ScreenSpot-Pro to enhance your professional workflows: - **Identify Automation Opportunities**: Spot areas where AI can be integrated. - **Define KPIs**: Measure the impact of your AI efforts. - **Select an AI Solution**: Choose tools that meet your needs. - **Implement Gradually**: Start small, gather data, and expand wisely. For AI KPI management advice, reach out at hello@itinai.com. Stay updated on AI insights via our Telegram or Twitter. **Transform Your Sales and Customer Engagement** Discover how AI can improve your sales processes at itinai.com.