OpenAI Researchers Introduce MLE-bench: A New Benchmark for Measuring How Well AI Agents Perform at Machine Learning Engineering

**Introduction to MLE-bench** Machine Learning (ML) models can do various coding tasks, but we need a better way to assess their skills in ML engineering. Current tests often focus on simple coding, missing out on complex tasks like data preparation and debugging. **What is MLE-bench?** MLE-bench is a new testing tool created by OpenAI researchers to evaluate AI in real-world ML engineering challenges. It uses 75 competitions from Kaggle, covering areas like natural language processing and computer vision. It checks important skills such as: - Training models - Data preprocessing - Running experiments - Submitting results MLE-bench compares AI performance to human experts using metrics from Kaggle. **Structure of MLE-bench** MLE-bench rigorously tests ML engineering skills. Each competition includes: - A problem description - A dataset - Local evaluation tools - Grading code The datasets are separated into training and testing sets, ensuring accurate evaluations. AI agents are compared to human results and can earn medals based on their performance. Key metrics used for evaluation include AUROC and mean squared error. **Performance Insights** Evaluation results show that OpenAI’s o1-preview model did well, achieving medals in 16.9% of competitions. Performance improved with repeated attempts, indicating that while AI can follow known methods, it needs several tries to correct mistakes. More resources, such as longer computing time, also led to better performance. **Conclusion and Future Directions** MLE-bench is an important step in assessing AI skills in ML engineering tasks, focusing on practical skills needed for real-life applications. OpenAI plans to make MLE-bench open-source to encourage collaboration and innovation in this space. This will help identify areas for AI improvement and lead to safer, more reliable AI systems. **Getting Started with MLE-bench** To use MLE-bench, some data is stored using Git-LFS. After installing Git-LFS, run: ``` git lfs fetch --all git lfs pull ``` You can install MLE-bench with: ``` pip install -e . ``` **Connect with Us** For updates and insights, follow us on social media and subscribe to our newsletter. If you want to integrate AI into your business, reach out at hello@itinai.com. **Transform Your Business with AI** Learn how AI can improve your workflows: - Find automation opportunities - Set measurable KPIs - Select the right AI solutions - Implement AI gradually with pilot projects For more information, visit itinai.com.