Shanghai AI Lab Releases OREAL-7B and OREAL-32B: Advancing Mathematical Reasoning with Outcome Reward-Based Reinforcement Learning

Mathematical Reasoning in AI: New Solutions from Shanghai AI Laboratory Understanding the Challenges Mathematical reasoning is a tough area for AI. While large language models (LLMs) have improved, they often struggle with multi-step logic. Traditional reinforcement learning (RL) faces limitations when the feedback is just right or wrong. Introducing OREAL Models Shanghai AI Laboratory developed the Outcome REwArd-based reinforcement Learning (OREAL) framework, which includes two models: OREAL-7B and OREAL-32B. These models perform effectively with binary feedback. OREAL uses Best-of-N sampling to boost learning and modifies negative rewards for steady performance. Performance Highlights - OREAL-7B: 94.0% pass rate on the MATH-500 benchmark, similar to larger models. - OREAL-32B: 95.0% pass rate, surpassing previous models. Technical Innovations and Advantages The OREAL framework offers key techniques for better mathematical reasoning: - Best-of-N Sampling: Chooses the best reasoning paths for improved learning. - Reward Reshaping: Adjusts negative rewards for consistency during training. - Token-Level Reward System: Focuses on crucial reasoning steps for complex tasks. - On-Policy Learning: Dynamically improves based on feedback, enhancing training efficiency. These innovations lead to better training and performance in lengthy reasoning tasks. Benchmark Performance OREAL models have shown strong results on various benchmarks: - MATH-500: Both models set new performance standards, matching or exceeding larger models. - AIME2024 and OlympiadBench: They excel across different problem types. - OREAL-32B outperforms competitors, showcasing effective training strategies. Conclusion and Future Directions OREAL-7B and OREAL-32B present innovative methods for mathematical reasoning in reinforcement learning. They tackle the challenge of limited feedback and achieve competitive performance, even at smaller scales, hinting at new opportunities for enhancing AI problem-solving. Get Involved To learn more about OREAL, check out our published research. Follow us on social media for updates and join our community for discussions on AI. Embrace AI for Business Success To stay competitive with AI, consider these steps: - Identify automation opportunities to enhance customer interactions. - Define KPIs to measure your AI projects' impact. - Choose AI solutions that suit your needs. - Implement gradually, starting small and expanding wisely. For AI KPI management advice, contact us. Stay informed about AI insights through our channels. Discover how AI can transform your sales and customer engagement on our website.