Salesforce AI Research Propose Programmatic VLM Evaluation (PROVE): A New Benchmarking Paradigm for Evaluating VLM Responses to Open-Ended Queries

Understanding Vision-Language Models (VLMs) Vision-Language Models (VLMs) help answer questions about images. However, they sometimes give answers that seem correct but are actually wrong, which is called hallucination. This can make people doubt these systems, especially in important situations. Evaluating VLMs is Challenging It's hard to judge how useful and accurate VLM answers are. This requires understanding the images and checking each statement. Traditional evaluation methods often focus on simple questions or lack the context needed for more complex ones. Introducing PROVE: A New Evaluation Method Salesforce AI Research has created a new evaluation method called Programmatic VLM Evaluation (PROVE). This method assesses VLM answers to open-ended visual questions using detailed scene graphs derived from thorough image captions. How PROVE Works PROVE uses a large language model (LLM) to create various question-answer pairs and programs to check these pairs. This results in a dataset of 10,500 challenging and visually grounded question-answer pairs. The evaluation measures both the usefulness and accuracy of VLM responses using a unified framework based on scene graph comparisons. Benefits of the PROVE Benchmark The PROVE benchmark improves VLM evaluation by using detailed scene graphs and verification programs. This ensures that only verifiable question-answer pairs are included, leading to a high-quality dataset. The evaluation compares scene graph representations from model responses and correct answers to assess usefulness and accuracy. Key Findings Current VLMs often find it hard to balance usefulness and accuracy. While models like GPT-4o and Phi-3.5-Vision are helpful, they don't always provide correct answers. Interestingly, smaller models like LLaVA-1.5 have better accuracy scores, showing that size doesn't always mean better accuracy. Conclusion PROVE is a major advancement in evaluating VLM responses. By using detailed representations and programmatic checks, it offers a more reliable assessment method. The findings emphasize the need for VLMs that can provide both informative and accurate responses, especially as their use increases. Transform Your Business with AI Stay competitive by using AI solutions. Here’s how: 1. Identify Automation Opportunities: Find customer interactions that can benefit from AI. 2. Define KPIs: Ensure measurable impacts from your AI projects. 3. Select an AI Solution: Choose tools that fit your needs and allow customization. 4. Implement Gradually: Start small, gather data, and expand wisely. For AI KPI management advice, connect with us at hello@itinai.com. For ongoing insights, follow us on Telegram or Twitter. Explore AI Solutions Discover how AI can improve your sales processes and customer engagement at itinai.com.