UC Berkeley and NYU AI Research Explores the Gap Between the Visual Embedding Space of Clip and Vision-only Self-Supervised Learning

UC Berkeley and NYU AI Research Explores the Gap Between the Visual Embedding Space of Clip and Vision-only Self-Supervised Learning AI News, AI, AI tools, Dhanshree Shripad Shenwai, Innovation, itinai.com, LLM, MarkTechPost, t.me/itinai **Advancements in Multimodal Large Language Models (MLLMs)** Recent research has shed light on the potential of Multimodal Large Language Models (MLLMs) in tasks such as visual question answering, instruction following, and image understanding. However, these models still exhibit visual flaws that impact their performance. **Identifying Visual Representation Issues** Studies from UC Berkeley and New York University have identified visual representation issues as a potential cause of MLLM deficiencies. The use of pretrained vision and language models, such as the Contrastive Language-Image PreTraining (CLIP) model, in MLLMs has been found to introduce flaws that affect their performance. **Introducing MultiModal Visual Patterns (MMVP)** A new benchmark called MultiModal Visual Patterns (MMVP) has been introduced to evaluate the visual capacities of MLLMs. This benchmark specifically addresses disparities in CLIP-blind pairings and has revealed significant performance gaps in state-of-the-art MLLMs. **Enhancing Visual Foundation of MLLMs** To address these challenges, a method called Mixture-of-Features (MoF) has been developed to improve MLLMs’ visual grounding capabilities. By integrating a vision-only self-supervised model like DINOv2, this approach has shown promising results in improving visual anchoring while maintaining the ability to follow instructions. **Implications for AI Solutions** The research findings emphasize the need for new assessment metrics and algorithms for visual representation learning. It also highlights the strengths and weaknesses of vision-and-language models and vision-only self-supervised learning models. This insight can guide the selection and implementation of AI solutions for middle managers. **Practical AI Solutions for Middle Managers** For middle managers looking to leverage AI, it’s essential to identify automation opportunities, define KPIs, select suitable AI solutions, and implement them gradually. By staying informed about advancements in AI and exploring practical AI solutions, companies can redefine their work processes and stay competitive in the evolving landscape. **Spotlight on a Practical AI Solution** Consider the AI Sales Bot from [itinaicom/aisalesbot](https://www.itinai.com/aisalesbot), designed to automate customer engagement and manage interactions across all customer journey stages. This solution can redefine sales processes and customer engagement, providing a valuable tool for middle managers seeking to evolve their company with AI. **List of Useful Links:** - AI Lab in Telegram [@aiscrumbot](https://t.me/aiscrumbot) – free consultation - [UC Berkeley and NYU AI Research Explores the Gap Between the Visual Embedding Space of Clip and Vision-only Self-Supervised Learning](https://www.marktechpost.com) - Twitter – @itinaicom