Is ConvNet Making a Comeback? Unraveling Their Performance on Web-Scale Datasets and Matching Vision Transformers

Is ConvNet Making a Comeback? Unraveling Their Performance on Web-Scale Datasets and Matching Vision Transformers AI News, AI, AI tools, Innovation, itinai.com, LLM, MarkTechPost, Pragati Jhunjhunwala, t.me/itinai 🔍 Researchers Challenge Belief: ConvNets vs. ViTs on Large Datasets In a recent study, researchers have challenged the prevailing belief that Vision Transformers (ViTs) outperform Convolutional Neural Networks (ConvNets) when given access to large web-scale datasets. They introduce a ConvNet architecture called NFNet, pre-trained on the massive JFT-4B dataset containing 4 billion labeled images from 30,000 classes. The aim is to evaluate the scaling properties of NFNet models and compare their performance to ViTs with similar computational budgets. 📈 The Rise of ViTs and the Need for Evidence ViTs have gained popularity in recent years, with many believing they outperform ConvNets, especially with large datasets. However, this belief lacks substantial evidence, as most studies have compared ViTs to weak ConvNet baselines. Additionally, ViTs have been pre-trained with significantly larger computational budgets, raising questions about the actual performance differences between these architectures. 💡 Introducing NFNet and Evaluating Performance ConvNets, specifically ResNets, have been the go-to choice for computer vision tasks for years. However, the rise of ViTs has shifted the focus to models pre-trained on large web-scale datasets. The researchers introduce NFNet, a ConvNet architecture, and pre-train it on the vast JFT-4B dataset without significant modifications. They examine how the performance of NFNet scales with varying computational budgets. 📊 Results and Findings The research team trains different NFNet models with varying depths and widths on the JFT-4B dataset. They find that larger computational budgets lead to better performance, observing a log-log scaling law. The most expensive pre-trained NFNet model achieves an ImageNet Top-1 accuracy of 90.3%. By introducing repeated augmentation during fine-tuning, they achieve a remarkable 90.4% Top-1 accuracy. Comparatively, ViT models achieve similar performance with more substantial pre-training budgets. 🔑 Implications and Conclusion This research challenges the belief that ViTs significantly outperform ConvNets when trained with similar computational budgets. It demonstrates that NFNet models can achieve competitive results on ImageNet, matching the performance of ViTs. The study emphasizes the importance of compute and data availability in model performance. While ViTs have their merits, ConvNets like NFNet remain formidable contenders, especially when trained at a large scale. This work encourages a fair evaluation of different architectures, considering both performance and computational requirements. 🤖 Practical AI Solutions for Middle Managers Discover how AI can redefine your way of work with these steps: 1️⃣ Identify Automation Opportunities: Find customer interaction points that can benefit from AI. 2️⃣ Define KPIs: Ensure your AI endeavors have measurable impacts on business outcomes. 3️⃣ Select an AI Solution: Choose tools that align with your needs and offer customization. 4️⃣ Implement Gradually: Start with a pilot, gather data, and expand AI usage judiciously. For AI KPI management advice, connect with us at hello@itinai.com. Stay tuned on our Telegram or Twitter for continuous insights into leveraging AI. 🌟 Spotlight on a Practical AI Solution Consider the AI Sales Bot from itinai.com/aisalesbot. It automates customer engagement 24/7 and manages interactions across all customer journey stages. Discover how AI can redefine your sales processes and customer engagement at itinai.com. 🔗 Useful Links: - AI Lab in Telegram @aiscrumbot – free consultation - Is ConvNet Making a Comeback? Unraveling Their Performance on Web-Scale Datasets and Matching Vision Transformers - MarkTechPost - Twitter – @itinaicom