Contextual SDG Research Identification: An AI Evaluation Agent Methodology

**Universities and Global Competition** Universities are competing globally, and their rankings are increasingly tied to the United Nations’ Sustainable Development Goals (SDGs). These rankings impact their funding, reputation, and ability to attract students. **Challenges with Current Research Tracking** Right now, tracking research related to SDGs mainly uses traditional keyword searches in academic databases. This approach often misidentifies papers that don't genuinely contribute to SDG goals. **Limitations of Traditional Methods** While there have been efforts to improve the identification of relevant research, it remains difficult to distinguish between meaningful contributions and superficial mentions. **Innovative AI Solutions** Researchers at Virginia Tech Libraries have developed a new AI method to identify SDG research. This approach uses a specialized Large Language Model (LLM) to analyze research abstracts and pinpoint real contributions to SDGs. **Structured Evaluation Process** The AI assesses research based on clear guidelines, focusing on measurable actions that align with SDG objectives. It employs data science and text analytics for a better understanding of context and language. **Research Methodology** The team analyzed 20,000 journal articles and conference abstracts related to each of the 17 SDGs, using Scopus as their data source. They recognized the interconnectedness of the goals through search queries from Elsevier’s SDG Research Mapping Initiative. **Implementation of LLMs** The evaluation uses three compact LLMs: Microsoft’s Phi-3.5-mini-instruct, Mistral-7B-Instruct-v0.3, and Meta’s Llama-3.2-3B-Instruct. These models are efficient, easy to host, and accurately classify research abstracts. **Results and Insights** The study found that different LLMs interpret relevance differently. For example, Phi-3.5-mini identified 52% of abstracts as ‘Relevant’, while Llama-3.2 was stricter, marking only 15% as ‘Relevant’. **Conclusion and Future Directions** This research shows that small LLMs can effectively classify research related to SDGs. However, there are still challenges, such as the sensitivity of prompt design and a focus on SDG 1. **Transform Your Organization with AI** Leverage AI to improve your operations: - **Identify Automation Opportunities:** Discover where AI can be integrated. - **Define KPIs:** Set clear goals for AI initiatives. - **Select an AI Solution:** Choose tools that meet your needs. - **Implement Gradually:** Start small, collect data, and scale up. **Contact Us** For AI management advice, reach out at hello@itinai.com. Stay informed about AI insights through our channels.