Key Findings:

This factsheet evaluates the effectiveness of two widely used generative AI chatbots, ChatGPT and Bard (now known as Gemini), in delivering the latest news headlines from specific outlets. We tested both chatbots by requesting the top five headlines from the most popular online news sources across ten countries and analyzing their responses to provide descriptive statistics. The more in-depth analysis primarily focuses on ChatGPT’s outputs from seven of the ten countries studied.

Based on 4,500 headline requests (across 900 outputs) collected from ChatGPT and Bard, covering ten countries, the findings include:

• When asked for the latest top news headlines from specific outlets, ChatGPT provided a non-news output 52–54% of the time, often with an “I’m unable to”-style message. Bard did this 95% of the time.
• Only 8–10% of ChatGPT’s responses included headlines that actually reflected the current top stories on the requested outlet’s homepage. This suggests that even when ChatGPT does deliver news-like content, it rarely aligns with the latest top news.
• About one-third (30%) of responses referred to real stories from the outlet in question, but these were often outdated or not among the latest top stories.
• Approximately 3% of ChatGPT’s outputs included headlines that, while real, were misattributed to the wrong outlet—an issue that can be considered a form of AI “hallucination.” Another 3% were too vague or ambiguous to be matched to any existing stories, also considered a form of hallucination.
• The accuracy of ChatGPT’s outputs is significantly affected by whether news websites block it, and the same prompts can yield different results over time for reasons unclear to users.
• A majority (82%) of news-like outputs from ChatGPT included a referral link to the outlet in question, but most of these links (72%) directed users to the homepage rather than a specific story (10%).

Background:

Large language models (LLMs) like ChatGPT and Bard typically cannot serve as reliable news sources because they are trained on outdated web data. However, some generative AI chatbots, such as ChatGPT (Enterprise) and Google Bard, are connected to the web and can theoretically retrieve real-time information, including news from established outlets.

Despite this potential, few people currently use AI chatbots for news. A December 2023 survey in the UK found that only 2% of the online population has used generative AI to get the latest news. This low usage may be partly because most generative AI tools, including ChatGPT, are only web-connected for paid subscribers, and Google Bard was still in the experimental phase during most of this study.

As generative AI tools become more widely connected to the web, the ability to reliably retrieve and present up-to-date information will become increasingly important.

Previous Research:

While there is a growing body of research on generative AI’s ability to complete specific tasks, few studies have examined its response to users’ timely questions. Some research has explored AI’s answers to questions about upcoming elections, often finding inaccuracies and potential for misinformation. This factsheet builds on previous accounts and provides a systematic, descriptive analysis of what happens when generative AI chatbots are asked for the latest news.

Dive deeper into our comprehensive analysis of how AI chatbots like ChatGPT and Bard respond to news requests. Understand their capabilities, limitations, and the implications for reliable news delivery. Discover insights from our study across ten countries, explore the findings, and learn what the future holds for AI-driven news. Download the full report now to stay informed and ahead in the evolving digital news landscape.

• Key findings
• Background
• Previous research
• Method
• Results
• Conclusion
• References
• Funding acknowledgement
• About the authors

Number of Pages:

• 11 pages

Pricing: 

• Free