summary Summary

The BBC is just one of many companies and organizations that block chatbots like ChatGPT from accessing their content.


The UK news organization plans to work with tech companies and regulators to safely develop generative AI, with a focus on maintaining trust in the news industry. The BBC's three guiding principles include acting in the public interest, prioritizing talent and creativity, and being transparent about AI-generated content.

However, the BBC has blocked web crawlers from OpenAI and Common Crawl from accessing its websites to protect copyrighted material. It joins other news organizations such as CNN and The New York Times.

ChatGPT can't access nearly half of the top 100 websites

OpenAI re-launched ChatGPT Browsing with Bing in late September, but the chatbot can't browse 47% of the top 100 websites, including Facebook, Google, and Amazon, according to a recent study by


The study found three reasons for this: being blocked by the site's Robots.txt, internal limitations of OpenAI, and technical difficulties. In one case, ChatGPT didn't respect a site's Robots.txt block. To enable ChatGPT Browse with Bing, users must be ChatGPT Plus subscribers.

Since the beginning of August, more and more websites are blocking ChatGpt and other chatbots from using their content. | Image:

Since the relaunch, OpenAI's ChatGPT Browse feature only provides 100-word summaries of web content to avoid copyright issues, but these summaries can be unreliable and inaccurate. The AI tool sometimes misses the point or omits important aspects of the content, so you need to check the source to be certain. This makes ChatGPT Browsing less useful than traditional search engines, which return relevant snippets directly from the source without errors or inaccuracies creeping in through rephrasing.

Chatbots as search replacement seems less likely now compared to earlier in the year

Between copyright issues, hallucinations, and websites blocking chatbots, a near-future scenario in which chatbots replace traditional search seems a lot further away than it did earlier this year, when Microsoft CEO Satya Nadella openly stated that he wanted a larger piece of Google's search market share.

Nadella recently backtracked, telling a court that he had overstated Bing Chat's potential to take market share from Google. While that statement in Google's antitrust trial was strategic, Microsoft hasn't actually taken market share from Google, even though it offers GPT-4 for free. It costs $20 per month if you get it with OpenAI's ChatGPT.

While chatbots may be able to compete with search engines in the future, they still have shortcomings in terms of reliability and copyright compliance. Microsoft rushed to release its Bing chatbot despite criticism that it could spread false information. That criticism is now proving true, according to a study by AlgorithmWatch. Apparently, being faster than Google is more important to Microsoft than safety.


That said, Google's generative AI search may be able to combine the best of both worlds by cherry-picking use cases for generative AI, implementing them in traditional search, and thus slowly beginning a transformation to a more closed platform that keeps users in a chatbot loop. If Google succeeds, it could change the face of the current Internet ecosystem.

Join our community
Join the DECODER community on Discord, Reddit or Twitter - we can't wait to meet you.
Support our independent, free-access reporting. Any contribution helps and secures our future. Support now:
Bank transfer
  • The BBC is exploring the use of generative AI for journalism and personalized experiences while blocking OpenAI data scraping. They want to work with tech companies and regulators to develop generative AI while maintaining trust in the news industry.
  • OpenAI's ChatGPT Browsing feature, which provides summaries of web content, is unable to access 47% of the top 100 websites, including Facebook, Google, and Amazon.
  • The reliability and copyright compliance of chatbots as a replacement for traditional search engines is questionable. Chatbots such as Bing Chat and ChatGPT still have limitations and have been criticized for misinformation and copyright issues.
Online journalist Matthias is the co-founder and publisher of THE DECODER. He believes that artificial intelligence will fundamentally change the relationship between humans and computers.
Join our community
Join the DECODER community on Discord, Reddit or Twitter - we can't wait to meet you.