Google is using its search engine dominance to gain an edge in the AI race. Publishers face a tough choice: include their content in AI answers or risk losing search visibility.
Bloomberg reports that Google is using its market power to create a dilemma for publishers. They must either allow their content to appear in Google's AI answers or risk a drop in search rankings.
Google uses its Googlebot crawler for both search and AI Overviews. While site owners can use a "nosnippet tag" to block portions of their content from becoming an AI answer, this also threatens their overall search visibility.
Publishers caught between a rock and a hard place
Many publishers rely on Google search traffic and can't afford to block Google's crawler entirely. Blocking just the AI crawler isn't a solution, as AI responses come from the main crawler. A Google spokesperson told Bloomberg that Googlebot governs AI results, as the AI and search engine are closely linked.
"It becomes like an existential crisis for these companies," Joe Ragazzo, editor of news site Talking Points Memo, told Bloomberg. "These are two bad options. You drop out and you die immediately, or you partner with them and you probably just die slowly, because eventually they’re not going to need you either."
Google told Engadget that blocking an entire article from AI Overviews wouldn't prevent the crawler from seeing "the full text of what's provided to us for ranking purposes" or "being indexed and appearing in our web search results." Publishers can also choose to block only parts of their sites from AI Overviews.
Following the Engadget and Bloomberg reports, I reached out to Google for comment. A Google spokesperson told me there were "some inaccuracies" in the reporting. However, the spokesperson did not respond to my request for clarification on these inaccuracies within a week. When I receive a more accurate explanation, I will update the article.
Google's AI strategy threatens traditional web content model
Google has reportedly ended talks with publishers about potential licensing deals for using their content in AI responses, according to two Bloomberg sources.
AI companies like OpenAI and Perplexity, and reportedly Apple, are planning to compensate publishers for the use of their content, either directly or through shared ad revenue. However, these companies lack the resources to transform the entire internet into a chatbot ecosystem where websites and their monetization become less relevant.
Such deals at scale could be prohibitively expensive even for Google, which would have to fund content creation indirectly through data deals because it couldn't return significant traffic and value, the tacit consent on which the search business has been built. This would put an end to Google's high margins on its current Internet search business model.
The negative impact of all these shifts on the web can be seen in one particular example. Google's $60 million deal with Reddit is designed to get maximum AI content for minimum cost. As a result, Reddit threads with links to an original source sometimes outranked original publisher content in search results - an unfair boost for Reddit at the expense of the broader web.
The Google deal also makes it harder for startups to buy Reddit's data at reasonable prices, and Reddit is now blocking other search engines' index crawlers, creating something historic: a search engine-exclusive site.
