Many large websites are blocking Apple's AI from accessing their content, forcing Apple to negotiate license agreements. This contrasts with Google, which can use its market power to pressure publishers into allowing AI access.
A WIRED report reveals that Facebook, Instagram, Craigslist, Tumblr, the New York Times, Financial Times, The Atlantic, Vox Media, USA Today Network, and Condé Nast are among those blocking Apple's AI training crawlers. Some of these publishers have already made deals with OpenAI.
Apple recently introduced Applebot-Extended, a new crawler that website operators can block via robots.txt files. About 7% of 1,000 analyzed websites currently block this AI crawler, according to Wired. A separate analysis by data journalist Ben Welsh found 294 out of 1,167 mainly US-based English-language publications blocking Applebot-Extended.
Anthropic, OpenAI, and Google face similar challenges, with OpenAI experiencing even higher block rates for its crawler.
Google has the most leverage
Because Apple and OpenAI need high-quality training data and breaking news for their AI products, they depend on licensing deals with major websites and publishers. OpenAI has already secured some deals, while Apple is reportedly in talks but has yet to announce any agreements.
But Google does Google things. It's using its search engine dominance to pressure publishers: allow AI access or risk lower search visibility. Google pulls this off by mixing its AI overviews with traditional web crawling.
The company has also reportedly ended earlier discussions with publishers about potential license agreements for using their content in AI answers. Given Google's daily search volume, the cost of such agreements is likely to be high, further squeezing search margins already under pressure from AI.
While Apple also has a crawler for its search products, publishers can block the AI crawler without affecting their visibility in Apple's search results. This approach is fairer, but search isn't Apple's core business. It remains to be seen how OpenAI's ChatGPT publisher licenses will work when accessed through Apple products.
