Meta is developing its own web search engine to make itself more independent of Google and Microsoft's Bing. The technology will primarily be used to feed the company's own AI chatbot with up-to-date information.
According to a report by The Information, citing a person familiar with the project, Meta is developing a search engine that crawls the web and provides the company's Meta AI chatbot with up-to-date information for conversations.
The project aims to make Meta less dependent on Google Search and Microsoft Bing. Both companies currently supply Meta AI with information about news, sports, and stock prices.
According to a person familiar with the strategy, Meta's own search engine could serve as a backup option in case Google or Microsoft decide to terminate their existing agreements. Meta recently also entered into an agreement with Reuters.
Report: Meta has been indexing the web for at least eight months
The search engine team, led by Senior Engineering Manager Xueyuan Su, has been working on web indexing for at least eight months. The focus is on capturing web pages and organizing their content into databases that Meta AI can use to answer relevant questions.
Meta's web crawler technology has been publicly known since July. The company states that it uses the crawler bot for "for use cases such as training AI models or improving products by indexing content directly."
In August, Meta CEO Mark Zuckerberg announced that Meta AI has over 185 million weekly active users. This number still lags OpenAI's ChatGPT, which recently reported over 250 million weekly active users.
Meta and Google declined to comment on the report to The Information. Microsoft did not respond to requests for comment.
Besides Meta, Google, and Microsoft itself, OpenAI and the startup Perplexity are also working on LLM-based search engines. However, the projects still suffer from high costs and sometimes dangerous errors or just plain bullshit in the results. Both limit the scalability of the systems. And we haven't even discussed the copyright issues. But still, investors seem willing to put hundreds of millions into Perplexity.
