Content
summary Summary

Data labeling platform Surge AI compared Google and ChatGPT results for 500 search queries - and calls ChatGPT an "existential threat" to Google.

Ad

For the study, Surge obtained five basic information search queries from 100 workers. The search queries had to relate to knowledge before 2022, as ChatGPT training only covers knowledge up to then.

ChatGPT is slightly ahead of Google

Employees resubmitted their query to Google and as a prompt to ChatGPT. They then rated the results and responses: ChatGPT came out ahead in 42 percent of the cases, and Google in 40 percent - even though ChatGPT is not optimized for search.

ChatGPT's answers were rated slightly better than Google's search results for queries about general information. | Image: Surge AI

The qualitative evaluation shows that ChatGPT had more outliers up and down, results that were categorized as "amazing" but also as "pretty bad" and "horrible".

Ad
Ad
ChatGPT gets more effusive praise and strong criticism than Google. | Image: Surge AI

Especially for code queries, ChatGPT is said to be superior, since it directly plays out the concrete code instead of links to websites that contain the matching code - or not.

The Surge team clearly preferred ChatGPT's answers to code queries over Google Search. | Image: Surge AI

Edwin Chen of Surge, a former Google employee, sees the results of the study as very positive for ChatGPT, since its interface is not even optimized for search or connected to the Internet. With WebGPT, OpenAI has already introduced an AI technology that can look up information on the Internet.

"Google was supposed to be the killer AI company; for a while, it was. But with transformative language models racing forward, is its reign – and dominance as the world's smartest search engine – about to be usurped?" writes Chen.

Formatting and convenience vs. accuracy and multimedia

A look at the detailed analysis of individual search results and chat responses shows that Surge AI's employees particularly preferred the convenience and pre-written responses of ChatGPT over manual searches across multiple websites.

"While Google offered me ample amounts of information, which did eventually lead me to a helpful answer, ChatGPT simplified the results I was looking for," writes one study participant.

Recommendation

In ChatGPT's favor, according to the study, are its ability to consolidate information from multiple sources into one answer, its minimalist interface with no ads, and ChatGPT's ability to handle complex queries.

"ChatGPT is very useful in eliminating the need to visit multiple web pages to get a complete answer," assessed one study participant.

The negative ChatGPT examples are answers with incorrect information that the model nevertheless presents with the same confidence as correct information, and refusal to answer, e.g., on topics related to more current references.

"I like the format of the AI's response - namely the fact that it provides an example citation instantly - but it's too inaccurate to be valuable," writes one study participant who compared a query about a specific citation format.

Ad
Ad
Join our community
Join the DECODER community on Discord, Reddit or Twitter - we can't wait to meet you.

One study participant criticized ChatGPT for having problems with any topic that was not established knowledge. "Given the lack of correct answers, I was not impressed and wouldn't use it over a search engine in its current state," writes another participant.

In addition, ChatGPT answers still lack images, videos, and audio. Google has already solved the latter problem with its LaMDA multimodal dialog AI.

Chat AI as search replacement still has many open questions to answer

In the context of systems such as ChatGPT as a search substitute, many questions arise, the most important of which are the citability of ChatGPT, ultimately the question of responsibility for the content, the verification of the information and its timeliness.

As I wrote elsewhere, I think it is more likely that Google and other search engine providers will use systems like ChatGPT to enrich their regular search results with verified AI responses as a first step.

Google has already integrated a corresponding interface with no-click search, i.e., displaying information from a web page directly in Google search. In the future, the information displayed there could be written by an AI and backed up with sources that confirm the AI answer.

This would have the charm for Google and Co. that they could provide even more precise answers to suitable search queries. They could circumvent the weaknesses of the AI systems by, for example, deactivating AI responses to search queries with a current reference.

Ad
Ad

In this scenario, a search provider like Google could attract even more users and retain them for longer. The original providers of information, such as publishers, would lose out because they would receive fewer hits via search. Further consolidation of the online media industry could be the result.

Support our independent, free-access reporting. Any contribution helps and secures our future. Support now:
Bank transfer
Summary
  • The data labeling platform Surge AI compared the results of 500 search queries using Google and ChatGPT.
  • The result: ChatGPT has a slight advantage.
  • Probably the biggest advantage of ChatGPT is its user-friendliness, according to the study. The biggest disadvantage is incorrect information.
Sources
Online journalist Matthias is the co-founder and publisher of THE DECODER. He believes that artificial intelligence will fundamentally change the relationship between humans and computers.
Join our community
Join the DECODER community on Discord, Reddit or Twitter - we can't wait to meet you.