Content
summary Summary

A recent study shows that ChatGPT advice is perceived as more balanced, comprehensive, empathetic, and helpful than advice column responses.

Ad

The study, conducted by researchers at the University of Melbourne and the University of Western Australia and published in Frontiers of Psychology, compared ChatGPT and human responses to 50 social dilemma questions randomly selected from ten popular advice columns.

People prefer AI when they don't know

For the study, the researchers used the paid version of ChatGPT with GPT-4, currently the most capable LLM on the market.

They showed 404 subjects a question along with the corresponding answer from a columnist and ChatGPT. The participants were asked to rate which answer was more balanced, comprehensive, empathetic, helpful, and overall better.

Ad
Ad

The researchers found that ChatGPT "significantly outperformed" the human advisors on each of the five randomly asked questions and in all categories queried, with preference rates ranging from about 70 to 85 percent in favor of the AI.

The study also showed that ChatGPT participants' answers were longer than those of the advice columnists. In a second study, the researchers shortened the ChatGPT participants' responses to about the same length as the advice columnists' responses. This second study confirmed the first, albeit at a slightly lower level, and showed that the advantage of ChatGPT was not solely due to more detailed responses.

Beide Umfragen zeigen, dass viele Personen die Beratung durch ChatGPT als ausgewogener, vollständiger, einfühlsamer, hilfreicher und insgesamt besser empfanden als die Beratung durch einen professionellen Berater.Von den 404 Teilnehmern in Umfrage 1 und den 401 Teilnehmern in Umfrage 2 stimmte eine Mehrheit für ChatGPT in allen Punkten.
Both surveys show that many people found ChatGPT advice to be more balanced, complete, empathetic, helpful, and overall better than advice from a professional counselor. Of the 404 participants in Survey 1 and the 401 participants in Survey 2, a majority voted in favor of ChatGPT on all counts. | Image: Howe et al.

People prefer people when asked

Despite the perceived quality of ChatGPT advice, the majority (77%) of study participants still preferred a human response to their social conflict questions. This preference for human responses is consistent with previous research.

However, participants could not reliably distinguish which responses were written by ChatGPT and which were written by humans. In other words, the preference for human responses is not directly related to the quality of the responses, but rather appears to be a social or cultural phenomenon.

The researchers suggest that future research should explore this phenomenon in more detail, for example by informing participants in advance which answers were written by the AI and which were written by humans. This could increase the willingness to seek advice from the AI.

Recommendation

Previously, a study by psychologists had shown that ChatGPT could describe the possible emotional states of people in scenarios on the Levels of Emotional Awareness (LEAS) scale in much more detail than humans.

Another study of AI empathy, published in April 2023, found that humans can perceive AI responses for medical diagnoses as more empathetic and of higher quality than those of physicians. However, the study did not look at the accuracy of the responses.

Ad
Ad
Join our community
Join the DECODER community on Discord, Reddit or Twitter - we can't wait to meet you.
Support our independent, free-access reporting. Any contribution helps and secures our future. Support now:
Bank transfer
Summary
  • A study shows that ChatGPT advice is perceived as more balanced, comprehensive, empathetic, and helpful than advice from advice columnists.
  • Researchers compared the responses of ChatGPT and human columnists to 50 questions about social dilemmas and found that ChatGPT scored higher in all categories, with preference rates of about 70 to 85 percent in favor of the AI.
  • Despite the positive evaluation of ChatGPT, 77 percent of study participants still preferred human responses to their questions.
Online journalist Matthias is the co-founder and publisher of THE DECODER. He believes that artificial intelligence will fundamentally change the relationship between humans and computers.
Join our community
Join the DECODER community on Discord, Reddit or Twitter - we can't wait to meet you.