Content
summary Summary

The consulting industry is about to be disrupted by large language models such as GPT-4, according to a study.

Ad

The study was conducted by the Boston Consulting Group together with researchers from Harvard Business School, MIT Sloan, Warwick Business School, and the Wharton School.

It analyzed the work of 758 randomly selected Boston Consulting Group consultants. Some were allowed to use GPT-4, while others worked without AI. The consultants using AI had access to the generally available GPT-4 via an API, without any special prompting or fine-tuning.

The great skill leveler

The team compared 18 typical real-world consulting tasks for a fictitious shoe company: writing press releases, conducting market analysis, developing creative ideas for new products, writing inspirational speeches, and so on.

Ad
Ad

The results, evaluated by humans and GPT-4, were identical. On average, consultants working with GPT-4 completed 12.2 percent more tasks, acted 25.1 percent faster, and achieved 40 percent better results than their non-AI counterparts.

"Consultants using ChatGPT-4 outperformed those who did not, by a lot. On every dimension," writes Ethan Mollick of the Wharton School, who participated in the study.

Distribution of output quality across all tasks. The blue group used no AI, the green and red groups used AI, with the red group receiving additional training on the use of AI. | Image: Dell'Acqua et al.

The study also found that advisors who underperformed without AI particularly benefited from using AI. With GPT-4's help, they achieved a 43 percent increase in performance, while high-performing advisors improved by 17 percent. This ability to level the playing field is still underappreciated, Mollick writes.

Image: Dell'Acqua et al.

In addition, the research team identified two usage patterns: advisors who outsourced individual tasks to AI ("centaurs") and advisors who fully integrated AI into their workflow ("cyborgs"). Both benefited from the use of AI.

The Jagged Frontier of AI

But the study also shows that while generative AI excels at many tasks, it fails at certain problems. The researchers call this problem the "jagged frontier" of AI capabilities.

Recommendation

On tasks outside this range, advisors with AI performed nearly 25 percent worse than advisors without AI because GPT-4 provided unreliable or incorrect information. As a result, the researchers caution against blindly using AI.

"On some tasks AI is immensely powerful, and on others it fails completely or subtly. And, unless you use AI a lot, you won’t know which is which," Mollick writes.

Image: Dell'Acqua et al.

Overall, however, Mollick says most advisors have been able to confidently navigate the frontier and leverage the positive aspects of AI in their work without being affected by the negative effects.

Mollick expects AI's current frontiers to continue to expand, and its capabilities to evolve and improve in the future. The researcher believes that at least two companies will release more powerful models than GPT-4 in the next year.

Ad
Ad
Join our community
Join the DECODER community on Discord, Reddit or Twitter - we can't wait to meet you.
Support our independent, free-access reporting. Any contribution helps and secures our future. Support now:
Bank transfer
Summary
  • A study conducted by The Boston Consulting Group and academic researchers found that consultants using GPT-4 AI completed tasks 40% better and 25.1% faster than those without AI. Consultants who performed below average benefited the most.
  • The study highlighted two usage patterns: advisors who outsourced tasks to AI ("centaurs") and those who integrated AI into their workflow ("cyborgs"), with both groups benefiting from AI.
  • However, the research team cautioned against blindly deploying AI, as it can fail on certain problems, resulting in nearly 25% worse performance on tasks outside its capabilities.
Online journalist Matthias is the co-founder and publisher of THE DECODER. He believes that artificial intelligence will fundamentally change the relationship between humans and computers.
Join our community
Join the DECODER community on Discord, Reddit or Twitter - we can't wait to meet you.