Content
summary Summary

Researchers have conducted a study to determine how prompt methods can be used to generate ideas and which prompt methods generate the greatest diversity of ideas.

The working paper by Lennart Meincke, Ethan Mollick and Christian Terwiesch from the Wharton School of the University of Pennsylvania focuses on idea generation with GPT-4.

The team investigated how different prompting methods can influence the diversity of ideas generated. Specifically, the goal was to develop new products for students that cost less than $50.

The researchers tested different prompting methods, including minimal prompts, prompts in which the AI model takes on different personalities, and prompts in which the AI model applies different creativity techniques from existing literature.

Ad
Ad
The eight top groups of prompts tested.| Image: Meincke et al.

The diversity of ideas was measured using cosine similarity, a measure of the similarity between two ideas, but without comparison to existing ideas. The researchers also measured the number of unique ideas and the rate at which the idea space was exhausted.

The team found that different prompting methods had different effects on the diversity of ideas generated. But "chain-of-thought" prompting, a long-established prompting method, was the one that nearly reached the level of one group of students in the tests and came out on top by a wide margin.

Image: Meincke et al.

This method was also the one that generated the most unique ideas. This suggests that CoT prompting can help to open up the idea space more effectively and generate a greater variety of possible solutions.

Many of the "advanced" prompts tested were inferior to a simple basic prompt. Only the step-by-step prompts showed a significant improvement in idea diversity.| Image: Meincke et al.

Getting to better AI ideas - step by step

CoT prompting asks the AI model to solve a task in multiple steps. You don't have to specify these steps; just asking it to proceed step by step can improve the outcome.

It's not exactly clear why this works - the basic premise is that the prompt causes the model to focus on higher-quality data from the training dataset, which is more analytical.

Recommendation

In their experiment, the researchers gave a task for each step: First, GPT-4 was to generate 100 ideas, from which the AI was then to filter out the strongest and most diverse ideas.

In the final step, GPT-4 was told to give the result a name and a product description. The researchers excluded about 15 percent of the runs from the statistics because the model had not completed the second step correctly.

Image: Meincke et al.Overall, the researchers conclude that AI can be a useful tool for improving the ideation process.

However, they emphasize that choosing the right prompting method is critical to maximizing the diversity of ideas generated. The overlap between prompts is relatively small, the team says, allowing for hybrid prompting, or the generation of smaller pools of ideas using different prompting methods.

Ad
Join our community
Join the DECODER community on Discord, Reddit or Twitter - we can't wait to meet you.
Ad
Join our community
Join the DECODER community on Discord, Reddit or Twitter - we can't wait to meet you.

The study's co-author, Ethan Mollick, has published a GPT for idea generation that follows the step-by-step principle, but it is not the prompt used in the study.

Another recent study showed that the length of reasoning steps in CoT prompts is directly related to the performance of language models in complex problem-solving tasks. This was true even when the longer prompt did not contain significant new information.

Support our independent, free-access reporting. Any contribution helps and secures our future. Support now:
Bank transfer
Summary
  • Researchers at the Wharton School studied how different prompting methods affected the diversity of ideas generated by GPT-4.
  • The Chain of Thought prompting method generated the most unique ideas and a greater variety of possible solutions that nearly matched the idea level of a group of students.
  • The study shows that AI can be a useful tool to improve the idea generation process, but that choosing the right prompting method is critical to maximizing the diversity of ideas generated.
Sources
Online journalist Matthias is the co-founder and publisher of THE DECODER. He believes that artificial intelligence will fundamentally change the relationship between humans and computers.
Join our community
Join the DECODER community on Discord, Reddit or Twitter - we can't wait to meet you.