Content
summary Summary

Methods that help people think can also boost language models. Representing step sequences as graphs can significantly improve performance.

Ad

Researchers from the University of Oxford, Alan Turing Institute, Allen Institute for AI, and LMU Munich explored how large language models handle complex planning tasks. They wanted to see if LLMs could optimize plans with both sequential and parallel actions under time and resource constraints. The team introduced a method called "Plan Like a Graph" (PLaG) in a recent paper.c

Method can be used for all language models

PLaG works with all language models tested, improving their performance. It's ready to use now and compatible with current models. The newest model tested was GPT-4, which is somewhat outdated by today's AI standards.

Image: Lin et al.

To evaluate PLaG, the researchers created a benchmark dataset called AsyncHow with over 1,600 realistic planning problems. It builds on the existing ProScript set and includes WikiHow instructions.

Ad
Ad
Image: Lin et al.

The team tested PLaG on various language models, including GPT-3.5, GPT-4, Cohere Command, and open-source options like LLaMA-2 and Mistral-7B. They noted that LLM performance drops sharply as tasks become more complex or involve more steps.

Image: Lin et al.

A variant called "Build a Graph" (BaG), where the LLM creates its own graph representation, performed even better than using a given graph. The researchers think this might be because the LLM can tailor the graph to its own reasoning process.

AGI still a long way off

Despite PLaG's improvements, the researchers say LLMs aren't yet robust enough for use as general intelligent agents in complex planning. There are still limits to using LLMs as standalone digital planners, at least for now.

The team sees several ways to expand this work, including adding more constraints to the benchmark and comparing how LLMs and humans perform on the same planning tasks.

Since PLaG's release, OpenAI has introduced its latest model, o1, which focuses on logical reasoning. It would be interesting to see if this technique also enhances its planning abilities.

Ad
Ad
Join our community
Join the DECODER community on Discord, Reddit or Twitter - we can't wait to meet you.
Recommendation
Support our independent, free-access reporting. Any contribution helps and secures our future. Support now:
Bank transfer
Summary
  • Researchers from institutions such as the University of Oxford and the Allen Institute for AI have investigated whether language models can solve complex planning tasks involving sequential and parallel actions under time and resource constraints.
  • The "Plan Like a Graph" (PLaG) method significantly increases the performance of all tested language models by representing the problem in the prompt with an additional graph. A variant in which the language model generates its own graph representation leads to even better results.
  • Despite the improvements made by PLaG, according to the researchers, language models are not yet robust enough to be used as general intelligent agents for complex planning. There are currently still limits to the use of language models as stand-alone digital planners.
Sources
Jonathan works as a technology journalist who focuses primarily on how easily AI can already be used today and how it can support daily life.
Join our community
Join the DECODER community on Discord, Reddit or Twitter - we can't wait to meet you.