LLMs give ridiculous answers to a simple river crossing puzzle

Tested LLMs often gave nonsensical answers, sometimes suggesting far more river crossings than necessary. Users have shared different versions of the puzzle on X, with some absurd results.

In one case, even when told that the farmer didn't need to cross the river at all, GPT-4o proposed a complex solution with nine crossings. And it ignored important constraints, such as not leaving chickens alone with wolves, which would have been perfectly feasible since the farmer didn't need to cross the river.

While LLMs can sometimes solve the puzzle with different clues in the prompt, critics argue that this highlights their lack of consistent reasoning and common sense. Current research supports the view that LLMs struggle to solve even the simplest logical tasks reliably.

These findings add to ongoing debates about the limitations of current LLM-based AI systems when it comes to logical reasoning and real-world problem-solving. "The problem is that LLMs have no common sense, no understanding of the world, and no ability to plan (and reason)," says Yann LeCun, Meta's director of AI research.

The question is whether research can find methods to improve the ability of LLMs to reason, or whether entirely new architectures are needed to fundamentally advance AI.