ChatGPT lost badly to Atari's 1979 Video Chess engine. It gave solid advice and explained tactics, but it forgot captured pieces, confused rooks and bishops, and lost board awareness turn after turn. Atari's 1.19 MHz engine had no such issues. It just remembered the state and followed rules.
Some critics say Caruso's experiment compares apples and oranges, but it underscores a core weakness of LLMs: ChatGPT didn't lose because it lacked knowledge. It lost because it couldn't remember. Symbolic systems don't forget the board.
"Regardless of whether we're comparing specialized or general AI, its inability to retain a basic board state from turn to turn was very disappointing. Is that really any different from forgetting other crucial context in a conversation?"
Robert Jr. Caruso
