Content
summary Summary

Researchers from Google DeepMind have developed a hybrid AI architecture that combines transformers with networks specialized in logical reasoning. The result is a language model that performs well at generalizing, even on complex tasks.

Ad

A team from Google DeepMind has developed a new AI architecture called TransNAR. It combines the strengths of two different AI approaches: the language processing capabilities of transformer language models and the robustness of specialized AI systems for algorithmic reasoning, known as Neural Algorithmic Reasoners (NARs).

The goal was to compensate for the weaknesses of the individual approaches. Transformer language models like GPT-4 are very good at processing and generating natural language. However, they often fail without external tools on tasks that require precise algorithmic calculations.

This is precisely where NARs excel. These AI systems based on Graph Neural Networks (GNNs) can robustly execute complex algorithms when the tasks are represented in graph form. However, they require rigid structuring of the input data and cannot be directly applied to unstructured problems in natural language.

Ad
Ad

The DeepMind researchers have now combined both approaches: A transformer language model processes the text of the task, while a NAR processes the corresponding graph representation. The two models exchange information via neural attention mechanisms - the NAR can thus also be understood as a kind of "internal tool": communication does not take place via an API, but within the embeddings.

Image: Google DeepMind

With the hybrid architecture, the language model can access the robust computations of the NAR and the system can execute algorithms based on a natural language description.

TransNAR outperforms pure transformers, sometimes by a large margin

The researchers tested their TransNAR model on the CLRS dataset. This dataset contains descriptions of 30 different algorithms from computer science, such as binary search or sorting methods.

The results showed that TransNAR performs significantly better than a pure transformer model, especially in out-of-distribution cases where the test data deviates from the training data, i.e., the system has to generalize. In several classes of algorithms, TransNAR achieved an absolute improvement of more than 20%.

Image: Google DeepMind

The team says the results show that by combining complementary AI approaches, the ability of language models to perform algorithmic reasoning can be significantly improved.

Ad
Ad
Join our community
Join the DECODER community on Discord, Reddit or Twitter - we can't wait to meet you.
Recommendation
Support our independent, free-access reporting. Any contribution helps and secures our future. Support now:
Bank transfer
Summary
  • Researchers at Google DeepMind have developed a hybrid AI architecture called TransNAR, which combines Transformer language models with specialized AI systems for algorithmic reasoning.
  • TransNAR compensates for the weaknesses of each approach: Transformer models are good at language processing but fail at precise computation, while Neural Algorithmic Reasoners (NARs) can robustly execute complex algorithms but require structured input data.
  • In tests on the CLRS dataset with computer science algorithms, TransNAR achieved significantly higher accuracy than pure Transformer models, especially on OOD test data, demonstrating the benefits of combining complementary AI approaches.
Sources
Max is managing editor at THE DECODER. As a trained philosopher, he deals with consciousness, AI, and the question of whether machines can really think or just pretend to.
Join our community
Join the DECODER community on Discord, Reddit or Twitter - we can't wait to meet you.