Google Deepmind researchers have introduced a new AI architecture called PEER that uses more than a million small "experts". This could significantly improve the efficiency and scalability of language models.
Scientists from Google Deepmind have developed a new method for constructing AI models that they call "Parameter Efficient Expert Retrieval" (PEER). This technique uses more than a million tiny "experts" - small neural networks with only one neuron - instead of the large feedforward layers used in conventional transformer models.
The researchers explain that PEER is based on the principle of "Mixture of Experts" (MoE). MoE is a technique where an AI system consists of many specialized sub-networks that are activated depending on the task - and the architecture that most likely powers current large language models like GPT-4, Gemini, or Claude. However, PEER goes a step further by using an extremely large number of very small experts.
To efficiently access this large number of experts, PEER uses a technique called "Product Key Memory". This allows quickly selecting the most relevant experts from millions without having to check them all individually.
In language modeling experiments, PEER outperformed both conventional transformer models and previous MoE approaches in efficiency. With the same computing power, PEER performed better in various benchmarks.
PEER shows that scaling laws apply to experts
The researchers explain the success of PEER with so-called scaling laws. These describe mathematically how the performance of AI models increases with their size and the amount of training data. The scientists argue that a very large number of small experts makes it possible to increase the overall capacity of the model without the computational cost increasing sharply.
The researchers see another advantage of PEER in the possibility of "lifelong learning". Since new experts can be added easily, a PEER model could theoretically constantly absorb new information without forgetting what it has already learned.
Overall, the researchers see PEER as a promising approach to making AI models more efficient and scalable. However, they point out that further research is needed to fully exploit the potential of this technology.
