Content
summary Summary

AI startup Inflection introduces Inflection 2.5, a new large language model that can catch up to GPT-4 with comparatively little effort.

Ad

AI startup Inflection, which specializes in developing AI assistants, recently unveiled its latest LLM: Inflection-2.5, which is designed to catch up with leading models like GPT-4 while being more efficient.

The new model is integrated into Inflection's "Pi", an AI assistant that has been "designed to be empathetic, helpful, and safe," according to the startup. It is now available to all Pi users through Inflection's website.

More efficient AI training

Inflection 2.5 is reported to achieve 94 percent of the average performance of GPT-4, while requiring only 40 percent of the estimated computational effort for training. Inflection particularly highlights advances in STEM (science, technology, engineering, and mathematics).

Ad
Ad
Image: Inflection AI

On the popular MMLU language comprehension benchmark, Inflection 2.5 is close to GPT-4, albeit with a more complex prompting scheme. The comparison to Inflection 1 is flawed because the 72.7 score on the MMLU benchmark was achieved with a simpler prompt (5-shot). Inflection 2 scored just under 80 percent on the MMLU using the industry standard 5-shot method.

Image: Inflection AI

In MT-Bench, which tests the conversational skills of AI models and how well they can follow a prompt, Inflection 2.5 scored between GPT-3.5 and GPT-4.

Curiously, during the evaluation, Inflection found that nearly 25 percent of the examples in the Reasoning, Mathematics, and Coding sections had incorrect reference solutions. The startup has corrected these and published a revised data set (MT-Bench Corrected). Yet another indication of the limited validity of synthetic benchmarks.

Image: Inflection AI

The model was also tested on the Hungarian math exam, which should be excluded from the training data, and on the Physics GRE, an entrance exam for graduate studies in physics. Inflection-2.5 scored at the 85th percentile of human test takers on the physics test, and came close to the maximum score using an extended prompting method, just behind GPT-4.

Image: Inflection AI

Inflection AI was founded by LinkedIn founder Reid Hoffman, Deepmind co-founder Mustafa Suleyman and former Deepmind researcher Karén Simonyan. The startup's goal is to create an interface that allows people to talk to computers and give them complex tasks without having to learn code.

Recommendation

The startup has received investments from well-known personalities and companies such as Nvidia and Microsoft, Reid Hoffman, Bill Gates and Eric Schmidt.

Ad
Ad
Join our community
Join the DECODER community on Discord, Reddit or Twitter - we can't wait to meet you.
Support our independent, free-access reporting. Any contribution helps and secures our future. Support now:
Bank transfer
Summary
  • AI startup Inflection has unveiled its new Inflection 2.5 language model, which aims to achieve GPT-4-like performance with less computation. It is available through the company's Pi AI assistant.
  • In the MMLU language comprehension benchmark, Inflection 2.5 achieves results similar to GPT-4 using a special prompting method; in the MT conversational performance benchmark, it performs between GPT-3.5 and GPT-4.
  • Inflection AI was founded by well-known tech leaders such as LinkedIn founder Reid Hoffman and Deepmind co-founder Mustafa Suleyman. The startup aims to replace code with natural language. Investors include Microsoft and Nvidia.
Sources
Online journalist Matthias is the co-founder and publisher of THE DECODER. He believes that artificial intelligence will fundamentally change the relationship between humans and computers.
Join our community
Join the DECODER community on Discord, Reddit or Twitter - we can't wait to meet you.