As we enter 2025, the AI community remains divided on what really drives progress in artificial intelligence.
Gary Marcus kicked off the year with his characteristic skepticism. "OpenAI will continue to preview products months and perhaps years before they are solid and widely available at an accessible price," he writes, predicting modest returns from AI models and warning that the much-hyped AI "agents" won't live up to their promise in 2025 because they are "far from reliable, except possibly in very narrow use cases."
AI researcher François Chollet - who recently left Google after a decade to start his own company and whose ARC-AGI benchmark recently made waves after OpenAI's o3 model performed well in it - suggests that throwing more compute and data at the problem isn't the answer. In fact, he claims, anyone who truly understands intelligence could develop an AGI with just a million dollars in training costs.
Toby Pohlen, who left DeepMind to join Elon Musk's xAI as a founding member, pushed back against Chollet. According to Pohlen, DeepMind used to share Chollet's view that massive scale wasn't necessary - but the evidence kept pointing in the opposite direction. However, even xAI's massive cluster, planned to reach 200,000 Nvidia GPUs, hasn't yet delivered on Musk's promise of "world's most powerful AI by every metric" by the end of 2024.
Tim Dettmers from Allen AI recently mapped out three possible paths: traditional data center scaling (viable for about two more years), dynamic scaling (leading to either specialized or flexible models), and knowledge distillation (which might follow entirely different rules than other scaling approaches). But ultimately, Dettmers sees a "perfect storm" brewing - a combination of physical limitations and diminishing returns that could spell the end of the scaling era.
New frontiers, same debate
A new frontier in this debate opened up with test-time compute scaling, the approach behind OpenAI's o-models. Hugging Face researchers recently validated this strategy, which shifts massive computing power to the inference stage rather than pre-training. While promising, the high costs of running o3 on benchmarks like ARC suggest we're trading one scaling challenge for another.
This isn't just an academic debate. The soaring valuations of Big Tech and AI companies like OpenAI and Nvidia have been built on the promise that scaling equals progress. If the scaling skeptics are right, we're not just facing a technical dead end - we could be looking at a massive correction in tech valuations that would send shockwaves through financial markets.
