AI agents can now complete 16 percent of freelance jobs at pro quality, up from 2.5 percent eight months ago

The key metric is the automation rate, meaning the share of projects where the AI's work is rated at least as good as a human's.

Top automation rate jumps from 2.5 to 16.1 percent

When the benchmark first launched, the best AI agent automated just 2.5 percent of projects. According to the latest results, Fable 5 now hits 16.1 percent, the highest score ever recorded. That's roughly double Opus 4.8's 8.3 percent. GPT-5.5 comes in at 6.3 percent. All three models beat every previously tested system. The prior leader, Opus 4.6 running on the Claude Cowork framework, sat at 4.17 percent.

The frontier has more than quadrupled in under eight months, according to the authors. One caveat about Fable 5's score: only 218 of 240 projects could be evaluated before the U.S. government restricted access to the model. Even in the worst case, where Fable 5 failed every missing project, its rate would still be 14.6 percent, higher than any other model.

Progress doesn't track neatly with release dates, though. On the full Scale Labs leaderboard, the newer Gemini 3 Pro lands near the bottom at just 1.25 percent, behind much older systems.

Some examples from the study also show where even top models still fall short. On a ring design task, Fable 5 is clearly better than earlier AIs but still looks unprofessional on closer inspection. On an architecture project, GPT-5.5 faked an appealing render using an image generator while its actual 3D model remained flawed.

Human evaluators still can't be replaced

The team tested whether expensive human evaluation could be replaced by AI judges. The answer was clear: AI judges rated the new models far too generously. For GPT-5.5, the AI evaluator's score was almost three times too high. For Opus 4.8, about two and a half times. The automated judge did get the ranking order right, but the actual numbers were way off.

The reason, according to CAIS: To fairly judge delivered work, you need to open the files in the right professional software, operate that software correctly, and form a judgment like a paying client would. That kind of hands-on software use is exactly what current AI agents are worst at. An AI judge runs into the same limits as the AI workers it's supposed to evaluate. GPT-5.5's faked rendering is a good example: catching the trick requires opening the 3D model and inspecting the actual geometry.

To let the models show their full ability, the team runs them in the same tools developers use day to day, like Claude Code and Codex CLI. These were extended with the ability to operate graphical programs directly. The work environment is a virtual Linux machine loaded with over 30 professional apps, including Blender, GIMP, and Audacity. Each project gets up to 24 hours of compute time. The setup also uses a critic loop: a second AI agent reviews the output as critically as a demanding client, and the first agent then revises its work.

AI still fails to hit professional quality on most projects. None of the three Fable 5 results shown in the blog post would pass as finished work. But the rise in automation rates within a single year is rapid, the authors say, and directly reflects how fast remote work automation is advancing.