AI search agents often confirm what they already know instead of actually researching the web

Searching can actually hurt the answer

The second test is more telling. The researchers left the search interface in place but removed all answer-supporting documents from the search index. Every model tested then performed worse than it did without any tool access at all. MiniMax M2.5 dropped from 44.5 to 8.0 percent. Kimi-K2.6 fell from 25.5 to 2.3 percent. The search actively pulls agents away from correct gut-feeling answers as soon as no confirming hits show up.

An analysis of the search paths explains why. More than half of all queries come from the model's own reasoning rather than from previously found hits. Even when relevant evidence does appear in search results, the agents fold it into their reasoning less than a third of the time. The loop is model-led, not evidence-led.

A benchmark beyond the knowledge frontier

To measure real search behavior, the authors built LiveBrowseComp. The benchmark contains 335 human-written questions, each depending on at least one fact from the 90 days before creation and impossible to answer without that current information.

The underlying events come from constantly updated sources like film databases, game directories, security vulnerability registers, and earthquake catalogs. Globally prominent events are filtered out deliberately, leaving obscure but publicly verifiable facts that had little chance of seeping into model parameters during training.

Human testers need about the same amount of time for LiveBrowseComp as for BrowseComp and solve a similar number of tasks. The performance drop among models is therefore due to losing the memory shortcut, not because the questions are harder.

Leaderboard rankings fall apart

On LiveBrowseComp, all models in the closed-book test fall below two percent accuracy. With tools turned on, scores land about 25 to 40 points below the same models' BrowseComp results.

This shifts the rankings. GLM 5.1 leads clearly among open-source models on BrowseComp but falls to mid-pack on LiveBrowseComp. DeepSeek v3.2 sat at the bottom on BrowseComp, then climbed to the top on LiveBrowseComp, passing several models that previously outperformed it. This shows that a model's spot on a static leaderboard mostly shows how much it already knows, not how well it searches.

Agents need more steps when they can't rely on memory

On BrowseComp, agents solve many questions in very few steps, a sign of quick memory confirmation. On LiveBrowseComp, that pattern disappears. The step counts shift much higher, which suggests the agents are doing real research instead of recalling stored knowledge.

The authors argue that dynamic, time-sensitive benchmarks should become the standard for evaluating AI agents. They also want training signals that reward evidence-based research over the typical guess-and-verify approach.

Other studies have flagged similar problems. A benchmark from Peking University found that top models often produce the right answer when analyzing documents but cite the wrong source, what the researchers call "attribution hallucination." A tool called CiteAudit recently discovered that fabricated references have already made it into accepted papers at major AI conferences. The reason: commercial models don't reliably catch made-up citations.