AI benchmarks have a trust problem and Google wants to fix it

To prevent that, Google Deepmind says it's launching the first double-blind evaluation of a proprietary frontier AI model. External tests stay locked in a cryptographic "box," so a model can't later use those questions to optimize itself specifically for the test.

For the pilot, Google is testing a model from the Gemini Flash Lite line against confidential benchmarks.

The tradeoff the method aims to fix

Highly sensitive external evaluations used to require a compromise, Deepmind says. Either the evaluators handed over their test prompts, which let the model provider see the questions in advance. Or the provider handed over its model weights and risked its intellectual property. A recent example of this dilemma was the delayed evaluation for the ARC-AGI benchmark of Anthropic's Fable 5, since the AI company enforces a 30-day data retention policy for its strongest models.

The double-blind evaluation is meant to eliminate that tradeoff. Google uses Confidential Space from Google Cloud's confidential computing portfolio to do it. The setup cryptographically verifies that both the external test data and the model stay private to their respective owners. The evaluator never sees the Gemini weights, and Google never sees the test prompts.

Until now, zero-logging protocols and contractual safeguards kept external prompts confidential. Adding technical and cryptographic protection is a big step forward for secure model evaluation, the company says.

Why this matters most for sensitive areas

The cryptographic proof aims to prevent contamination and protect sensitive data. Deepmind says this matters most for highly sensitive evaluations, such as cybersecurity or tests run by government agencies. Independent organizations could rigorously test advanced models without giving up data sovereignty or security.

Google hopes the effort sets a new standard for model oversight and helps the industry build more reliable and widely trusted AI systems. Google lays out the details on methodology and results in a technical report.