Google's WikiSkill gives AI agents a persistent memory of past mistakes to sharpen future performance

So the agent, aka the model, doesn't truly learn in a continuous sense, which remains an unsolved problem. But it writes better instructions for itself after each run and can pull them up the next time around. It's not elegant, and it's probably more error-prone than real learning, but the study shows it's an effective workaround.

The work draws on a perspective from Andrej Karpathy about the idea of an "LLM Wiki," which argues for compiling experience into persistent, cumulative knowledge. WikiSkill applies that thinking to automatic skill development for AI agents.

Three layers separate experience, knowledge, and action

WikiSkill organizes an AI agent's workspace into three tiers. The "Raw Layer" stores complete execution traces, from tool calls to results. This data is immutable and serves as raw material.

Above it sits the "Wiki Layer," where raw data gets distilled into structured insights like documented failure patterns and successful strategies. This knowledge layer never resets and only grows with each iteration, according to the researchers.

The top tier, the "Skill Layer," holds the active procedural instructions the agent follows when executing tasks. Unlike the wiki, skills can be rolled back if an update hurts performance.

First, an inference agent executes tasks using the current skills and generates execution traces. A "Wiki Maintainer" then analyzes those traces, spots failure patterns and successful strategies, and writes the findings into the wiki.

A "Skill Proposer" uses the updated wiki and execution data to suggest targeted skill changes. Finally, a gating mechanism tests the proposed change on a separate validation set to confirm it actually helps. If it doesn't, the skill gets rolled back, but the wiki stays intact. Even failed proposals aren't lost because the wiki documents what was tried and why it failed, so the Skill Proposer can build on that knowledge in later iterations.

Bigger models gain more, but smaller ones close the gap

The researchers tested WikiSkill across five benchmarks covering math reasoning, web search, spreadsheet manipulation, document question-answering, and interactive tasks in a virtual environment. The models they used were Qwen (4B, 9B, 27B), Gemma-4-31B, and Gemini-3.5-Flash.

WikiSkill consistently outperforms all previous skill evolution methods in the study. On average, the framework boosts Gemini-3.5-Flash from 49.5 percent to 68.1 percent and Qwen-3.6-27B from 39.4 percent to 63.3 percent. On individual benchmarks, the jumps can be bigger: Gemini-3.5-Flash climbs from 33.0 percent to 72.6 percent on LiveMath and from 50.5 percent to 76.6 percent on SpreadSheet.

The gains vary a lot by task type. Math problems and spreadsheet manipulation see the biggest improvements, while tasks with long document contexts (OfficeQA) show much smaller gains. The researchers say smaller models like Qwen-3.5-4B struggle to reliably execute evolved multi-step search strategies across long contexts and fall back to their default behavior.

Larger models tend to benefit more from evolved skills. But smaller models running WikiSkill can match the performance of larger models that don't use the framework. Skills developed by one model often transfer to another and sometimes even work better than skills the receiving model built on its own. Since that's not always the case, transferability should probably be checked case by case.

Table: WikiSkill (highlighted) achieves the best or statistically equivalent performance across most model-benchmark combinations. Bold values indicate top scores; multiple bold values in the same column are not statistically different from each other. All values are averages across three independent runs.