WikiSkill: How Agents Evolve SKILL.md from Experience

WikiSkill is a skill-evolution architecture that turns agent execution traces into persistent evidence, then uses that evidence to propose and validate edits to reusable skills. In the paper's five-benchmark average, Qwen-3.5-9B with WikiSkill scored 47.4%, while the larger Qwen-3.6-27B without skills scored 39.4%. No fine-tuning was used.

Research date: 19 September 2026. The source is an arXiv preprint from researchers affiliated with Google Research and Virginia Tech, not a peer-reviewed production study. Its result is important because it treats experience as an external, testable software asset rather than assuming model scale is the only upgrade path. WikiSkill paper and author record

This page owns one narrow intent: what the WikiSkill paper demonstrates and how to adapt its persistent-wiki architecture for SKILL.md evolution. For the general difference between Agent Skills, MCP, RAG and Custom GPTs, use our Agent Skills comparison. For cross-model manuals without an optimizer wiki, read the agent knowledge-transfer guide. A company knowledge base is a different problem, covered in our AI-ready company wiki guide.

What is WikiSkill?

WikiSkill is an outer-loop optimizer for procedural agent skills. The task agent runs examples using the current skills. A Wiki Maintainer converts successful and failed traces into durable patterns. A Skill Proposer reads those patterns and proposes one skill change. A validation gate keeps the change only when held-out performance improves.

The key design choice is separation. Raw traces remain evidence. The wiki becomes an accumulated diagnosis of what works, what fails and which edits were rejected. The active skill stays concise enough to guide execution. When a proposed edit fails, the skill rolls back, but the wiki retains the failed experiment so the optimizer does not have to rediscover it.

That makes WikiSkill different from simply asking an LLM to rewrite a prompt after each mistake. The system has an audit trail, a distinct validation split and two memories with different lifecycles: reversible instructions and persistent learning history.

How does WikiSkill's three-layer architecture work?

The paper structures the workspace into raw/, wiki/ and skills/. The inference agent can use active skills while solving tasks, but it does not read the wiki during those rollouts. The maintainer and proposer can inspect the evidence and wiki. This restriction matters because direct wiki access by the task agent reduced final skill quality in the authors' ablation.

Each iteration follows a controlled sequence: run training tasks, sample successes and failures, update the wiki, propose one atomic skill creation or patch, run the candidate on validation tasks, then accept only a strict improvement. The paper used up to eight evolution iterations and reports results as averages across three independent full runs. Full WikiSkill method, tables and limitations

The word “wiki” can be misleading. This is not a user-facing RAG corpus and not a place to dump every document. It is optimizer memory: compact root-cause patterns, supporting traces, proposal diffs and acceptance outcomes that help the next skill edit start from accumulated evidence.

What did WikiSkill score across the five benchmarks?

WikiSkill produced the highest average score for every tested model and beat the strongest competing skill-evolution method by 3.3 to 12.0 percentage points. The suite covered recent mathematics, web search, spreadsheet manipulation, long-context document QA and interactive embodied tasks.

The gains were not uniform. Qwen-3.6-27B improved by 40.9 points on SpreadsheetBench but only 11.6 on OfficeQA. Qwen-3.5-4B slightly degraded on OfficeQA because it struggled to execute the long-context search workflow encoded by the evolved skill. The useful claim is therefore not “skills always help.” It is that validated procedural knowledge improved most tested model-benchmark pairs and that the ability to execute a sophisticated skill still depends on the model and task.

The comparison was reasonably controlled for a research paper: all methods started with an empty skill set, the evolved skill was inserted into the inference prompt, and significance was tested with paired bootstrap resampling. It still remains a benchmark study, not evidence of lower production cost, latency or error rates for an arbitrary agent.

Did a 9B model really beat a 27B model?

Yes on the reported five-benchmark average, but the headline needs its qualifiers. Qwen-3.5-9B with WikiSkill averaged 47.4%, compared with 39.4% for Qwen-3.6-27B without skills. Both are Qwen models, but they are different releases as well as different sizes. The experiment does not prove that a 9B model with any skill beats every 27B model on every workload.

What it does show is operationally more useful: model capability and procedural context are separate levers. A smaller model that receives tested instructions can outperform a larger model that must rediscover the procedure on each run. At the same time, scaling and skill evolution were complementary. The 27B model gained the most from WikiSkill, rising by 23.9 points to 63.3.

Before paying for a larger default model, freeze an evaluation set and compare four lanes: current model without the skill, current model with a hand-written skill, current model with an evolved skill, and the larger model without it. Measure accepted outcomes, not just token price. The cheapest inference call is irrelevant if its failures trigger retries or manual repair.

Do evolved skills transfer between models?

Often, yes, and sometimes a skill evolved by another model beat the receiving model's own skill. On SpreadsheetBench, Qwen-3.5-9B scored 24.3 with no skill, 33.6 with its self-evolved skill and 50.5 with a skill evolved by Qwen-3.6-27B. On ALFWorld, the same receiving model moved from 63.4 with its own skill to 70.2 with the 27B skill.

The negative result matters. A portable skill must encode general procedure, not a workaround for one model's habits. Treat model transfer as a fresh release: rerun the target model's evaluation suite, compare against no skill and self-evolved skill, and reject the import when it degrades a slice even if its overall wording looks sensible.

This complements the frontier-to-cheaper model transfer pattern. WikiSkill adds a persistent optimizer history and validation loop; it does not merely hand one model's notes to another.

Why does the persistent wiki drive the gain?

The strongest ablation isolated the wiki's contribution. With Gemini-3.5-Flash and no wiki for the Skill Proposer, the four-benchmark average was 48.7%. Giving the proposer access to the persistent wiki raised it to 63.7%, a 15.0-point gain. Giving the Inference Agent direct wiki access as well reduced the score to 60.9%.

That result supports a clean architectural boundary:

• Execution receives the released skill. The task agent should prove the skill itself is sufficient.
• Optimization receives the evidence history. The proposer needs recurring failures, successful strategies, rejected diffs and validation outcomes.
• Release remains gated. A plausible edit does not become production instruction until it improves a held-out measure.

The wiki is useful because optimizer history otherwise disappears into conversations and temporary trace batches. It turns “we tried that” into inspectable state. It also reduces repeated bad proposals: the paper's case study records a rejected loop-breaking edit, then uses the retained evidence to create an accepted rule in the next iteration and refine it later.

What does the WikiSkill paper not prove?

The paper is strong enough to justify experiments, not blind deployment. Its own limitations define several important boundaries.

• No skill retrieval test. Active skills were inserted directly into the system prompt. The study isolates skill quality but does not show how to select the right skill from a large library.
• Strict improvement can block stepping stones. Every accepted patch had to beat the best validation score immediately. A neutral intermediate refactor that enables a later gain would be rejected.
• The wiki only grows. There is no automatic pruning, contradiction retirement or evidence aging mechanism.
• No very long-horizon evaluation. The suite includes multi-step tools and long documents, but not workflows lasting hundreds of actions or hours.
• Small validation splits can be noisy. The benchmark validation sets range from 10 to 40 examples, so a production gate needs more representative slices and repeated runs.
• No complete cost accounting. The optimizer adds task rollouts, a maintainer call, multi-turn proposal work and validation runs. Better task scores do not automatically mean a cheaper system.

The study also keeps the broader harness fixed. Tool contracts, model routing, permissions, memory, context assembly and runtime observability remain separate engineering concerns. Use an agent harness to control those boundaries and an independent verifier pattern when one scalar benchmark score cannot capture correctness.

How should a team pilot WikiSkill-style evolution?

Start with one repeated task whose success can be scored independently. The Agent Skills specification makes SKILL.md the required instruction file inside a skill directory, with frontmatter for its name and description and optional scripts or resources. That filesystem contract is a practical release unit for an evolved skill. Agent Skills specification

Do not optimize directly against a few attractive examples. The Agent Skills evaluation guide recommends trigger tests and output-quality evaluations with expected outcomes. For WikiSkill-style evolution, extend that discipline with regression slices, adversarial tasks and a frozen final test set that the proposer never sees. Agent Skills evaluation guidance

Security is part of the optimizer, not a postscript. A poisoned retrieved document, compromised tool output or malicious training task can be compiled into durable instructions. SkillJack demonstrates that experience-derived skills can preserve a backdoor after the original poisoned source is gone. Keep untrusted content labeled, require provenance for every pattern, sandbox candidate runs, review changes to permissions or network behavior, and never let the same model both propose and authorize a consequential capability expansion. SkillJack preprint on persistent skill poisoning

Our agent evaluation and sandbox checklist covers the surrounding test controls. Wavect's AI agent engineering work can turn one repeated workflow into a trace, wiki, skill and release pipeline. Use the software QA checklist for release readiness, or bring us the workflow and its current failure traces.

Our verdict: evolve the procedure before replacing the model

WikiSkill's most valuable idea is not that an agent can edit Markdown. It is that procedural improvement needs its own evidence architecture. Traces preserve what happened. A persistent wiki compiles recurring causes and failed interventions. A concise skill carries only released procedure into execution. Validation decides whether the procedure advances.

The benchmark headline is real: a 9B Qwen model with evolved skills exceeded a 27B Qwen model without skills on the study's average. The deeper result is that larger models benefited even more, cross-model transfer sometimes worked, and the persistent wiki accounted for a large part of the gain. Skills are not a substitute for model quality. They are a second scaling axis.

For a production team, the next move is measurable: choose one repeated task, freeze the evaluation set, version the current SKILL.md, retain immutable traces, propose atomic patches and promote only proven improvements. Test that before making a larger model the default answer to every failure.