Agent-scale code protocol revised after adversarial review, tested on Rust history
The lab spent the day on a design for a code protocol: rules for how changes are proposed and accepted when very large numbers of AI coding agents work on one codebase at the same time. The design was reviewed, revised and then tested against real project history. The work ran in Claude Code.
The review ran as a structured adversarial review, with a fixed objective, separate lenses, blind reviewers and an adjudicator from a different model lineage (gpt-6.1-sol). That adjudicator ruled on 89 findings: 8 Kills, 65 Degrades and 16 Cosmetic, with 4 rejected. The four confirmed Kills were folded into the design. Identity is now based on a hash of the interface, and the design adds staged admission, bonded challenges and a lease for an aged candidate. The lesson was written down plainly: "Open-ended review by a language model produces restatements and cosmetic nits. Contested review against a fixed objective, with roles, a finding format, and a settlement rule, produces holes you can act on."
A simulator then replayed the history of three Rust projects: tokio, meilisearch and arrow-rs. The share of changes that conflicted within one day was 18% for arrow-rs, 22% for meilisearch and 8% for tokio. At a rate of 1e6 per second in the replay, the hottest definition in tokio demands 280 times the per-definition cap. One variant let agents claim work when they start on it. That moved the point where half of all changes conflict by about 3x on the libraries and about 10x on meilisearch, the one service among the three, and showed that "Claims stop helping once there is no unclaimed work to resample into." The next three planned simulator phases were deferred in favour of two cheap variants, which were run. The simulator code and results are on branches of the package repository.
The last step uses real agents. A first calibration from 42 real build workspaces was committed. Then headless Claude agents started on 120 tokio issues. Under the first, permissive prompt, eight of ten agents declined their tasks within one or two turns, so the prompt was tightened. Three separate build caches filled the disk at about 9 GB and were replaced by one shared cache. That run is live, with three workers, and resumed from 32 finished tasks.
Still open: the run's results, a queue that holds agents until the code they need to change is free, and hot-definition demand that remains far above the per-definition cap.
- Research on LLM compute forecasts and whether the proposal holds over 10 to 20 years
- Comparison of the design with git and with closed competitors
- Simulator replay of tokio, meilisearch and arrow-rs history
- Real headless agents on 120 tokio issues
- Review the design as a structured adversarial review with a fixed objective, lenses, blind reviewers and a different-lineage adjudicator
- Fold the four confirmed Kills into the design
- Defer simulator phases 3 to 5 and run two cheap phase 2 variants instead
- Tighten the agent prompt after eight of ten agents declined under the permissive wording
- Real-agent run results and analysis still to be written up
- A work-assignment queue that holds agents until the code they need to change is free is not yet in the design
- Simulator phases 3 to 5 deferred
- Hot-definition demand still far above the per-definition cap in the replay data