AIT vs Git worktrees · game-development benchmark
Same coding agent. Same game tasks. Two repository workflows.
We gave fresh GPT-5.6 Sol agents the same maintenance and feature tasks on the same browser game. One side worked through an AIT Task and isolated worktree; the other used a conventional Git linked worktree. We then asked: how many provider tokens does each workflow require to reach an independently accepted result?
- 5 concrete game tasks
- 20 admitted pairs per task
- 100 AIT + 100 Git sessions per campaign
- GPT-5.6 Sol · max reasoning
Measured results
Both complete campaigns found about one-third lower token use for AIT.
Scope: five game tasks with one model pin. These are separate campaigns with disclosed failures and replacements; effective acceptance includes the admitted replacements. Token use is not a dollar-cost estimate. The campaigns did not measure high-concurrency throughput.
The primary figure gives every workload equal weight. Raw provider totals are shown as a descriptive cross-check, not substituted for that workload-balanced result.
| Campaign | Workflow | Token saving | 95% CI | Raw provider totals | Elapsed saving | Evidence history |
|---|---|---|---|---|---|---|
| Released baseline | Sprint off | 34.95% | 27.85–39.77% | 46,300,272 AIT / 70,140,925 Git 33.99% lower | 21.04% | 201 executed / 200 effective |
| Natural-inspection replication | Sprint on | 36.28% | 28.26–41.83% | 45,432,262 AIT / 71,238,660 Git 36.23% lower | 15.22% | 203 executed / 200 effective |
…b004-gd-05-ait—not generated artwork or a separate mockup.
What the agents actually changed
An existing deterministic browser shooter.
The fixture is a dependency-free JavaScript game called Starline Defender. It already had movement, enemies, scoring, pause, replay, settings, keyboard controls, and a regression test surface. The benchmark is therefore repository maintenance and feature work—not a contest to generate a game from scratch.
The playable page is the exact public web output of one accepted AIT session. It scored 100/100 and lets you inspect the kind of artifact the evaluator tested. It is illustrative only: the aggregate result below is calculated from every admitted session, not from this example.
The five workloads
From a one-line regression to a beta hardening pass.
Every session received one of these frozen task specifications and the same acceptance contract. The tasks deliberately increase in scope.
Stop rapid-fire projectiles
Repair the firing regression so the existing 180 ms firing cooldown is enforced without breaking normal play.
Add a two-hit armored enemy
Make it visually distinct, require two hits to destroy it, and award exactly 250 points.
Fix pause/resume replay divergence
Freeze paused time and input so a recorded replay converges to the same final state after resuming.
Split the monolithic game safely
Extract model, renderer, and input modules while preserving the game’s behavior and public API.
Prepare a release-ready artifact
Add versioned replay, settings normalization, mobile input, focus recovery, a benchmark panel, release checks, and documentation—without dependencies or external network access.
What one paired attempt means
The same task was run twice; only the repository workflow changed.
- Start from identical bytes.The AIT and Git lanes received the same frozen fixture, task text, model revision, reasoning effort, and validation commands.
- Open fresh agent sessions.Each lane used a new GPT-5.6 Sol max session in its own isolated workspace; no conversation history carried across attempts.
- Apply one workflow treatment.The AIT lane used AIT’s Task lifecycle. The Git lane used a temporary branch and linked worktree.
- Let the agent work naturally.The agent could inspect files, edit, test, and use meaningful status or diff inspection on either side. Those actions were neither required nor banned.
- Evaluate the result independently.The harness ran frozen functional checks and scored the resulting game. Workflow completion alone could not make a broken game pass.
- Count the complete provider usage.The admitted session’s machine-readable provider token total and elapsed time entered the paired analysis.
AIT treatment
Task + isolated worktree
The agent started an AIT Task, worked in its bound worktree, could create an intermediate Snapshot, and finished through the Task closeout. The baseline ran with sprint mode off; the replication bound an exact sprint card.
Git treatment
Linked worktree + temporary branch
The agent worked in a Git linked worktree on a temporary branch, committed the result, integrated it into the clean target branch, and removed the temporary worktree and branch.
How 200 sessions are constructed
Enough repeated pairs to estimate variation, not just compare two anecdotes.
5 workloads×20 pairs×2 workflows=200 sessions
Each workload contributes 20 admitted AIT/Git pairs: 100 AIT and 100 Git effective sessions per campaign. We ran this complete design twice and analyze the campaigns separately. The released baseline executed 201 raw sessions to form its 200-session effective view; the replication executed 203 raw sessions to form its own 200-session effective view. Every excluded or replacement execution is disclosed below.
What the numbers mean
Function first, provider cost second, uncertainty included.
Accepted outcome
A frozen evaluator checked startup, required behavior, determinism, score thresholds, workflow closeout, and the absence of external dependencies or network access. Both workflows reached 100/100 effective acceptance in each campaign.
Provider tokens
The cost measure is the provider’s machine-readable total-token usage for each admitted agent session. It is not inferred from command count or the size of the final patch.
Workload-balanced saving
We calculate AIT’s relative token saving inside each workload, then take the median across all five. This prevents the largest task from dominating the headline.
How to read the 95% confidence interval
The analysis resamples the 20 observed pairs within each workload 2,000 times, recalculates the workload-balanced saving, and reports the middle 95% of those bootstrap results. An interval crossing zero would leave the direction uncertain. Both campaign intervals—and every workload interval—remain above zero. The interval describes uncertainty for this measured test design; it does not prove the same saving for every repository or task.
Results by task
Every workload favored AIT in both campaigns.
Released 1.1.0 sprint-off baseline
| Workload | AIT effective tokens | Git effective tokens | Token saving | 95% CI | Pairs |
|---|---|---|---|---|---|
| GD-01 | 213,481.0 | 317,517.8 | 32.77% | 22.07–41.92% | 20/20 |
| GD-02 | 266,652.6 | 428,522.2 | 37.77% | 27.59–45.66% | 20/20 |
| GD-03 | 376,821.3 | 496,364.8 | 24.08% | 12.41–34.32% | 20/20 |
| GD-04 | 580,373.9 | 915,416.8 | 36.60% | 25.68–45.84% | 20/20 |
| GD-05 | 877,684.7 | 1,349,224.8 | 34.95% | 22.36–44.82% | 20/20 |
Sprint-on natural-inspection replication
| Workload | AIT effective tokens | Git effective tokens | Token saving | 95% CI | Pairs |
|---|---|---|---|---|---|
| GD-01 | 254,455.2 | 363,422.4 | 29.98% | 15.11–41.62% | 20/20 |
| GD-02 | 286,894.6 | 454,931.1 | 36.94% | 26.82–46.31% | 20/20 |
| GD-03 | 355,256.8 | 557,551.1 | 36.28% | 22.92–47.19% | 20/20 |
| GD-04 | 636,606.8 | 904,065.0 | 29.58% | 21.19–37.73% | 20/20 |
| GD-05 | 738,399.7 | 1,281,963.4 | 42.40% | 34.53–49.32% | 20/20 |
Every workload interval remained above zero in each campaign’s frozen analysis.
Execution history
Failures and interrupted runs remain visible.
Baseline: 201 sessions were executed for 200 effective sessions
The original AIT GD-05 attempt failed the frozen functional check because its change did not preserve soundEnabled: 0. That result remains in the ledger and was replaced exactly once under the disclosed same-pinned policy. Only the accepted replacement entered the effective view.
Replication: 203 sessions were executed for 200 effective sessions
Three source executions are excluded and disclosed: one GD-02 Git lane contaminated by an executor-infrastructure failure, one GD-05 Git lane interrupted by host shutdown before a terminal summary existed, and one valid GD-05 AIT functional failure admitted to one exact replacement. The infrastructure and shutdown cases used whole-pair recovery. A separate recovered-spawn classification was corrected through digest-linked adjudication without re-executing that session.
The Source-protocol flag remains false in the public record. Current policy revision game-development-2026-08-29.36 separately records that its criteria were met and sets effective claim_eligible=true; the earlier flag is not erased.
Interpretation boundary
Strong evidence for this test set, not a universal law.
What both campaigns support
For these five fixtures and this model pin, each separately analyzed campaign found about one-third lower provider-token use for AIT while preserving the same effective functional acceptance.
What this comparison does not establish
The campaigns differ in workflow mode, prompts, AIT binary, seed, date, and recovery history. They are not pooled and are not a causal sprint-on/off A/B test. The result is not a promise for every codebase, model, or task, and the linear sessions did not measure high-concurrency throughput.
Audit the evidence
Each campaign has its own immutable summary, JSON, ledger, and checksums.
Released 1.1.0 baseline
summary.txtHuman-readable baseline result, method, and disclosure.
result.jsonMachine-readable baseline aggregate and workload results.
runs.jsonBaseline session-level admission, timing, and token ledger.
SHA256SUMSIntegrity hashes for the released baseline set.
Sprint-on natural-inspection replication
Conclusion
On these five game-development tasks, AIT used about one-third fewer provider tokens than Git worktrees—and reached the same accepted outcomes.
The separately analyzed campaigns measured 34.95% saving (95% CI 27.85–39.77%) and 36.28% saving (95% CI 28.26–41.83%). Every workload favored AIT, both workflows reached 100/100 effective acceptance, and both intervals stayed above zero.