1.1.0 Benchmark
Read the frozen 100-session-per-treatment AIT versus Git worktree result, confidence interval, acceptance disclosure, and concurrency boundary.
Audience: Developers, evaluators, and technical decision makers
Published result#
The frozen 1.1.0 campaign compared AIT Task worktrees with Git worktrees over five game-development workloads. It admitted 100 AIT sessions and 100 Git sessions, with 20 paired sessions per workload and 100/100 accepted in both treatments.
The primary result gives each workload equal weight:
- workload-median token saving: 34.95%;
- bootstrap 95% confidence interval: 27.85% to 39.77%; and
- workload-median elapsed-time saving: 21.04%.
The pooled provider-token totals are a descriptive cross-check rather than the primary estimator: AIT used 46,300,272 tokens versus Git's 70,140,925, or 33.99% fewer tokens for AIT in this campaign.
Workload results#
| Workload | AIT median tokens | Git median tokens | Token saving | 95% CI | Pairs |
|---|---|---|---|---|---|
| GD-01 | 213,481.0 | 317,517.8 | 32.77% | 22.07–41.92% | 20/20 |
| GD-02 | 266,652.6 | 428,522.2 | 37.77% | 27.59–45.66% | 20/20 |
| GD-03 | 376,821.3 | 496,364.8 | 24.08% | 12.41–34.32% | 20/20 |
| GD-04 | 580,373.9 | 915,416.8 | 36.60% | 25.68–45.84% | 20/20 |
| GD-05 | 877,684.7 | 1,349,224.8 | 34.95% | 22.36–44.82% | 20/20 |
Every workload interval remained above zero in the frozen analysis.
Protocol boundary#
Both treatments used the same GPT-5.6 Sol model with maximum reasoning, the same pinned fixtures, the same task prompt and functional acceptance checks, fresh sessions, and provider-reported token accounting. Sessions were run linearly; this campaign did not measure concurrent throughput or scaling. It does not guarantee the same saving for every repository, model, or task.
Acceptance disclosure#
The campaign executed 201 sessions for 200 admitted observations. The original AIT GD-05 attempt failed the frozen functional check because its change did not preserve soundEnabled: 0. The failure remains in the run ledger, was excluded under the frozen acceptance rule, and was replaced once under the same model and fixture pin. The replacement passed and is the admitted observation.
Task-driven concurrency is a separate capability#
AIT makes the Task the unit of agent work. Independent Tasks own separate Changes, feature Lines, and worktrees, so commands can run concurrently across many agent sessions. Admission to a shared target Line is revalidated and serialized. This protects the target from stale writes; it does not make concurrent writes to one worktree safe.
See Parallel Task Isolation for the command and authority boundaries. This product capability is not part of the linear benchmark result above.