Android Bench
2026-10-26 · Mobile
2.0 long-horizon
- What it measures
- Long Android changes across modules: a whole feature, not a single edit.
- How
- Long-horizon task results on developer.android.com/bench. Each row is a model plus its agent harness. Columns are pass rate, CI, completion, average latency, and average cost.
- What the score means
- The percent is the average pass rate across 30 tasks. Completion is how close a failed run got. Latency is in hours. Cost is one full run.
- Caveats
- Pass rates are low and intervals are wide. First place, 24.0–56.0, overlaps second place, 22.0–52.0. Version 1.0 stays complete on this page as an archived version, and is no longer the scenario's current column.
- Updates
- The page says Latest results as of October 26th. This page checked all 11 rows against #long-horizon-task-results on 2026-10-28.
Full board
11
All
| # | Model | Agent | Pass rate % | CI | Completion % | Avg latency h | Avg cost $ |
|---|
| 1 | Claude Sonnet 5.5 | claude-code | 40.0 | 24.0–56.0 | 80.9 | 70.7 | $1574.9 |
| 2 | Claude Opus 5.5 | claude-code | 37.3 | 22.0–52.0 | 85.6 | 56.6 | $1150.3 |
| 3 | GPT 6.1 Sol | codex | 29.3 | 15.3–42.7 | 82.8 | 26.6 | $99.5 |
| 4 | GPT 6 Astra | codex | 28.0 | 13.3–42.0 | 82.2 | 7.9 | $375.7 |
| 5 | Claude Fable 5.1 | claude-code | 22.7 | 10.7–36.0 | 82.4 | 22.2 | $492.6 |
| 6 | Gemini 4 Argon | antigravity-sdk | 21.3 | 10.0–33.3 | 81.8 | 52.8 | $267.6 |
| 7 | Qwen3.8 Max | qwen-coder | 12.5 | 4.2–21.7 | 75.0 | 45.4 | $258.4 |
| 8 | Kimi K3 | kimi-code | 12.0 | 3.3–22.0 | 72.8 | 66.2 | $418.3 |
| 9 | GPT 6 Luna | codex | 10.7 | 4.0–17.3 | 53.2 | 6.2 | $5.3 |
| 10 | Gemini 3.8 Flash | antigravity-sdk | 8.0 | 3.3–13.3 | 47.4 | 12.1 | $34.5 |
| 11 | Gemini 3.7 Flash | antigravity-sdk | 7.3 | 1.3–14.7 | 50.1 | 9.9 | $26.0 |