Convex Evals
2026-10-28 · Framework eval · 92a60e0bb2f156b5ac44e6ea3f5d03a91deb434b9d002d14819d1fc2b6e35cdb
- What it measures
- Models write Convex code and the harness runs it. The current version has 112 tasks. The baseline is No Guidelines: no supplied guidelines and no web access.
- How
- The official Convex coding evals. Rank uses only No Guidelines. With Guidelines is another column: the official AI rules are loaded. No Guidelines + Web is a third column: the model may search and open pages, still without guidelines. The columns are not averaged or added.
- What the score means
- The percent is the pass rate on 112 tasks, kept to two decimals. The with-guidelines and web columns appear only when that model has a run. A missing run is not a zero.
- Caveats
- Convex writes the tasks and grades them. This measures a model together with Convex, not one framework against another. The front is near the ceiling. With Guidelines comes from the unscoped runs:leaderboardScores query; it matches the With Guidelines tab on the page. The experiment=with_guidelines parameter still returns a server error.
- Updates
- Version 92a60e0bb2f156b5ac44e6ea3f5d03a91deb434b9d002d14819d1fc2b6e35cdb. This page synced No Guidelines (41→35 mapped) and With Guidelines (40→35 mapped via empty-args default query; experiment=with_guidelines still 500s) from fabulous-panther-525.convex.cloud on 2026-10-28, keeping the best row per model; No Guidelines + Web was not refreshed this round. On With Guidelines, Gemini 4 Argon leads near 98.7%, GPT-5.6 Sol is about 98.2% at #2, and GPT-6 Astra is about #7.
Baseline board
| # | Model | Baseline | With guidelines | No guidelines + web | Cost per run |
|---|---|---|---|---|---|
| 1 | GPT-6 Astra 3 runs | 89.88 | 97.02 | 91.07 | $6.05 |
| 2 | Claude Opus 5.5 3 runs | 89.58 | 97.77 | 95.09 | $3.63 |
| 3 | Gemini 4 Argon 8 runs | 88.62 | 98.72 | 95.76 | $1.62 |
| 4 | GPT-6.1 Sol 5 runs | 88.57 | 96.43 | 92.19 | $1.19 |
| 5 | Claude Fable 5 1 run | 85.71 | 97.32 | 89.29 | $7.90 |
| 6 | Claude Sonnet 5.5 4 runs | 85.49 | 97.68 | 88.39 | $1.45 |
| 7 | GPT 5.6 Sol 2 runs | 84.82 | 98.21 | 89.29 | $2.52 |
| 8 | Claude Fable 5.1 1 run | 83.04 | 97.32 | 89.29 | $8.63 |
| 9 | Grok 4.6 1 run | 81.25 | 94.64 | — | $2.97 |
| 10 | Claude Opus 5 2 runs | 80.8 | 94.64 | 85.71 | $4.46 |
| 11 | Grok 4.7 1 run | 80.36 | 94.64 | — | $4.65 |
| 12 | Claude Opus 4.8 1 run | 79.46 | 92.86 | 85.71 | $4.47 |
| 13 | GPT 5.5 1 run | 77.68 | 89.29 | — | $6.69 |
| 14 | Grok 4.5 2 runs | 77.68 | 95.09 | 80.36 | $1.58 |
| 15 | Muse Wonder 9 runs | 76.59 | 93.45 | 84.46 | $6.49 |
| 16 | Claude Haiku 5.5 8 runs | 75.0 | 89.62 | 78.44 | $0.15517 |
| 17 | Claude Sonnet 5 1 run | 74.11 | 86.61 | 80.36 | $1.86 |
| 18 | Gemini 3.1 Pro Preview 1 run | 72.32 | 84.82 | — | $3.99 |
| 19 | Kimi K3.1 Pro 2 runs | 71.43 | 93.3 | — | $2.05 |
| 20 | GPT 5.6 Terra 2 runs | 69.64 | 89.73 | 79.46 | $1.93 |