Supabase Evals
2026-10-30 · Framework eval · benchmark suite · 2026-10-30
- What it measures
- Models work Build / Deploy / Investigate / Resolve tasks across the Supabase developer journey. The public table aggregates Total % by agent.
- How
- The official Supabase evals. On the public page Experiment suite, Without skills (skills: []) is the baseline; Benchmark (loads supabase and supabase-postgres-best-practices) is the with-skills arm. The two columns stay separate and are not merged into one pass rate.
- What the score means
- The percent is Total % from the public table. Rank follows the Without skills baseline. The change is Benchmark (with skills) minus baseline, in percentage points, and it can be zero or negative.
- Caveats
- Supabase writes the tasks and grades them. This measures a model together with Supabase, not one framework against another. The public board lists six agents. The vendor authors the tasks and grades them. Page scores are whole percents.
- Updates
- Synced the public Experiment suite on 2026-11-02 (board dated to the 2026-10-30 monitor expect). Without skills and Benchmark are kept as separate arms.
Baseline board
| # | Model | Baseline | With skills |
|---|---|---|---|
| 1 | Claude Opus 5.5 medium · Claude Code | 90.0 | 93.0 |
| 2 | Kimi K3 OpenCode | 88.0 | 86.0 |
| 3 | Grok 4.7 high · Grok | 84.0 | 83.0 |
| 4 | GPT-6.1 Sol medium · Codex | 78.0 | 81.0 |
| 5 | Claude Sonnet 5.5 medium · Claude Code | 77.0 | 86.0 |
| 6 | GPT 6 Luna medium · Codex | 70.0 | 77.0 |