Supabase has open sourced Supabase Evals, a benchmark framework that runs AI coding agents such as Claude Code, Codex, and OpenCode through real Supabase development and troubleshooting work rather than isolated code-generation prompts. The project, published under the Apache-2.0 license, is designed to answer a harder and more useful question for teams using AI in production: can an agent safely build, deploy, investigate, and repair a backend that includes databases, authentication, Row Level Security, Edge Functions, and the surrounding tooling?
MarkTechPost first highlighted the release on August 1, while Supabase’s own July 31 announcement describes the repository as both the engine behind a public leaderboard and a daily internal regression system. That distinction matters. This is not simply a vendor scorecard showing which frontier model writes the best SQL; it is a test harness intended to catch the precise ways an agent can appear successful while leaving a Supabase project broken, insecure, or impossible to maintain.
For Windows developers running Docker Desktop, Node.js, and the Supabase CLI locally, the repository is deployable now through
Supabase Evals divides its scenarios across three axes: product areas, technical topics, and the stage of work an agent is being asked to complete. The product list includes database, auth, storage, Edge Functions, Realtime, cron, queues, vectors, and the Data API. Topics span RLS, security, SQL, migrations, SDKs, observability, self-hosting, tests, and declarative schemas.
The stages are equally important. A coding assistant that can build a table from scratch is not necessarily capable of diagnosing why an existing Edge Function is failing, or repairing a policy that returns empty data to legitimate users. Supabase therefore uses build, deploy, investigate, and resolve as separate categories, selecting a minimal scenario set that touches every dimension at least once.
Those scenarios are based on problems familiar to backend administrators: support tickets, bug reports, and GitHub issues. One task may ask an agent to construct an application schema; another may supply a broken hosted project and require an investigation; another may require a repair to RLS behavior. This gives the benchmark considerably more operational relevance than evaluations based solely on whether a generated code snippet compiles.
The public suite is intentionally broad. It captures representative tasks and feeds the published scores. A separate regression suite is deeper and tracks known failure modes, with results refreshed daily but excluded from public benchmark totals. That separation is a sensible safeguard: it allows Supabase to test a newly reported bug without turning one narrow issue into a permanent distortion of a model’s public ranking.
For hosted-style tasks, the project uses a component called
For CLI-oriented scenarios, Supabase Evals starts a Docker sandbox that includes Bash, file operations, and the real Supabase CLI. The agent can use commands including
That latter detail deserves attention. The scorer is intended to evaluate the agent’s delta, not reward environment setup completed by the harness. If a local project is pre-started or seeded for the task, the grading logic should focus on whether the agent made the correct repair or implementation on top of that state. It is a more credible approach than giving agents credit simply because a prebuilt environment was already working.
Each evaluation has a predictable shape: a
The LLM judge covers requirements that cannot be reduced cleanly to a boolean without losing important context. Supabase’s earlier Agent Skills reporting described this type of grading as reviewing transcripts, generated files, tool calls, SQL, and real outputs—not merely accepting an agent’s assertion that it completed the job. That is the right standard for agent evaluation, though it still means users should treat leaderboard percentages as a directional measure rather than a final certification of production readiness.
The harness permits one retry after an agent fails and before its work is graded. That may improve resilience against transient tool errors and prevent a single recoverable misstep from becoming a false negative. It also means teams comparing results should pay attention to the evaluation protocol, not just the headline percentage: an agent that needs a retry may be perfectly adequate for supervised development but inappropriate for unattended remediation.
For organizations making purchasing or policy decisions around coding assistants, reproducibility is the major value. A model label alone is not enough. The result depends on the model, the agent harness, the enabled skills, MCP availability, the CLI version, the exact task state, and whether the agent can read current documentation. Supabase Evals represents those conditions explicitly as experiments rather than treating them as invisible variables.
It also found that agents writing Edge Functions tend to manually verify authentication through
The RLS angle is more consequential for IT professionals than either workflow preference. A malformed policy can turn into a data exposure, a silent application failure, or a support incident that is hard to reproduce. Supabase’s April Agent Skills release had already warned that AI tools can omit RLS policies, misuse user metadata for authorization, expose service-role credentials in front-end code, or create views that bypass RLS unless
These are not academic mistakes. In a fintech, healthcare, SaaS administration, or internal business system, a plausible-looking agent patch can pass superficial testing while weakening tenant boundaries. Supabase Evals does not eliminate that risk, but it makes it possible to turn a bad incident pattern into a regression test—and to reject a skill, documentation change, SDK release, or agent configuration if it reintroduces the problem.
The figures are vendor-published results from a fast-moving benchmark, so they should not be read as permanent model rankings. Supabase itself advises readers to treat them as a point-in-time snapshot. The broader conclusion is more durable: structured, current product guidance is most valuable at the edges, where a model’s general coding ability collides with a service-specific convention, a recent API change, or a security requirement that training data may not emphasize.
The documentation behavior is especially revealing. Supabase says Codex-based agents, including a GPT-5.6 configuration, read about eight documentation pages per scenario, compared with roughly two pages for Claude Code. Claude Code reportedly checked Supabase documentation in fewer than 40 percent of scenarios even when skills were loaded.
That does not prove one product is inherently safer than another. It does show why enterprises should test the whole agent setup they intend to authorize. A capable model that skips current docs can confidently implement an outdated pattern. A smaller model with a well-designed skill, tool access, and a verification workflow may be a safer fit for constrained tasks.
A team maintaining a Supabase-backed application could add internal scenarios before allowing an AI agent to modify database code or operational runbooks. That might include a test for a tenant-isolation policy, an Edge Function authentication regression, a migration conflict, a Storage permission issue, or an incident-resolution playbook. The agent’s output can then be tested against a real local stack before it reaches a pull request or a staging environment.
The caveat is that Docker socket access is powerful. Supabase Evals mounts the host Docker socket into its sandbox so
Supabase says it plans to expand edge-case coverage, improve scoring stability, and potentially promote mature regression scenarios into the published benchmark. It is also working on a CLI command and MCP tool that would let agents report where they struggled. The immediate test for the project will be whether outside teams turn its open framework into their own deployment gates—especially for the unglamorous failures that only become visible after an AI-written backend reaches real users.
For Windows developers running Docker Desktop, Node.js, and the Supabase CLI locally, the repository is deployable now through
pnpm. But its local-stack mode is a serious test environment, not a quick demo: it requires a working Docker daemon, API credentials for the selected model provider, and Supabase’s default ports 54321 through 54329 to be available.
The Benchmark Tests the Work Agents Actually Perform
Supabase Evals divides its scenarios across three axes: product areas, technical topics, and the stage of work an agent is being asked to complete. The product list includes database, auth, storage, Edge Functions, Realtime, cron, queues, vectors, and the Data API. Topics span RLS, security, SQL, migrations, SDKs, observability, self-hosting, tests, and declarative schemas.The stages are equally important. A coding assistant that can build a table from scratch is not necessarily capable of diagnosing why an existing Edge Function is failing, or repairing a policy that returns empty data to legitimate users. Supabase therefore uses build, deploy, investigate, and resolve as separate categories, selecting a minimal scenario set that touches every dimension at least once.
Those scenarios are based on problems familiar to backend administrators: support tickets, bug reports, and GitHub issues. One task may ask an agent to construct an application schema; another may supply a broken hosted project and require an investigation; another may require a repair to RLS behavior. This gives the benchmark considerably more operational relevance than evaluations based solely on whether a generated code snippet compiles.
The public suite is intentionally broad. It captures representative tasks and feeds the published scores. A separate regression suite is deeper and tracks known failure modes, with results refreshed daily but excluded from public benchmark totals. That separation is a sensible safeguard: it allows Supabase to test a newly reported bug without turning one narrow issue into a permanent distortion of a model’s public ranking.
Real Tooling, Not a Decorative Sandbox
The technical design is the release’s central contribution. Supabase says each scenario runs against a real environment, with a hosted-like project surface and a local CLI project created in containers. Agents invoke the actual Supabase MCP server or the actual Supabase CLI, rather than interacting with a mock command interface that only resembles the production tools.For hosted-style tasks, the project uses a component called
platform-lite, which exposes a Management API-compatible interface backed by @supabase/lite. This means tools such as the Supabase MCP server can work against a lightweight but compatible project target. The test can seed database state, deployed Edge Functions, and observability logs before the agent begins.For CLI-oriented scenarios, Supabase Evals starts a Docker sandbox that includes Bash, file operations, and the real Supabase CLI. The agent can use commands including
supabase init, supabase start, supabase db, and supabase test against a local stack. The test harness later exports the modified workspace and checks what the agent actually changed.That latter detail deserves attention. The scorer is intended to evaluate the agent’s delta, not reward environment setup completed by the harness. If a local project is pre-started or seeded for the task, the grading logic should focus on whether the agent made the correct repair or implementation on top of that state. It is a more credible approach than giving agents credit simply because a prebuilt environment was already working.
Each evaluation has a predictable shape: a
PROMPT.md file containing task instructions and metadata, an EVAL.ts scoring file, plus optional remote/ and local/ starting states. This makes the project potentially useful beyond Supabase itself. Teams can inspect a scenario, alter a starting condition, add an internal failure case, and compare different agent runtimes under a repeatable framework.Scoring Leaves Room for Judgment but Not Empty Claims
Supabase Evals combines deterministic checks with LLM-as-a-judge assessment. Deterministic checks handle outcomes that can be verified mechanically: whether a user can access the correct data, whether a policy allows or blocks the intended operation, whether an Edge Function returns an expected result, or whether a project’s test suite passes.The LLM judge covers requirements that cannot be reduced cleanly to a boolean without losing important context. Supabase’s earlier Agent Skills reporting described this type of grading as reviewing transcripts, generated files, tool calls, SQL, and real outputs—not merely accepting an agent’s assertion that it completed the job. That is the right standard for agent evaluation, though it still means users should treat leaderboard percentages as a directional measure rather than a final certification of production readiness.
The harness permits one retry after an agent fails and before its work is graded. That may improve resilience against transient tool errors and prevent a single recoverable misstep from becoming a false negative. It also means teams comparing results should pay attention to the evaluation protocol, not just the headline percentage: an agent that needs a retry may be perfectly adequate for supervised development but inappropriate for unattended remediation.
For organizations making purchasing or policy decisions around coding assistants, reproducibility is the major value. A model label alone is not enough. The result depends on the model, the agent harness, the enabled skills, MCP availability, the CLI version, the exact task state, and whether the agent can read current documentation. Supabase Evals represents those conditions explicitly as experiments rather than treating them as invisible variables.
The Most Important Failures Are Security and Workflow Failures
Supabase’s initial findings are less interesting as a horse race between named models than as a catalog of where modern agents still take shortcuts. The company found agents commonly hand-write database migrations even when a project is already using declarative schemas, which are meant to provide a single source of truth for database structure. Supabase responded by revising its skill guidance around when the workflows should be used.It also found that agents writing Edge Functions tend to manually verify authentication through
supabase-js rather than choose @supabase/server, a newer library intended to reduce boilerplate around secure Edge Function work. Supabase has since published package-selection guidance, using the evaluation system to see whether the documentation and skill changes improve behavior.The RLS angle is more consequential for IT professionals than either workflow preference. A malformed policy can turn into a data exposure, a silent application failure, or a support incident that is hard to reproduce. Supabase’s April Agent Skills release had already warned that AI tools can omit RLS policies, misuse user metadata for authorization, expose service-role credentials in front-end code, or create views that bypass RLS unless
security_invoker = true is set.These are not academic mistakes. In a fintech, healthcare, SaaS administration, or internal business system, a plausible-looking agent patch can pass superficial testing while weakening tenant boundaries. Supabase Evals does not eliminate that risk, but it makes it possible to turn a bad incident pattern into a regression test—and to reject a skill, documentation change, SDK release, or agent configuration if it reintroduces the problem.
Skills Help, but They Are Not a Substitute for Verification
Supabase reports that the strongest agents already passed most benchmark scenarios without a loaded skill. In the Build stage, Opus 5 and Kimi K3 scored 100 percent without skills in the snapshot Supabase published. But skills narrowed the gap for other configurations: Sonnet 5 reportedly moved from 78 percent to 100 percent, GPT-5.6 Sol from 89 percent to 100 percent, and GPT-5.4 mini from 78 percent to 89 percent.The figures are vendor-published results from a fast-moving benchmark, so they should not be read as permanent model rankings. Supabase itself advises readers to treat them as a point-in-time snapshot. The broader conclusion is more durable: structured, current product guidance is most valuable at the edges, where a model’s general coding ability collides with a service-specific convention, a recent API change, or a security requirement that training data may not emphasize.
The documentation behavior is especially revealing. Supabase says Codex-based agents, including a GPT-5.6 configuration, read about eight documentation pages per scenario, compared with roughly two pages for Claude Code. Claude Code reportedly checked Supabase documentation in fewer than 40 percent of scenarios even when skills were loaded.
That does not prove one product is inherently safer than another. It does show why enterprises should test the whole agent setup they intend to authorize. A capable model that skips current docs can confidently implement an outdated pattern. A smaller model with a well-designed skill, tool access, and a verification workflow may be a safer fit for constrained tasks.
Windows Teams Can Use It as a Pre-Deployment Gate
The immediate audience is not limited to teams building directly on Supabase. The framework provides a practical pattern for Windows-based developer environments where Docker Desktop hosts local services, PowerShell or Windows Terminal launches agent tools, and CI runs on GitHub Actions or another build system.A team maintaining a Supabase-backed application could add internal scenarios before allowing an AI agent to modify database code or operational runbooks. That might include a test for a tenant-isolation policy, an Edge Function authentication regression, a migration conflict, a Storage permission issue, or an incident-resolution playbook. The agent’s output can then be tested against a real local stack before it reaches a pull request or a staging environment.
The caveat is that Docker socket access is powerful. Supabase Evals mounts the host Docker socket into its sandbox so
supabase start can launch sibling containers. That is appropriate for an isolated developer workstation or carefully controlled CI runner, but it is not a workload to run casually on a shared or sensitive build host. API keys and generated results also need the same care as other development credentials and logs.Supabase says it plans to expand edge-case coverage, improve scoring stability, and potentially promote mature regression scenarios into the published benchmark. It is also working on a CLI command and MCP tool that would let agents report where they struggled. The immediate test for the project will be whether outside teams turn its open framework into their own deployment gates—especially for the unglamorous failures that only become visible after an AI-written backend reaches real users.