The headline result is not that an agent can write a plausible SQL model. data-eng-bench treats a task as solved only when every hidden assertion passes after dbt materializes the pipeline. That is the right bar for data engineering, where a model can look reasonable in code review while quietly double-counting revenue, mishandling a late-arriving record, or changing the grain of a downstream fact table.
But Snowflake is also the vendor selling CoCo, the harness that wins its first published comparison. The benchmark itself is open source and worth attention; the vendor’s performance table should be read as a reproducible product claim, not an independent verdict that CoCo is universally the best data-engineering agent.
A benchmark aimed at the part agents usually avoid
According to Snowflake AI Research’s launch post, data-eng-bench gives an agent a ticket-style natural-language request, a starting dbt project, and a retail data estate covering 579 source tables across 19 schemas and roughly 8,000 columns. The 84 build tasks range from blank-project work to changes inside an existing layered dbt repository, while 19 fix tasks require diagnosis and repair of deliberately broken models.
This is far closer to an analytics-engineering assignment than traditional text-to-SQL evaluation. Agents must create source declarations, use dbt macros, choose materializations, and wire staging, intermediate, and mart-layer models through
refdependencies. Snowflake says 82% of reference solutions use explicit refs, with roughly nine calls on average and as many as 42 models in the largest pipeline.
The test cases also target the failures that make autonomous changes difficult to trust. The verifier checks output grain, column contracts, formulas, edge cases, and repeatability. For more difficult tasks, Snowflake says the verifier independently reconstructs expected results in Python rather than merely comparing an agent’s SQL text with a reference file.
That testing design deserves credit. A model that gets 49 of 50 assertions right may be helpful to an engineer, but it has still failed a deployment-grade task if the remaining assertion represents an incorrect currency conversion, an improper FIFO inventory calculation, or a bad customer-segmentation boundary. data-eng-bench reports both partial assertion success and an all-or-nothing task completion rate, separating “nearly correct” from work that can pass a full hidden test suite.
Snowflake also makes the suite runnable with DuckDB as well as Snowflake. The project’s GitHub repository describes the default DuckDB route as fully hermetic: the retail dataset is contained in a local image, does not require a Snowflake account, and should let teams test whether an agent writes correct dbt logic without involving a live cloud warehouse.
“Shared warehouse” does not mean agents share a changing environment
The launch post describes every task as running against “one shared, persistent retail data warehouse,” an attractive framing for teams that maintain a common production data platform. The implementation documentation tells a more limited story: benchmark runs are isolated.
For the Snowflake backend, the repository says each task clones the source database into an isolated
retail_clone_*database, points the agent and verifier at that clone, and drops it after completion. For DuckDB, each task uses the bundled local database. The shared element is the common retail source dataset and its broad schema, not a live environment where one task’s altered models, grants, stale objects, or data writes become another task’s operational problem.
That is a necessary design choice for repeatable benchmarking. It prevents prior runs from contaminating later results and makes a three-run measurement meaningful. It also means data-eng-bench does not yet measure several risks that administrators and data-platform teams face when turning a coding agent loose on a real shared Snowflake account:
- It does not test whether multiple agents’ changes conflict in Git, dbt manifests, deployment jobs, or shared production schemas.
- It does not test whether an agent can safely operate amid warehouse queuing, changing source-data freshness, permission drift, or a partially completed prior deployment.
- It does not test an approval workflow where a human rejects an agent’s diff, changes the ticket scope, or requires a rollback after downstream consumers report breakage.
Those omissions do not invalidate the benchmark. They define it. data-eng-bench is a strong test of isolated repository-level dbt work over a large schema and complex business rules; it is not a test of unattended change management in a living enterprise data estate.
That difference should shape how teams interpret the results. Passing a task means an agent reached the correct output under controlled data and hidden checks. It does not establish that the same agent can be granted broad warehouse access, commit directly to a production branch, or replace data-engineering review.
CoCo’s advantage is a harness result, not a model result
Snowflake tested three proprietary models — Opus 5, Sonnet 5, and GPT 5.6 Sol — through CoCo, Claude Code, and Codex. In the published table, CoCo with Opus 5 achieved 73.8% Pass@1 and 64.1% Pass^3. Claude Code with Opus 5 posted 69.6% and 60.2%, respectively. With Sonnet 5, CoCo and Claude Code both recorded 56.6% Pass@1 and 40.8% Pass^3. With GPT 5.6 Sol, CoCo reached 64.1% Pass@1 versus 60.5% for Codex.
The most defensible conclusion is that the surrounding tools meaningfully affect outcomes. Snowflake’s own trace analysis says CoCo performed fewer SQL queries, tool operations, agent steps, file writes, and unnecessary DuckDB cross-validation checks than Claude Code when using Opus 5. A data-aware agent that explores the schema efficiently and validates against the actual target warehouse should have an advantage over a general coding workflow that spends extra turns checking a second dialect.
Snowflake’s CoCo documentation confirms that the product is designed around this exact use case. CoCo can access local repositories, execute SQL against a Snowflake account, invoke shell commands and Git operations, and includes built-in skills for data engineering, governance, machine learning, and agent creation. Comparing that purpose-built integration against more general agent environments is useful for buyers deciding which tool to deploy.
It is not, however, a neutral comparison of underlying model intelligence. The harness is part of the product. CoCo’s Snowflake-specific knowledge, tools, default workflows, and built-in skills are precisely the differentiators Snowflake wants customers to purchase. Claude Code and Codex could improve with different prompts, plugins, project instructions, database tools, or task-specific skills; Snowflake’s published results do not establish the ceiling for either competing environment.
The especially revealing result is Sonnet 5: CoCo and Claude Code had identical Pass@1 and Pass^3 figures, even though Snowflake says CoCo cost less. That suggests the harness can sharply affect efficiency without necessarily raising task-completion quality for every model. For IT teams, that may be the more practical finding. A coding agent that produces the same correct result with fewer tool calls, fewer tokens, and less unnecessary validation could reduce both API spend and operational noise.
The quoted dollar figures are not the full cost of a Snowflake run
Snowflake lists CoCo with Opus 5 at $0.756 per trial, Claude Code with Opus 5 at $2.959, and Codex with GPT 5.6 Sol at $0.538. Those figures are useful for comparing the token and agent-execution efficiency Snowflake measured. They should not be treated as an all-in cost to run the benchmark or operate an agent on a warehouse.
The benchmark repository says a full three-trial pass over all 103 tasks on the Snowflake variant consumes roughly six to nine warehouse-hours, even when performed with a free-tier account. Snowflake’s own CoCo documentation separately states that virtual warehouse compute and storage are billed at standard Snowflake rates apart from CoCo usage. The blog’s per-trial table does not state that it includes those warehouse charges.
For an enterprise deployment, the costs that matter extend further: warehouse consumption during agent exploration, cloned-database storage, model tokens, failed or repeated runs, CI infrastructure, observability, human review, and the occasional rollback. CoCo may still be cheaper in practice, but the published 3.9-times figure for Opus 5 is an agent-cost comparison, not proof of a 3.9-times lower total cost of ownership.
The consistency figures also argue against treating these systems as fire-and-forget automation. Snowflake’s best configuration passed 73.8% of tasks on the first-attempt average but passed all three independent attempts only 64.1% of the time. In other words, a substantial fraction of tasks that can succeed once are not reliably solved every time. For dbt work involving financial attribution, inventory valuation, or customer metrics, that reliability gap is where review gates remain necessary.
What teams can do with data-eng-bench now
The practical value of Snowflake’s release is that it gives data teams a public suite they can run against their own preferred agent, model, prompts, skills, and deployment rules. The repository supports Harbor, lists a balanced 30-task subset for faster experiments, permits DuckDB-only evaluation, and asks leaderboard submitters to run each task at least three times and publish results for review.
No independent outlet or third-party research group had published a reproduction of Snowflake’s exact CoCo-versus-Claude Code-versus-Codex results by August 7. That leaves Snowflake’s table as the only available result set for the newly released benchmark. The code and tasks are public, so that limitation is temporary — but users should wait for independent runs before turning a vendor-authored scorecard into a procurement conclusion.
For Windows-based data teams, the immediate test is straightforward: run the DuckDB variant in a controlled local container environment, then compare the agents your developers already use against the same hidden-verifier discipline. If an agent cannot repeatedly complete a synthetic dbt task with tests, it has no business receiving broad credentials on a production Snowflake account.
References
- Primary source: snowflake.com
Published: August 6, 2026 at 6:36 PM UTC
Loading…
www.snowflake.com - Related coverage: github.com
Loading…
github.com - Related coverage: techtarget.com
Loading…
www.techtarget.com - Related coverage: best-data-engineering-companies-usa.com
Loading…
best-data-engineering-companies-usa.com - Related coverage: recommend-data-engineering-consulting-companies.com
Loading…
recommend-data-engineering-consulting-companies.com - Related coverage: github.com
Loading…
github.com - Related coverage: simplywall.st
Loading…
simplywall.st