GitHub's project page describes it as a way to create, run and compare tests for Copilot skills, custom agents and MCP tools. The short launch post makes the same pitch in plainer terms.
What an eval is
An eval is a repeatable test for AI behavior. You give an agent a realistic task and describe what a good result looks like. The eval then checks both the result and how the agent got there.
GitHub's reasoning is simple. AI output varies from run to run, so one good answer says little about whether a customization works for you. The suggested habit is to change a skill, agent or MCP tool, rerun the same eval, and look for regressions.
What you can do with it
You can ask Copilot in plain language or use the /eval command. The announced workflow covers four things:
- Create an eval for a skill, custom agent or MCP tool in your project.
- Run it and see which checks passed and which failed. You approve each run before it starts.
- Dig into why a check failed and decide what to change.
- Compare runs from before and after a customization change.
According to the project documentation, the /eval workflows are /eval setup, /eval create, /eval review, /eval improve, /eval run and /eval compare in Copilot CLI and the GitHub Copilot app. The same documentation lists an Evals Canvas. That is a visual side-panel for browsing project assets, drafting evals, inspecting YAML and reading saved run reports.
Where to get it and who gets it
The preview covers Copilot Individual, Business and Enterprise plans. GitHub names three install routes:
- VS Code: install the GitHub Copilot Evals extension from the Visual Studio Marketplace. The Marketplace listing shows it as a free preview extension from GitHub.
- Copilot CLI: install the plugin. GitHub's repository gives these commands:
copilot plugin marketplace add github/evalscopilot plugin install eval@github-copilot-evals
- GitHub Copilot app: open Customize, add the
github/evalsmarketplace and install the plugin.
The repository says the /eval commands and the Canvas currently need Copilot CLI experimental mode. You turn it on with /settings experimental on. Without it, you can still ask for the six skills in natural language. The extension-only commands, tools and Canvas stay unavailable.
The announcement gives no preview end date or general-availability date. It also says nothing on pricing, regional availability or admin controls.
A first eval, step by step
The repository's walkthrough runs like this:
- Open a local project in Copilot CLI or VS Code, trust the folder and sign in to Copilot.
- Run
/eval create, or open the Canvas, and pick the skill, agent or MCP tool to test. - Describe what passing looks like. The repository's example is a release-notes skill. It checks that the output includes only completed work, keeps issue IDs and leaves the input files unchanged.
- Review the proposed test cases, fixtures and checks, then save the eval.
- Run
/eval run, review the disclosed execution effects and token or credit costs, then approve. - Inspect the saved report for passing checks, outputs and the conversation trace.
- After editing your skill or agent, rerun the eval and use
/eval compareagainst the earlier saved result.
Prerequisites
Running evals uses a local runner. The repository lists these minimums:
| Dependency | Minimum |
|---|---|
| Node.js | 24.0.0 (CLI plugin) or 22.14.0 (VS Code) |
| @microsoft/vally-cli | 0.19.0 |
| GitHub Copilot CLI | 1.0.85 |
| Git | 2.28.0 |
These are repository-documented figures and may change during the preview. The repository adds a caveat of its own. A separate GitHub starter template pins a newer Vally version, 0.21.0. Check the live documentation rather than treating 0.19.0 as a ceiling.
For VS Code, the extension offers a runner setup command, "GitHub Copilot Evals: Set up test runner", or you can go to Evals Canvas > Settings > Advanced. For a missing or incompatible runner, the listing says to rerun that setup command. For run diagnostics, it points to Output > GitHub Copilot Evals.
Cost, safety and privacy
This is the part admins should read before rolling it out to a team.
- Saving is not running. Validating files and reading or comparing saved results make no model calls. Each eval run or grader test asks you to review its execution effects and token or credit costs first.
- Evals can execute code. The documentation warns that setup scripts, tools, custom graders and AI judges can run code and incur charges under your normal local permissions. GitHub's plugin-marketplace starter adds a note. Execution uses the host, not a container sandbox, so review setup, tools, network access and credential exposure before approving.
- A pass is not a guarantee. The docs say a passing check doesn't mean the AI will get it right every time.
- Results are stored in your project. Saved results go to
.copilot\evals\results. Review them before sharing, since they can include outputs and conversation traces. - Telemetry. The standalone plugin's telemetry is on by default in released versions. The docs say it records activation, workflow requests, outcomes, error categories, durations and basic environment and version information. They say it excludes source code, prompts, model outputs, project paths, repository names and raw error messages. Opt out by setting any one of
COPILOT_EVALS_TELEMETRY=off,DO_NOT_TRACK=1orOTEL_SDK_DISABLED=truebefore starting the host. The VS Code extension's separate telemetry follows VS Code'stelemetry.telemetryLevelsetting. These are project statements, not a blanket Copilot privacy guarantee.
Licensing
The repository says the project is source-available under the GitHub Limited Use Source License and is not open-source software. The documentation says you may inspect and contribute to the code and use it internally with authorized tooling. It forbids monetizing it, offering it as a product or service, or using it with unauthorized tooling. Legal teams should read the full license.
CI and the wider ecosystem
GitHub has related repositories beyond the launch post. An Evals Action lints evaluation definitions, previews execution plans and runs Vally suites in CI with regression reporting. It updates a single sticky pull request comment with verdicts and evidence links. I haven't seen GitHub describe it in the launch post, so treat it as a separate, evolving project. Its release notes call the current version a prerelease.
There is also community tooling around Vally. One example is a third-party executor that runs Vally evals against Copilot Chat in a real VS Code workspace. It isn't GitHub's, but it shows the engine being used beyond this launch.
Analysis: who should care
Teams that have built custom agents, skills or MCP servers are the obvious audience. A skill library that nobody tests is a collection of educated guesses. A saved baseline run, one change, then a compare gives you something closer to evidence. The explicit run-approval step is a sensible guardrail for something that can spend credits and run scripts.
There are fair reasons for caution:
- It's a preview. Prerequisites, experimental-mode requirements and behavior can shift.
- Evals test what you define. Vague checks produce vague confidence. A green result only means the agent passed the checks you wrote.
- Non-determinism cuts both ways. Repeated runs cost tokens, and one pass or one failure still isn't proof.
- Governance is unclear. The launch post doesn't address admin controls, so enterprises will want to confirm how it fits their Copilot policies.
A sensible pilot is to pick one important skill and write two or three observable checks. Save a baseline, change one thing, and compare. Check the cost disclosure before every run.
References
- GitHub Copilot Evals is now in public preview GitHub Changelog · 2026-11-17T18:00:33+00:00
- GitHub - github/plugin-marketplace-starter: A starter for building a GitHub Copilot plugin marketplace with skills, agents, MCP tools, and evals. · GitHub github.com
- GitHub - github/evals: Explore, evaluate, and improve GitHub Copilot customizations with repeatable evals and saved results. · GitHub github.com