The experiment: five green runs, fifteen problems
Dev Proxy is a local proxy that intercepts an app's HTTP requests. According to Microsoft's write-up, the team asked GitHub Copilot Chat, using a model it names as Claude Sonnet 5.5 and no extensions, to write a CI script five times. The script had to use the Dev Proxy API to record which GitHub APIs a Node.js tool calls.
All five scripts worked. The trajectories still showed 15 problems. Each run took 29 to 39 requests and used up to 1.2 million tokens.
These figures come from Microsoft's own experiment. We have no access to the underlying trajectory files and haven't reproduced them. Treat them as reported results, not independently verified ones.
This sits inside a larger effort. Microsoft's DevRel team has published an "AX Practitioner Playbook." Since fall 2025 the team has measured how agents perform with Azure, Cosmos DB, SharePoint Framework (SPFx), and Microsoft 365 Copilot extensions, using the same kinds of prompts real developers use. The playbook's site pairs a tool for observing agent behavior (it names "atifact") with Dev Proxy for testing changes before shipping them.
Seven patterns worth hunting for
1. Agents guess URLs before they search
Four of the five runs fetched a plausible but wrong docs path ending in technical-reference/api. The real page is technical-reference/proxy-api. Two runs burned five to seven extra steps finding it. The other two gave up on the docs and worked from memory. One run later guessed a plugin page path wrongly, and eight fetches across four runs returned 404.
Fix: Check your 404 logs for near-miss paths and add redirects. Print the docs URL in CLI output so nobody has to guess.
2. Errors that say what's wrong, not what's right
Dev Proxy's API listing shows GET /proxy/rootCertificate with no parameters. One run called it as listed and got a 400 saying format was required. It then tried pem, crt, cer and der. Only crt worked.
Microsoft Learn's Proxy API page shows the full contract. The endpoint is GET /proxy/rootCertificate?format=crt. The page says the only supported format is crt, and any other value, or none, returns 400. The shorter devproxy api show page lists the endpoint without that detail.
Fix: Put valid values and a working example in help text and error responses.
3. Every run reinvents the same workaround
Dev Proxy writes its reports shortly after recording stops and gives no completion signal. Four runs polled the output folder for up to 30 seconds, and the fifth just slept for two seconds. Polling is safer, but each polling run had to guess which file to wait for.
There was also no command to list plugins. All five runs read the plugin DLL as raw bytes and searched it for class names. No score would ever show that, but the same odd step appears in every run. That repetition is the signal that a feature is missing.
Fix: Give asynchronous operations a way to say they're done. Give every choice set, such as plugins or templates, a list command.
4. Agents fill documentation gaps from memory
All five runs knew that Node's fetch needs an opt-in setting to honor proxy environment variables. None found it in the Dev Proxy docs. The article's point is that a model that knew less would have recorded nothing and raised no error.
Node's own documentation adds version detail that the shorthand skips. Node's enterprise network guide says proxy support via NODE_USE_ENV_PROXY or --use-env-proxy works with fetch() from v22.21.0 or v24.0.0. The same guide says it works with node:http and node:https from v22.21.0 or v24.5.0. Node's v22.21.0 release notes list the --use-env-proxy flag and proxy support for fetch among the notable changes. Which Node version you're on, and which API you use, decides whether traffic actually routes through the proxy.
Fix: Document the setup for each runtime and framework your developers pair with your tool.
5. Agents reject defaults built for a first run
Dev Proxy's default config simulates random errors against a sample API. That's good for a first run and wrong for recording real traffic. The agents avoided it, created empty configs, and then built their own. Across runs they chose three different plugins for the same goal.
Fix: Keep the first-run default. Add task-specific presets and list them in CLI help.
6. Search serves stale snapshots
Since v3.3.1, the Dev Proxy API requires a token. One run's search surfaced two results indexed before that change: a Ukrainian translation of the API reference crawled in June, and a file from Dev Proxy's own agent skill. Both showed unauthenticated calls that now return 401. That run got authentication right only because it had already read the CLI help, which ships with the installed version.
Current docs match the change. The Proxy API page says all operations require a bearer token, obtainable with devproxy status or devproxy api token. Requests without a valid token return 401. The v3.3.1 boundary and the search-index incident come from Microsoft's write-up alone.
Fix: Run the searches an agent would run and check whether the results describe your current version. State version requirements at the top of each page so they show up in snippets.
7. Unstated defaults with side effects
Dev Proxy registers itself as the system proxy by default. The help text reads like an opt-in: --as-system-proxy followed by "Set Dev Proxy as the system proxy." Turning it off takes an explicit --as-system-proxy false.
Only one run noticed, because a status command showed the proxy as enabled. Another disabled it in config, and three never noticed. On a Windows CI runner, those scripts would reroute all the runner's traffic. Their kill fallback would also skip the cleanup that undoes it.
Microsoft Learn's proxy settings page supports the underlying default. It lists asSystemProxy as defaulting to true, and says enabling it requires installCert to be true. That also defaults to true.
The scoring problem is the most interesting part. All three unaware runs passed because the system proxy wasn't in the evaluation criteria. The judge wrongly failed the run that turned it off. So the score was wrong in both directions.
Fix: State every default in help text. Show how to turn off anything that's on by default. Pay extra attention to defaults that touch system settings or certificates, and consider disabling them for unattended runs.
Product and docs first, skills second
The write-up argues against reaching for an agent skill first. A skill only helps developers who find and install it, and it becomes one more copy of your docs that can go stale. A fix in the product or docs reaches every agent with nothing to install.
The example is SPFx. A trajectory showed the agent reading release notes while it worked, so the team fixed the release notes instead of shipping a skill. The playbook site describes the change. The agent had planned a manual upgrade and read the 1.22 release notes instead of 1.22.2. Release notes pages now use full version numbers, and the migration guide warns that hand-editing package.json breaks the build. The site reports the eval going from 30 of 80 configuration checks to 83 of 85. It doesn't specify which change drove that lift.
How to start
- Read the trajectories yourself, including passing runs. You know your product well enough to spot workarounds a reviewing agent would treat as normal.
- Use a viewer. The article names ATIF Preview, a VS Code extension. Its README describes an interactive step-by-step view of messages, tool calls, reasoning traces and token cost for ATIF trajectory files. ATIF is an open format for recording agent sessions.
- Let an agent help compare runs, but only if it cites the step behind each finding.
- Classify each friction point. It's a product gap, a docs gap, a stale-source problem, a default or configuration issue, or an eval blind spot.
- Make the smallest change, then test it. Each finding is a hypothesis until you run the same tasks with and without the fix and compare quality and cost.
What to make of it
The experiment has caveats. It's one product, one task, five runs, and a vendor testing its own tool and writing about it. The numbers are self-reported. Several of the fixes, such as redirects and documented defaults, are good practice even without agents. The seven patterns are still concrete and checkable against your own tooling.
For Windows and CI administrators, the system-proxy item is the one to take away from the article. An unattended agent-written script that quietly reroutes a build machine's traffic, and skips its cleanup, is the kind of failure no pass/fail score will report. If agents write scripts that touch your runners, read the trajectories and the help text, not just the green checkmarks.
Correction (November 5, 2026): The SPFx release-notes fix took the eval from 30 of 80 configuration checks to 83 of 85. An earlier version of this article cited 72 to 85 of 85, a figure from a different experiment on the same playbook page.