Microsoft Test Finds AI Cutoffs Don’t Predict Product Skills

The experiment used GPT-5.6 Luna through the GitHub Copilot SDK, with external information access removed. Its findings concern that restricted evaluation—not the performance of every GitHub Copilot configuration, and not an agent equipped with current documentation and tools.

What Microsoft’s experiment measured​

Mastykarz started with a seemingly useful proposition: identify the newest product version a model knows, then provide documentation for everything introduced afterward. That would leave more room in the agent’s context window, the finite amount of information available while it works.

To build the evaluation, he supplied Dev Proxy and SPFx changelogs and release notes to GPT-5.6 Sol. The model extracted product changes, excluding dependency updates and continuous-integration infrastructure work. A model then selected changes with enough behavioral difference to support a test and generated a task prompt and scoring rubric for each.

Mastykarz ran those tasks in Vally using GPT-5.6 Luna, with GPT-5.6 Terra judging the answers against the rubrics. He reports removing external information access, including documentation and web search, to avoid measuring the agent’s ability to look up answers instead of its unaided capability.

The reported results were:

Those percentages describe performance on this particular set of release-derived tasks. They should not be read as the probability that the model will complete an ordinary Dev Proxy or SPFx development assignment successfully. Task selection, prompt wording, scoring criteria, and the restricted environment all define what was measured.

The more useful finding was the distribution of successes. According to Mastykarz, Dev Proxy 0.3.0 tasks passed four out of five times, while the tasks for 0.4.0 all failed. Much newer versions again produced some passes. SPFx showed the same broad pattern: correct and incorrect answers appeared throughout its release history rather than separating neatly at one version.

Why a cutoff cannot establish product competence​

Mastykarz reports that OpenAI lists February 16, 2026, as GPT-5.6 Luna’s knowledge cutoff. His account identifies Dev Proxy 2.1.0, released January 28, as the latest stable version available before that date.

Yet when asked to estimate its own knowledge boundary, the model named approximately Dev Proxy 0.29.0 and SPFx 1.21.1. Mastykarz dates Dev Proxy 0.29.0 to June 25, 2025—almost eight months before the stated cutoff.

That answer was itself an unreliable measuring instrument. Multigrid’s explanation of knowledge cutoffs makes the same methodological point: asking a model for its cutoff produces a generated answer, not a measurement of its training corpus. The response may reflect provider instructions, fine-tuning material, or a guess.

A published cutoff describes the potential recency of training information. It does not promise comprehensive coverage of every product before that date, nor reliable retrieval and application of every fact the model encountered.

Two failures in Microsoft’s experiment illustrate the practical difference:

• According to Mastykarz, an SPFx 1.11.0 task asked about using a third-party module loader in the local workbench. The model discussed AMD, webpack, and iframe isolation, but missed the required version-specific fact: SystemJS support had been removed from that workbench.
• For a Dev Proxy 0.5.0 task, Mastykarz reports that the model supplied devproxy as the executable name. The rubric required mgdp, the historical command from when the product was called Microsoft Graph Developer Proxy.

Both answers could sound technically informed while failing the actual task. The executable example also shows why supplying only the newest documentation is not automatically enough: a task targeting an older version needs the behavior and naming appropriate to that version.

What the post-cutoff passes establish​

Mastykarz reports that the model passed one of two tasks for each of three later Dev Proxy releases: 2.3.4, dated April 23, 2026; 3.0.0, dated May 28; and 3.1.0, dated July 1. All three dates follow the February 16 cutoff cited in his report.

These are small samples, and the successful answers do not establish that the model had those released versions represented in its training data. Mastykarz identifies several possible explanations: inference from older functionality, transfer from related technologies, general programming knowledge, or a correct guess. An idea associated with a release might also have been public before that release shipped.

A correct answer demonstrates task success, not where the answer came from. Conversely, a wrong answer cannot by itself distinguish missing training material from a failure to recall or apply something that was present.

The results remain a single reported experiment, rather than an independently reproduced benchmark. Its tasks and rubrics were model-generated, and another model graded the outputs. That makes rubric accuracy and judge reliability important when interpreting the exact pass rates. The observed pattern supports workload-based evaluation; it does not establish a universal performance score for the model or reveal the contents of its training data.

How to evaluate an agent extension instead​

For a team using Dev Proxy in Microsoft Graph development, the useful question is whether the agent can complete the team’s actual tasks—and whether additional information improves the outcome.

Mastykarz’s June guidance, “How to measure AI agent extension effectiveness,” provides a concrete starting procedure:

1. Choose three to five representative scenarios. Use tasks developers actually perform, with a starting workspace that matches the work. Avoid rewriting prompts to advertise the extension or guarantee that the agent selects it.
2. Define specific acceptance criteria. Check correct use of the required API, executable, or behavior rather than merely checking whether its name appears. For generated projects, include build-and-run verification.
3. Run a baseline without the extension. Record the model, agent environment, prompt, workspace, and available tools so that the comparison has a stable starting point.
4. Repeat with the extension available, keeping the other conditions unchanged. This isolates the effect of adding documentation, instructions, or tools.
5. Run each scenario five times in each configuration. Compare the pattern of outcomes rather than treating a single success or failure as decisive.
6. Compare both correctness and resource use. Record prompt and completion tokens, tool calls, and turns alongside task results.

Only the first step requires selecting the workload; the rest makes the comparison interpretable. If an extension improves correctness consistently but greatly increases token consumption, the team can assess that cost. If it gets invoked without improving the output, invocation alone is not evidence of value.

The evaluation environment also needs to match the question. An experiment about unaided model knowledge requires strict information boundaries: Mastykarz warns that even a local product installation can expose answers. A practical extension evaluation, by contrast, can retain the normal workspace in both configurations because the question is whether the extension helps within that environment.

For Windows development teams, the June guidance specifically warns against evaluating a Linux command-line agent and assuming the results describe developers using Visual Studio Code on Windows. The operating system, agent environment, and available tooling can change the behavior being measured.

Verify the grading before trusting the score​

Microsoft’s evaluation guidance recommends checking model judges against outputs whose correct verdicts are already known. Teams should also grade identical outputs repeatedly to see whether the judge returns consistent results.

A rubric such as “uses the correct SDK” is too loose if a comment mentioning that SDK can pass. The criterion needs to require actual, correct use. Where the task produces runnable code, a plausible explanation or favorable model judgment should not replace compilation and execution checks.

For extension builders, this produces a more defensible documentation strategy than drawing a line through release history. Target the gaps demonstrated by the workload, preserve the product version each task requires, and rerun the same tests after adding context. Keep the cutoff as a recency signal; use measured task performance to decide what the agent needs.