Microsoft Debug-Gym: AI Coding Agents Use pdb, SWE-Bench Results and Windows WSL 2 Setup

To break that cycle of blind trial and error, researchers at Microsoft Research Montréal introduced debug-gym, an open-source, text-based interactive debugging framework designed for Python programs. Highlighted on the Microsoft Research project page and detailed in an arXiv technical report authored by Xingdi Yuan, Marc-Alexandre Côté, and colleagues on March 27, 2025 (corresponding to version 1.0.0 of the project on GitHub), debug-gym gives LLM-based coding agents access to interactive runtime tools—most notably Python's built-in debugger, pdb. Instead of just guessing why a test failed, an agent can pause execution, inspect live variable states, and test hypotheses before writing a patch.

For software engineers and Windows developers experimenting with agentic workflows, the project offers a fascinating look at how interactive tool use changes AI code repair—along with a few sobering reality checks about model capability, benchmark hype, and local platform support.

Moving AI Agents Beyond Blind Code Rewrites​

Most automated code-repair loops follow a rigid pattern: execute the code, capture the terminal error message or test failure, and ask the model to generate a new version of the file. As Microsoft Research explains, that setup leaves the hidden semantic space behind the running code locked away. Static source code rarely tells the whole story when a program is dealing with dynamic types, deeply nested objects, or shifting runtime state.

debug-gym reframes code repair as an iterative, information-seeking loop. Rather than proposing a brand-new model architecture or claiming to have "solved" autonomous software engineering, Microsoft Research built a lightweight Python library and environment to encourage research into interactive agent behaviors.

Here is how the interaction cycle works under the hood:

• Observation and Action Loop: Built around the familiar Gymnasium reinforcement-learning paradigm (RepoEnv), an episode starts with env.reset() and progresses as the agent calls env.step(action). At each step, the agent receives a text observation describing the current environment and tool states, issues a text command, and gets back a fresh observation detailing what changed.
• Interactive pdb Inspection: When equipped with the pdb tool, an agent can set breakpoints, step through code execution line by line, navigate stack frames, inspect live variable values, and evaluate Python expressions directly inside a paused frame.
• Modular Tooling and Sandboxing: Agents can alternate between investigating a repository, running small test functions, editing files, and verifying patches. According to the technical report, the environment scales to repository-level tasks and uses configurable Docker containers—allowing developers to specify base images, environment variables, setup commands, and session startup scripts—to isolate code execution.

Section Summary: Instead of forcing LLMs to guess fixes from static code and stack traces alone, debug-gym wraps repositories in a text-based Gymnasium environment where agents can interactively run pdb, inspect live memory states, and test hypotheses inside Docker containers.

Inside the "Mini-Nightmare" Benchmark and Python Rounding Traps​

To test whether interactive debugging actually helps on compact, gnarly bugs, Microsoft Research created Mini-nightmare, a suite of ten intentionally buggy Python scripts averaging roughly 40 lines of code. These aren't trivial typo puzzles; they cover multithreaded race conditions, boundary errors, tricky condition coverage, string manipulation quirks, and complex data structures—the exact kinds of problems where human developers reach for a debugger.

A standout example on the Microsoft Research project page is the shopping_cart problem. The script implements a basic shopping cart class, but there is a catch: the original logic was written assuming Python 2 behavior, whereas the debug-gym experiments run in Python 3. In Python 2, the built-in round() function rounds half-way values away from zero. In Python 3, round() switched to round half to even (often called banker's rounding).

If an LLM only stares at the source code, the math looks completely fine. In Microsoft Research's trajectory demonstration using Claude 3.7 Sonnet, the agent uses pdb to pause the program, inspect the intermediate numbers, test its hypothesis about how Python 3 is rounding the cart totals, and then import an external library to ensure the rounding behavior matches expectations. It is a textbook illustration of why checking runtime state beats staring at static text.

Section Summary: The Mini-nightmare benchmark tests agents on ten short, deceptive Python bugs, such as a Python 2-to-3 round() discrepancy where interactive pdb inspection helps an agent spot runtime behavior that looks innocent in static code.

Does a Debugger Automatically Make an AI Smarter?​

If you hand a powerful debugger to an LLM, does its code-fixing accuracy automatically skyrocket? According to the data in Microsoft Research's technical report, the honest answer is: it depends heavily on the model, the task, and when you hand over the debugger.

The researchers evaluated models across three distinct setups:

1. Rewrite-only: The standard baseline where the agent edits code using only test output and error messages.
2. debug: The agent has full access to interactive pdb debugging right from the start.
3. debug(5): A delayed-access strategy where the agent starts in rewrite-only mode and only unlocks the pdb tool after failing five rewrite attempts.

Across three runs on both Mini-nightmare and SWE-bench-Lite (a widely used benchmark of 300 real-world GitHub repository issues), the results tell a nuanced story:

Look closely at those numbers and a fascinating pattern emerges. Stronger reasoning and coding models like Claude 3.7 Sonnet and OpenAI o1-preview saw substantial gains on SWE-bench-Lite as soon as pdb was introduced—o1-preview nearly tripled its success rate from 10.7% to 30.2%.

However, giving immediate pdb access to GPT-4o on SWE-bench-Lite or Llama 3.3-70B-Instruct on Mini-nightmare actually lowered their scores compared to simple rewriting! Why would a helpful tool make an agent perform worse? Based on the researchers' findings, minimally designed agents built on less capable models don't always know how to use a debugger strategically; given a complex tool on turn one, they can get lost stepping through frames instead of fixing obvious issues. When the researchers held pdb back until the fifth failed attempt (debug(5)), every single model listed above beat its rewrite-only baseline. Let the agent try the easy fixes first, and bring out the heavy machinery only when it is genuinely stuck.

Furthermore, on the simpler, function-level Aider benchmark, adding interactive debugging showed no clear overall advantage. Not every coding problem requires runtime archaeology. The authors also candidly note that their primary evaluation metric—whether the final patch passes the unit test suite (or runs without error if no tests exist)—does not guarantee code efficiency or catch every performance bottleneck.

Section Summary: Experimental benchmarks show that while interactive pdb access boosts strong models like Claude 3.7 Sonnet and o1-preview on SWE-bench-Lite, giving the debugger immediately to other models can hurt performance—making delayed tool access (debug(5)) the most consistently effective strategy.

What Windows Developers and IT Pros Need to Know​

If you are working on a Windows 11 workstation and itching to spin up debug-gym to test your own local coding agents, there is a major platform caveat you need to know before touching your terminal.

According to the project's GitHub documentation, while debug-gym is installable via PyPI (pip install debug-gym) or a local repository clone and requires Python 3.12 or later, its interactive pdb tool currently works only on Linux. Because debug-gym relies on interactive pseudo-terminal (PTY) sessions inside Docker containers to communicate with pdb, and those PTY sessions are not fully supported in Docker on Windows or macOS, running the debugger natively on a standard Windows host is a no-go.

For Windows developers, the practical path forward is straightforward:

• Use WSL 2 or a Linux VM: Run your Python 3.12+ environment and Docker engine inside Windows Subsystem for Linux 2 (WSL 2) with a full Linux distribution, or target a remote Linux dev container or cloud VM.
• Treat Containers as a Design Feature, Not a Magic Shield: While debug-gym isolates target repositories in Docker containers, enterprise IT administrators and developers should still treat autonomous code execution with standard security hygiene, especially when connecting external LLM APIs to proprietary codebases.
• Keep the Dates in Perspective: While Microsoft Research updated and published its project showcase page in October 2026, the underlying technical report (arXiv:2503.21557) and v1.0.0 baseline date back to March 2025. Developers pulling the code from GitHub should check the repository's CHANGELOG.md to see what API refinements or tool additions have landed since the initial paper.

Ultimately, debug-gym is a breath of fresh air in AI software engineering research precisely because it doesn't pretend that bigger context windows will magically solve every bug. By giving agents a standard command-line debugger—and proving both when that tool shines and when it trips models up—Microsoft Research has handed developers a practical sandbox for building smarter, more methodical coding assistants. Just remember to fire up WSL 2 before you set your first breakpoint.