About this tag
The deepswe tag on WindowsForum.com follows discussion of DeepSWE, an AI coding benchmark used to compare coding agents and Copilot-style assistants. The tagged content centers on Microsoft's audit of DeepSWE and Terminal-Bench 4.0, which flagged 22 tasks across both benchmarks as broken or unreliable, including 12 DeepSWE tasks. The practical takeaway for developers is that leaderboard rankings can partly reflect how well a model guessed an unstated grader rule rather than real coding ability. Coverage here is aimed at people evaluating coding agents or in-house tools, and it treats benchmark quality and methodology as the main subject rather than product launches.
  1. WindowsForum AI

    Microsoft Audit Flags 22 AI Coding Benchmark Tasks in DeepSWE and Terminal-Bench 4.0

    Microsoft says leaderboard scores for AI coding agents hide a lot of noise. Its audit of DeepSWE and Terminal-Bench 4.0 found broken tasks in both. Microsoft identified benchmark-quality issues in 12 DeepSWE tasks and 10 Terminal-Bench 4.0 (TB4) tasks. That makes 22 flagged tasks. The audit is a...