If you build, ship or support AI features on Windows PCs, this is the clearest public statement yet of how Microsoft is thinking about the problem. It covers why a model that is "portable" can still run slowly, and where the DirectX Compute Graph Compiler (DXCGC) and Windows ML fit.
The core problem: portable or fast, rarely both
Microsoft's framing starts with a familiar pattern. Vendors ship their own stacks to deploy models on their silicon, and frameworks bring their own compilers, such as PyTorch's torch.compile. MLIR is also in the mix. The authors say each of these serves a particular niche, but few deliver high performance while staying independent of both hardware vendor and framework.
A neural network is a graph of mathematical operators. The obvious approach is to map each operator to its own kernel, such as a shader. The authors call this intuitive but slow to reach the market and very hard to scale as hardware and models keep changing. Their central claim is that individual operator speed is often not the main bottleneck. The harder part is running whole models efficiently across a wide range of hardware.
Why "AI TOPS" don't predict real speed
Matrix multiply-accumulate (MMA) sits underneath dense layers, MLPs, CNNs and transformers. GPUs and NPUs have had dedicated MMA hardware for years. The post says that having a fast instruction is not enough. High-level abstractions give portability, but they usually don't reach hardware efficiency by themselves.
The post then gives a rough expectation-versus-reality example:
- You take a model's operation count and divide it by the hardware's advertised "AI TOPS", and expect that runtime.
- In practice, the authors say you may often reach only 5–10% of the expected throughput.
- They attribute the gap to the ratio of compute to memory access. High-end GPUs need hundreds of operations per byte of DRAM traffic to stay busy.
- They say processing a network layer by layer can hardly reach that ratio, which is why operator fusion matters.
Treat the 5–10% and "hundreds of operations per byte" figures as Microsoft's general illustration. The post names no model, hardware or test setup, so they are not benchmark results.
The practical takeaway for buyers and developers is that a headline TOPS number says little about delivered performance. Memory movement and how well the compiler fuses operators matter at least as much.
Fusion is harder than it sounds
The post says fusion is not just merging two adjacent element-wise operators. Useful fusions have to span heavyweight operations, including matrix multiplication, convolution, reduction and upsampling. Fusion and decomposition also don't map neatly onto kernel boundaries. The authors treat them as part of a larger question about how control and information move between the layers of a compiler.
Device-agnostic versus device-aware optimization
The authors separate two kinds of graph rewrites:
- Device-agnostic rewrites standardize and optimize common operator patterns. They can be done at the framework level.
- Device-aware rewrites, such as decompositions and fusions, are driven by the device's architecture and instruction set (ISA). The authors define the ISA as the operations a device can run without an extra trip to memory.
Pattern matching is one practical way to do both, the post says, but not the only one. The authors call infrastructure that supports both kinds of optimization an important early step for compilers that need to scale across hardware.
Multiple layers, split responsibility, late surprises
A serious cross-device stack ends up spanning several IRs and abstraction levels. Sometimes a backend needs a large graph so it can reason about end-to-end execution. Sometimes you carve out a subgraph where hardware can do something special. Sometimes you need tile- or kernel-level control.
The post argues that the fragile part is how responsibility is divided. If an upper layer assumes the lower layer will optimize things, outcomes become hard to predict. The upper layer then can't tell which parts of the graph will stay on fast paths and which will hit slow paths, fallbacks or late reshaping.
Late discoveries are expensive. The authors list the kinds of rework they can force for each hardware target:
- hand-editing models
- changing how the graph is partitioned
- reordering transformations
- changing tiling and layout
They say static primitive granularity combined with greedy partitioning is often not enough. They want a richer cooperative interface that shares graph-level visibility and control across layers.
Anyone who has hit an ONNX model that silently falls back to the CPU for one unsupported operator will recognize this "you only find out at the end" pattern. That comparison is my own illustration, not something from the post.
Scheduling and memory: client is not server
Layout, splitting and streaming, cache use, data formats, tiling and compute stencils are all interconnected. They also depend on hardware details such as MMA sizes, on-chip memory, register-file limits, and synchronization across threads, waves, thread groups and clusters.
The post contrasts two deployment settings:
| Setting | Example constraints from the post |
|---|---|
| Client real-time inference | Batch size 1, latency targets around 2 ms, so shuttling lots of data through DRAM is impractical |
| Server inference | High-bandwidth memory (HBM) and larger batches shift the constraints |
These are illustrative scenarios, not requirements. The authors compare running complex networks efficiently to building a high-performance rendering engine, and say it takes far more than writing a single compute shader.
Where Microsoft says the answer lies
The post sorts existing approaches into two groups:
- Frontend or framework-level solutions are easy to use and often fast. The authors say they can struggle with cross-device consistency, unified tooling, potentially unbounded compile times, limited low-level control and graphics interoperability.
- Operator-level solutions work well with graphics and low-level engines and are generally vendor-independent. The authors say they take more effort to use and raise scalability and performance concerns compared with vendor-specific options.
Microsoft's view is that the answer likely sits between the two. It says it sees an opportunity for Windows ML and DirectX to help, and that it is exploring how DXCGC, Windows ML and a multi-level IR architecture could give developers more consistent and portable AI on diverse hardware. "Exploring" is the operative word. The post announces no shipping product and gives no performance data for the combined approach.
How this fits what Microsoft has already said
Microsoft introduced DXCGC at GDC in March 2026. Its DirectX blog described it as a new ML compiler API for executing full model graphs. Models flow in from modern frameworks, and DirectX analyzes and specializes the graph for a given device. It then lowers the graph into workloads that integrate with D3D12 queues and command lists. The stated benefits were:
- automatic graph optimization, memory planning and operator fusion
- portable performance across hardware vendors
- unified PIX tooling
Microsoft positioned it for larger models, with HLSL Linear Algebra covering small inline shader workloads. At that time it said DXCGC would be available for private preview this summer, by contacting a Windows representative. Coverage from AMD's GPUOpen blog also tied the compiler to Windows MLIR.
Public sources I found only restate the planned summer private preview. I found nothing confirming general availability or measured results, and the November post doesn't claim either.
What Windows developers can use today
For shipping apps, Microsoft's Windows ML documentation describes the current route. Windows ML includes CPU and legacy DirectML providers. Other execution providers are downloaded on demand through the ExecutionProviderCatalog APIs on Windows 11 24H2 (build 26100) or later, depending on device and driver compatibility. The documented providers include:
- AMD GPU
- NVIDIA TensorRT RTX
- Intel OpenVINO
- Qualcomm QNN
- AMD VitisAI
- an experimental WebGPU provider
The documentation marks AMD's MIGraphX provider as deprecated in Windows ML 2.x, with AMD recommending a move to the AMD GPU provider.
DirectML, meanwhile, is documented as being in maintenance mode. Microsoft suggests Windows ML for PCs on Windows 11 24H2 or later. DirectML remains supported on DirectX 12-compatible hardware.
The execution-provider model is a pragmatic answer to the same fragmentation the blog post describes. It does not, however, prove that the unified multi-level compiler is already in place.
What to watch
- Ignite BRK388: the post says this session will go deeper on multi-level IRs and local AI on Windows. I couldn't verify a session date or details, so check the Ignite schedule.
- DXCGC availability: watch for preview status, supported hardware and any published benchmarks.
- Relationship to Windows ML and DirectML: the post positions DirectX and Windows ML as complementary, but it does not say how DXCGC would relate to existing execution providers.
- Vendor buy-in: AMD, Intel, NVIDIA and Qualcomm all offered supportive statements in March. Real portability will depend on driver and compiler-layer cooperation.
Bottom line
This is a candid engineering essay. Peak performance on a given NPU or GPU comes from deep, hardware-specific compiler work, while portability requires abstraction, and the hard part is making the layers cooperate rather than guess. Microsoft's proposed middle path is promising but unproven in public. The numbers it cites are illustrative, and DXCGC is still a direction rather than a product anyone can ship against. For now, developers should keep optimizing their own models and testing on each target hardware family, since Microsoft's own documentation says Windows ML handles provider distribution, not model optimization.