Azure GB300 NVL72 Rack Scale AI with 4608 GPUs for Inference

Microsoft Azure has quietly deployed what both vendors call the world’s first production-scale GB300 NVL72 supercomputing cluster, linking more than 4,600 NVIDIA Blackwell Ultra GPUs into a single, rack-first fabric intended to accelerate reasoning-class inference and large-model workloads for OpenAI and Azure AI customers.

Background / Overview​

The announcement marks a deliberate shift in cloud AI infrastructure design: treat the rack as the fundamental accelerator, not the individual server. Microsoft’s new ND GB300 v6 virtual machines are the cloud-exposed face of a liquid-cooled, rack-scale appliance (the GB300 NVL72) that pairs 72 Blackwell Ultra GPUs with 36 NVIDIA Grace-family CPUs and a pooled “fast memory” pool to present each rack as a single, tightly coupled accelerator. Microsoft and NVIDIA say this production cluster stitches roughly 64 such NVL72 racks—arithmetically consistent with 64 × 72 = 4,608 GPUs—into a single Quantum‑X800 InfiniBand fabric, delivering what vendors describe as supercomputer-scale inference and training capacity.
This feature unpacks what the hardware actually is, verifies the most important technical claims where possible, evaluates likely performance and operational trade-offs, and explains what this means for enterprises, developers, and the Windows ecosystem as AI workloads move from single‑GPU instances toward rack-as-accelerator deployments.

---

Technical anatomy: what’s inside a GB300 NVL72 rack​

Rack as a single accelerator​

At the core of the GB300 NVL72 design is the intent to make a whole rack behave like one massive accelerator. Each NVL72 rack is described by vendors as containing:

• 72 × NVIDIA Blackwell Ultra (GB300) GPUs.
• 36 × NVIDIA Grace-family Arm CPUs (co‑located for orchestration and memory services).
• A pooled “fast memory” envelope in the tens of terabytes (vendor materials generally cite ~37–40 TB).
• A fifth-generation NVLink switch fabric delivering on-the-order-of-130 TB/s intra-rack bandwidth.
• Liquid cooling and facility plumbing sized for extremely high thermal density.

Treating the rack as an accelerator reduces cross-host copy overheads and lets key-value caches and working sets for transformer-style models remain inside a single high-bandwidth domain—critical for reasoning models and very long context windows.

Memory composition and pooled fast memory​

Microsoft and NVIDIA describe the rack’s pooled “fast memory” as a roughly 37‑terabyte envelope that’s the sum of GPU HBM and Grace CPU-attached memory. Published vendor breakdowns indicate something like:

• ~20 TB HBM3e (aggregate across GPUs) and
• ~17 TB LPDDR5X (Grace CPU-attached, used as part of the pooled addressable working set).

The vendors emphasize that NVLink/NVSwitch technology presents this combined memory as a high-throughput domain so model shards and KV caches can be remoted inside the rack with much lower latency than traditional PCIe-hosted architectures. That pooled memory figure appears consistently in vendor and partner briefings, though exact configurations may vary across deployments.

Compute: how to read the PFLOPS claims​

Vendor material quotes the GB300 NVL72 rack as capable of up to roughly 1,400–1,440 PFLOPS of FP4 Tensor Core performance for the full 72‑GPU domain. It’s critical to interpret this carefully:

• These figures are quoted for FP4 tensor core metrics (low-precision formats optimized for inference), not for full double-precision or typical CPU-style FLOPS.
• Peak PFLOPS claims depend heavily on numeric format (FP4, FP8, sparsity options) and software stack support; sustained throughput on real models will be lower and highly workload-dependent.
• Some publications conflate “PFLOPS” with “exaflops” in round numbers; the correct vendor figure for the rack domain is presented as ~1,440 PFLOPS (i.e., 1.44 × 10^3 PFLOPS, often contextualized as 1.44 exaFLOPS in FP4—which is a precision-specific qualification).

Flag: treat peak PFLOPS as a vendor-rated upper bound for a specific precision and benchmark mode, not an automatic indicator of real-world model throughput.

NVLink v5 / NVSwitch: intra-rack fabric​

Inside each NVL72 rack, NVIDIA’s fifth-generation NVLink / NVSwitch fabric is used to form an all-to-all NVLink domain among the 72 GPUs and 36 Grace CPUs. Vendors report a combined intra-rack NVLink bandwidth on the order of 130 TB/s, which is the primary ingredient that allows GPUs inside the rack to behave like slices of a single accelerator. That intra-rack coherence is essential to reduce synchronization overheads for attention layers and AllReduce-style operations.

Quantum‑X800 InfiniBand: stitching racks into a supercluster​

To scale beyond a single rack, Microsoft’s deployment uses NVIDIA’s Quantum‑X800 InfiniBand fabric and ConnectX‑8 SuperNICs. Microsoft and NVIDIA state that Quantum‑X800 provides ~800 Gb/s class rack-to-rack bandwidth per GPU-equivalent link, and that Azure intentionally deployed a fat-tree, non-blocking topology with in-network compute features (SHARP v4 offload) to preserve near-linear scaling as workloads span hundreds or thousands of GPUs. These network-level offloads and telemetry-driven congestion control are as important to multi-rack scaling as raw per-GPU performance.

---

What Microsoft actually deployed (claims versus arithmetic)​

• Microsoft publicly positioned the rollout as a single production cluster containing “more than 4,600” Blackwell Ultra GPUs. NVIDIA’s NVL72 definition (72 GPUs per rack) makes a neat arithmetic fit: 64 racks × 72 GPUs = 4,608 GPUs. That appears to be the deployment arithmetic Microsoft and partners are using to ground the “more than 4,600” claim.
• Vendor materials align on the ND GB300 v6 VM family as the cloud-facing unit built from these racks, aimed at OpenAI-scale inference and reasoning workloads. Microsoft says the fleet is already dedicated to the heaviest OpenAI inference tasks.

Caveat: vendor “first” claims and GPU counts should be treated as vendor-provided statements until independently audited inventory or third‑party telemetry is published.

---

Performance: benchmarks, claims, and real-world meaning​

MLPerf and vendor-submitted numbers​

NVIDIA and partners submitted GB300 / Blackwell Ultra results to MLPerf Inference that show notable gains on reasoning-oriented workloads and large-model inference scenarios. Vendors attribute the highest gains to a combination of:

• Hardware (expanded NVFP4-friendly tensor cores; more HBM3e per GPU),
• Software (inference compilers, runtime optimizations), and
• Architecture (pooled memory and NVLink coherence that reduce cross-host transfers).

These benchmark submissions establish directionally that the GB300 generation delivers higher tokens-per-second and better inference efficiency versus previous generations.

From peak PFLOPS to usable throughput​

Benchmarks are directional; production performance is bounded by many real-world constraints:

• Model architecture and tokenizer behavior.
• Batch size, latency SLAs (tail latency matters for interactive agents), and cold-start patterns.
• Data ingestion and storage throughput; GPUs cannot help if I/O or preprocessing stages are bottlenecks.
• Software maturity around new numeric formats (FP4/NVFP4) and operator support in frameworks that power LLM serving.
• The impact of sparsity, quantization, and compiler/runtime optimization on accuracy/performance trade-offs.

Vendors’ “months to weeks” training-time reductions and “support for hundreds-of-trillions‑parameter models” are plausible in ideal configurations with optimized stacks—but they are not universal guarantees. Each workload must be validated on the stack to estimate real-world gains.

---

Operational engineering: facilities, cooling, power and networking​

Liquid cooling and datacenter changes​

Dense rack configurations with 72 GPUs and co-located CPUs drive extreme thermal density. Microsoft’s deployment is liquid-cooled and uses dedicated heat exchangers and facility loops to minimize water usage. The engineering effort touches every datacenter layer:

• Power distribution rework to support higher per-rack power draws and redundancy.
• Chilled water or liquid-loop plumbing for heat rejection at pod scale.
• On-site transformers, breakers and capacity planning to deliver multi-megawatt pods reliably.
• Maintenance and safety processes adapted to liquid-cooled gear.

These facility changes are non-trivial capital and operational investments—far beyond buying commodity servers.

Networking: topology-aware scheduling and orchestration​

To get full value from NVL72 racks and pod-scale fabrics, schedulers and orchestration stacks need to be topology-aware. Key changes include:

• Placement policies that respect NVLink domains and avoid unnecessary cross-domain hops.
• Collective-aware orchestration that maps AllReduce and AllGather onto SHARP-enabled paths.
• Telemetry-driven congestion control integrated into jobs to avoid noisy-neighbor effects that kill scaling efficiency.
• Storage and IO systems sized to feed GPUs at multi-GB/s rates so accelerators aren’t IO-starved.

---

Business and strategic implications​

For Microsoft and OpenAI​

This cluster underlines the depth of the Microsoft–NVIDIA–OpenAI co-engineering triangle: Microsoft hosts and operates the scaled GB300 fabric; NVIDIA supplies the chip, NVLink, and InfiniBand fabric; OpenAI is listed as a primary consumer. The deployment serves both as a capability demonstrator for Azure’s AI services and a practical platform for OpenAI’s production inference. Microsoft frames these GB300 clusters as the first of many “AI factories” intended to scale across global datacenters.

For cloud competition and industry concentration​

Rack-first superclusters raise questions about vendor and cloud concentration. Building and operating GB300 NVL72 pods requires:

• Deep vendor relationships (chip supply, fabric provisioning).
• Large capital investments in facility modernization.
• Highly specialized operational expertise.

That raises the barrier to entry and tends to concentrate frontier AI infrastructure among the largest cloud providers and a few hardware suppliers—an industry trade-off between capability and centralization.

---

Risks, unknowns, and what to watch​

• Vendor-quoted peak numbers vs. production reality. Peak PFLOPS and “hundreds-of-trillions” model claims depend on precision, sparsity, and software optimizations. Treat peak numbers as directional, not guaranteed.
• Lock-in and portability. The rack-as-accelerator model relies on NVLink/NVSwitch coherence and in-network compute features that are tightly coupled to NVIDIA’s stack. Moving workloads to different hardware or hybrid environments will likely require significant reengineering.
• Cost and utilization. High capital and operating costs mean ROI depends on strong utilization and carefully priced service agreements. Enterprises need clear SLAs, cost-per-inference models, and fallback options.
• Supply-chain and geopolitical risk. Large-scale procurement of next-gen accelerators concentrates demand and may be sensitive to supply-chain disruptions or export controls.
• Environmental and site-level constraints. Liquid-cooling and power upgrades impose local constraints on where these clusters can be deployed. Not every Azure datacenter will be able to host NVL72 pods without significant upgrades.

---

What enterprises, developers, and Windows ecosystem partners should do now​

Immediate checklist for IT and AI teams​

• Profile workloads for topology sensitivity: quantify how much communication and memory-bound your models are and whether a rack-first domain benefits them.
• Demand topology-aware SLAs from cloud vendors: ask for predictable tail-latency, availability, and cost-per-token metrics.
• Plan for portability: maintain model checkpoints and fallback deployment paths to alternative hardware or lower-cost instances.
• Invest in toolchains that support NVFP4, compiler optimizations, and Collective-Aware schedulers if you plan to target GB300-class infrastructure.
• Include facility constraints in procurement: ask about cooling, location, and regional availability if you’re purchasing reserved capacity.

For Windows ecosystem ISVs and OEM partners​

• Re-evaluate desktop-to-cloud workflows: expect new server-side capabilities (longer contexts, faster reasoning) to shift where inference runs.
• Update deployment guides and performance testing harnesses to include topology and precision-aware metrics.
• Build integrations that can transparently use ND GB300 v6 instances for heavy inference while falling back to smaller GPU classes for cost-sensitive workloads.

---

Strengths: why this architecture matters​

• High per-rack memory and bandwidth dramatically reduce the friction of sharding very large models, enabling longer context windows and larger KV caches.
• NVLink/NVSwitch intra-rack coherence shifts the balance from network-bound to compute-bound for many reasoning workloads.
• Quantum‑X800 fabric and in-network compute primitives enable more efficient multi-rack scaling for synchronous collectives.
• Vendor-validated benchmark gains demonstrate directionally better tokens/sec and efficiency for targeted inference workloads.

---

Weaknesses and trade-offs​

• Vendor specificity creates portability challenges; code and models optimized for NVFP4/NVIDIA Dynamo may not run equivalently elsewhere.
• Operational complexity (liquid cooling, power, fat-tree fabrics) increases CAPEX/OPEX and requires specialized teams.
• Unproven long-tail performance for arbitrary customer workloads; benchmarks are positive but not determinative.
• Market concentration risk as the largest providers and vendors consolidate next-gen AI infrastructure.

---

Conclusion​

Microsoft’s GB300 NVL72 production cluster on Azure is a milestone: it operationalizes a rack-as-accelerator design that combines tens of terabytes of pooled fast memory, 130 TB/s NVLink intra-rack bandwidth, and Quantum‑X800 InfiniBand scale‑out to present a supercomputer-class surface for reasoning‑focused inference and large‑model training. The deployment aligns with vendor MLPerf submissions and Microsoft’s ND GB300 v6 product framing, and the arithmetic behind the “more than 4,600 GPUs” claim is consistent with 64 NVL72 racks (64 × 72 = 4,608).
That said, the bold performance headlines must be read with discipline: PFLOPS claims depend on numeric format and sparsity; MLPerf and vendor benchmarks are directional; and the real value for any given customer depends on topology-aware engineering, software maturity, and cost‑utilization trade-offs. Organizations planning to use ND GB300 v6 capacity should demand transparent SLAs, run topology-aware profiling, and prepare for vendor-specific software stacks while negotiating fallback options and portability strategies.
The era of rack-scale, NVLink-dominant “AI factories” is operational—and Azure’s GB300 NVL72 installation shows the path forward. The practical benefits are substantial for workloads that match the architecture; the commercial and operational trade-offs are equally material. IT leaders and developers must balance ambition with engineering rigor to turn vendor promise into predictable, sustainable production capability.

---

Source: Tom's Hardware Microsoft deploys world's first 'supercomputer-scale' GB300 NVL72 Azure cluster — 4,608 GB300 GPUs linked together to form a single, unified accelerator capable of 1.44 PFLOPS of inference