Azure ND GB300 v6 Delivers 1.1M Tokens/sec Inference

Microsoft’s new ND GB300 v6 virtual machines have cracked a milestone that changes the practical limits of public‑cloud AI inference: one NVL72 rack of Blackwell Ultra GPUs sustained an aggregated throughput of roughly 1.1 million tokens per second, a result validated by an independent benchmark lab and reported by multiple outlets.

Background / Overview​

Microsoft packaged NVIDIA’s latest GB300 NVL72 rack into the Azure ND GB300 v6 VM family and used an MLPerf‑style Llama 2 70B inference setup to demonstrate this throughput. The test runs aggregated 18 ND GB300 v6 VMs inside a single NVL72 rack (72 GB300 GPUs + 36 NVIDIA Grace CPUs in the rack domain) and reported an aggregate of ~1,100,948 tokens/sec; that aggregates to about 15,200 tokens/sec per GPU. The measurement and analysis were publicly discussed by Signal65 and covered by industry press. This result is being framed inside the industry as a generational leap: Microsoft’s ND GB200 v6 family previously set a high bar (reported ~865,000 tokens/sec on a GB200 NVL72 rack), and the GB300 result documents a meaningful uplift over that baseline. Both Microsoft’s posts about ND GB200 and the new GB300 briefings are part of the verification fabric for these claims.

---

Why the million‑token milestone matters​

Tokens as the currency of inference​

Tokens are the operational unit for LLM workloads: models tokenize inputs and generate tokens one or more at a time. For enterprises, tokens per second (TPS) is now a first‑order performance metric because it maps directly to user concurrency, latency, and — increasingly — cost. Billing models and capacity planning use token consumption to size systems, so raising sustained TPS fundamentally alters cost and scale calculations for conversational agents, retrieval‑augmented generation (RAG) pipelines, and multi‑step agentic systems.

What 1.1M tokens/sec buys an operator​

• Support for thousands of concurrent interactive users or very high‑frequency batching for summarization and long‑context reasoning.
• Fewer racks required to reach the same QPS (queries per second) for a given service level — reducing scheduling complexity for the largest models.
• The practical ability to host longer context windows or larger KV caches inside a single coherent domain, improving throughput for reasoning workloads.

These outcomes are conditional: they rely on the same optimizer, numeric formats, and orchestration tricks that the vendor used in the MLPerf‑style run. Expect real‑world, end‑to‑end throughput to be lower once storage, retrieval, and network tails are included.

---

The technical anatomy: how Azure delivered 1.1M tokens/sec​

Rack‑as‑accelerator architecture​

The GB300 NVL72 rack treats the entire rack as a single coherent accelerator: 72 Blackwell Ultra GPUs paired with 36 NVIDIA Grace CPUs, pooled HBM/“fast memory” in the tens of terabytes, and a high‑bandwidth NVLink switch fabric inside the rack. This moves the bottleneck from cross‑host communication to intra‑rack coherence, enabling much larger working sets and lower synchronization overhead for attention‑heavy models.

Key hardware numbers reported​

• Aggregate throughput on the demonstrated rack: ~1,100,948 tokens/sec.
• Per‑GPU rough average: ~15,200 tokens/sec per GB300 GPU (1.1M ÷ 72).
• Intra‑rack NVLink bandwidth: ~130 TB/s (vendor‑published NVLink/NVSwitch figure for NVL72 configurations).
• Pooled fast memory per rack: ~37–40 TB reported in vendor materials and industry briefings.

These are vendor and benchmark‑submission figures; they align across Microsoft / NVIDIA documentation and independent analysis from benchmarking labs, but they depend heavily on precision modes, quantization, and optimized runtimes.

Software and numeric formats​

The GB300 results lean on software and numeral innovations such as NVFP4 (FP4‑style low‑precision formats), highly tuned inference runtimes (TensorRT‑LLM / NVIDIA Dynamo or similar compiler/serving stacks), and topology‑aware sharding that minimize cross‑device synchronization. Those software layers are as essential as the hardware to reach the reported tokens/sec.

---

Independent validation, benchmarks and caveats​

• Signal65 performed a verification analysis and published an assessment that highlights both the technical achievement and the enterprise implications; their write‑up confirms the 1.1M figure as observed in a sustained run and calls out its potential for regulated industries.
• The claim was reported widely in trade coverage and reproduced in MLPerf‑style submission summaries; NVIDIA’s own MLPerf and technical blog posts document Blackwell family gains in inference, reinforcing the direction and scale of improvement.

Important caveats:

• MLPerf‑style runs and vendor‑optimized submissions are directional: they prove what is possible under controlled, optimized conditions. They do not automatically translate to identical uplift for every model, prompt shape, tokenizer, or production pipeline.
• Precision tradeoffs (FP4/NVFP4, quantization, sparsity, etc. can affect model quality. Vendors report adherence to MLPerf accuracy thresholds, but end users must validate quality for their particular prompts and safety constraints.

---

Strengths — what this enables​

• Massive concurrent inference: A single rack that can push >1M tokens/sec lets SaaS and platform providers run very high‑QPS services or consolidate hundreds of smaller racks.
• Feasible long‑context reasoning: The pooled memory and NVLink coherence make long KV caches and extended contexts practically usable inside a single rack, which benefits chain‑of‑thought and multi‑step agents.
• Performance‑per‑watt improvements: The vendor and third‑party analyses claim meaningful gains in tokens/sec per watt relative to previous generations, improving the economics and sustainability profile at scale.
• Faster iteration cycles: For labs training and fine‑tuning very large models, the raw compute and rank‑scale fabric can shorten training wall times when combined with efficient pipelines.

---

Risks, tradeoffs and operational realities​

1) Benchmark vs. production delta​

Benchmarks are controlled scenarios. End‑to‑end applications include retrieval latencies, storage I/O, network tail latency, and multi‑tenant noise; those factors frequently halve or worse the headline tokens/sec in practice. Validate with your full pipeline.

2) Software and numeric lock‑in​

The throughput advantage depends on NVFP4 and vendor inference toolchains. That introduces potential portability and vendor‑stack lock‑in: migrating optimized FP4 stacks to other platforms or maintaining numerical parity across providers can be non‑trivial. Enterprises with regulatory or portability requirements must weigh this carefully.

3) Cost and procurement constraints​

High density racks consume significant power and cooling resources. List price for such racks is only one part of TCO: power, facility modifications (liquid cooling), specialized networking (Quantum‑X800 InfiniBand + ConnectX‑8), and support contracts are substantial. For many teams, renting managed ND GB300 v6 capacity will be the only realistic path — and that raises concerns about capacity availability and long‑term pricing.

4) Operational complexity​

To reliably exploit NVL72 coherence you need topology‑aware schedulers, placement guarantees (so a job grabs contiguous NVLink domains), storage that can feed GPUs at high sustained rates, and power/facility planning. This is not turnkey for most IT teams.

5) Governance, sovereignty and export controls​

Consolidating frontier compute into hyperscalers concentrates power and raises policy questions: data residency, regulated workloads, cross‑border export controls on advanced compute, and national security filters come into play when access to huge model training/serving capacity is centralized. Enterprises in regulated industries must verify compliance features and geographic availability.

---

Practical guidance for enterprise architects​

Validate, don’t assume​

• Run end‑to‑end pilots using your production prompt mix, retrieval pipeline, and client‑side latency budgets. Simulated MLPerf runs are useful signals, but they’re not a production SLA.
• Measure both Time to First Token (TTFT) and sustained Tokens Per Second (TPS) with real-world retrieval, caching, and multi‑tenant loads. Use the same tokenizer and generation settings your production models will use.

Tech checklist before committing​

• Confirm NVLink domain placement guarantees in your region and subscription.
• Validate that model quantization (FP4 or other) preserves acceptable output quality across your prompt set.
• Ensure storage and network bandwidth can sustain the GPUs (I/O starvation is a common practical bottleneck).
• Negotiate SLAs that cover availability and resource reservation for contiguous NVL72 domains if you plan to rely on consistent performance.

Cost/scale modeling​

• Model per‑request cost by pairing vendor‑reported tokens/sec with expected real‑world utilization, not vendor peak numbers.
• Consider a hybrid strategy: use ND GB300 racks for peak, low‑latency production traffic and less‑costly instances for background batch or fine‑tuning workloads.

---

Competitive and market implications​

• Hyperscalers that deploy GB300‑class racks gain a practical edge in offering extremely high QPS inference services, and that advantage compounds because large providers can also offer integration, compliance, and geographic reach.
• Specialized GPU clouds and “neoclouds” may respond with differentiated pricing or alternative accelerators, but the rack‑as‑accelerator model prioritizes memory and fabric bandwidth — areas where NVLink + in‑network compute meaningfully advantage Blackwell Ultra family designs.
• Expect continued MLPerf rounds and more independent validation runs; watch for third‑party studies that replicate vendor stacks in non‑vendor‑controlled testbeds.

---

What to watch next​

• MLPerf and independent labs running longer, application‑level tests (RAG, multimodal pipelines, agentic workflows) that include storage and retrieval in the loop. These will help quantify the real production delta.
• Availability of ND GB300 v6 SKUs across Azure regions and the pricing schemes Microsoft offers for provisioned throughput versus on‑demand use.
• Competitor responses: whether other clouds match GB300 (or offer alternative design wins such as TPUs or other accelerators) and how quickly they publish comparable MLPerf entries.

---

Final analysis and conclusion​

Microsoft’s ND GB300 v6 result — a sustained, validated rack‑level throughput of roughly 1.1 million tokens per second on Llama 2 70B — is a real and consequential engineering milestone. It demonstrates that the rack‑as‑accelerator design, paired with new numerical formats and inference compilers, can materially raise the throughput and efficiency ceilings for inference in the public cloud. Signal65’s validation and multiple vendor and press writeups corroborate the headline number and its immediate implications for enterprise‑grade AI deployments. At the same time, the path from benchmark to robust production value is nontrivial. Organizations must validate model quality under FP4/quantized regimes, account for storage and retrieval latencies, negotiate placement guarantees, and evaluate the TCO implications of running or consuming extremely dense rack‑scale instances. The ND GB300 v6 era broadens what’s technically possible; responsible adoption will require disciplined benchmarking, careful SLAs, and architectural work to ensure portability and governance. For teams planning to exploit this new capability, the sensible strategy is pragmatic: use ND GB300 v6 for workloads where the throughput and extended context materially change product capability (high‑concurrency chat, real‑time RAG, multi‑step agent orchestration), validate end‑to‑end performance with real pipelines, and build fallback paths for portability and cost management. The million‑token barrier is broken — the next challenge is turning that headline capability into repeatable, secure, and cost‑effective production services.

---

Source: SDxCentral Microsoft’s Blackwell Ultra VMs push AI performance past the million token milestone

 

[ChatGPT]

Microsoft’s Azure team has pushed a single rack‑scale system to an industry record of roughly 1.1 million tokens per second, using ND_GB300_v6 virtual machines built on NVIDIA’s GB300 (Blackwell Ultra) NVL72 rack — a headline milestone that proves rack‑scale inference at industrial throughput is now a reality, but one whose practical impact is far more nuanced than the press release suggests.

Background​

The Azure announcement describes a run of the MLPerf Inference v5.1 Llama 2 70B offline scenario across an NVL72 GB300 domain comprised of 18 ND_GB300_v6 VMs (72 GB300 GPUs in total). Microsoft reports an aggregate throughput of ~1,100,948 tokens/sec — about 15,200 tokens/sec per GPU — achieved using NVIDIA’s TensorRT‑LLM stack and FP4 quantization. The company published logs, replication steps and a detailed technical brief alongside the claim. MLCommons (MLPerf) released the Inference v5.1 benchmark set in 2025 and enumerated participating submitters and new models for the suite; it’s the framework Microsoft used to configure the workload. That same v5.1 release introduced new reasoning and interactive tests (including DeepSeek‑R1), which show the benchmark suite is evolving to match modern inference needs. NVIDIA’s GB300 NVL72 rack is a purpose‑built, liquid‑cooled rack platform that pairs 72 Blackwell Ultra GPUs with 36 Grace CPUs, very high NVLink/InfiniBand bandwidth and dramatically larger pooled HBM capacity compared with previous generations — specifications that deliberately optimize these racks for reasoning‑class inference. Independent system vendors and OEMs (GIGABYTE, Ingrasys and others) have documented GB300 rack characteristics that align with the Azure brief.

---

What Microsoft actually proved​

The measured numbers, in plain language​

• Aggregate throughput: ~1.1 million tokens/sec on a single NVL72 GB300 rack running a single benchmark scenario (MLPerf v5.1 Llama 2 70B, Offline).
• Per‑GPU throughput: ~15,200 tokens/sec (72 GPUs).
• Measured HBM throughput: Microsoft reports extracting 92% HBM efficiency and ~7.37 TB/s aggregate memory throughput from the rack during this run.
• Software stack: NVIDIA TensorRT‑LLM, model run at FP4 precision (quantized).

These are load‑bearing technical facts: a rack configured like this can process orders of magnitude more token throughput than prior H100‑based systems under the very specific conditions of the test. Multiple trade outlets and benchmarking observers restated the Microsoft brief shortly after publication.

Why these gains are feasible (hardware + software synergy)​

The GB300 NVL72 is not a generic GPU cluster — it’s a co‑engineered rack-scale system that increases GPU memory capacity, raises thermal/power limits, and stitches GPUs together with denser NVLink/InfiniBand fabrics to reduce cross‑device communication costs. Software advances — notably TensorRT‑LLM and FP4 quantization — amplify those hardware gains by lowering memory bandwidth and compute costs per token. The combination is what unlocks the per‑GPU ~15,200 tokens/sec figure Microsoft reports.

---

Why the milestone matters for enterprise AI​

Practical implications​

• Single‑rack feasibility: Workloads that previously needed multiple racks (or longer processing windows) to hit enterprise throughput targets can now be concentrated into a single NVL72 domain. That simplifies operational design for extremely high‑throughput batch inference or heavy concurrent workloads.
• Cost and margin impact: Every percentage point of throughput improvement translates into real dollars at hyperscale. For cloud providers and customers running millions of API calls, a 25–30% jump in per‑GPU throughput can materially lower cost per token or raise gross margins on inference services.
• Time‑to‑market leadership: Azure’s value here is partly productization: making GB300 NVL72 capability available as ND_GB300_v6 VMs and publishing reproduction instructions helps enterprises try it quickly and makes Azure a practical choice for those who need heavy inference capacity today. Microsoft’s transparency around logs and replication steps is a pragmatic product play, not simply marketing.

Who benefits first​

• Large conversational AI providers and enterprise customers running high‑concurrency chat, retrieval‑augmented generation (RAG), multimodal agents, or multi‑step reasoning pipelines.
• Organizations with heavy offline inference workloads (e.g., bulk content generation, large‑scale summarization, or nightly re‑scoring jobs) where batch throughput maps directly to cost reductions.

---

Important caveats and the limits of the headline​

The headline — “1.1 million tokens/sec” — needs context. There are several nontrivial limitations that temper a simple “Azure wins” narrative.

1) It’s an Offline/Batch benchmark, not an interactive latency guarantee​

The run Microsoft describes is the MLPerf Offline scenario: optimized for sustained throughput, not time‑to‑first‑token or low tail latency under multi‑tenant, mixed workloads. Real‑world conversational services prioritize low latency and consistent concurrency handling (many small requests), not just bulk token throughput. The Azure result does not directly prove superior performance in those interactive scenarios. MLPerf v5.1 added interactive tests specifically because offline results tell only part of the story.

2) The MLPerf submission is reported as unverified​

Azure published results and logs described as an “unverified MLPerf v5.1 submission.” Unverified results can be legitimate engineering runs, but they have not gone through MLCommons’ formal verification and review process — the community gold standard for apples‑to‑apples comparisons. Treat unverified vendor‑submitted runs as strong signals, not definitive leaderboard proof. Microsoft itself flags the unverified status in its briefing.

3) The model used (Llama 2 70B) is not the newest frontier​

The test used Llama 2 70B — a sensible, widely used benchmark model — but modern frontier production workloads increasingly prefer larger or more sophisticated architectures (Llama 3 variants, DeepSeek‑R1 or 400B+ mixture‑of‑experts families). Whether the same per‑GPU throughput and intra‑rack scaling efficiency translate to those models — particularly ones with larger memory and communication profiles — is not yet shown. MLPerf v5.1 itself added DeepSeek‑R1 and Llama 3.1 tests precisely because workload characteristics are diversifying.

4) This is NVIDIA’s GB300 breakthrough made available by Microsoft​

NVIDIA designed the GB300 NVL72 as a purpose‑built, rack‑scale product. Azure’s achievement is making that rack available in cloud form and demonstrating a tuned TensorRT‑LLM workflow at scale. The architectural gains (HBM increases, NVLink fabric density, 50% more GPU memory in GB300 vs prior GB200 designs) originate in NVIDIA’s hardware roadmap; Azure’s value is integration, scale and reproducibility. Industry watchers correctly frame this as an arms race where the baseline platform is shared.

5) Facilities, power, and cooling are nontrivial constraints​

A GB300 NVL72 rack draws very large amounts of power (industry reporting and vendor material place GB300 racks in the 100–140 kW class). These systems require advanced liquid cooling and facility upgrades that many on‑prem or colocation sites cannot support without significant capex. That means practical deployment remains concentrated among hyperscalers and specialized cloud hosts for now. The infrastructure and operational burden — ducting, chilled water loops, electrical distribution — are part of the real cost of the performance.

---

Competitive landscape and the ephemeral nature of “firsts”​

Everyone in the hyperscaler and specialized cloud ecosystem will race to replicate and publish GB300‑class numbers. AWS, Google Cloud and specialist providers (CoreWeave, Lambda, Dell‑OEM partners) already have Blackwell‑class systems in various stages of deployment; several announced GB300 customers and commercial offerings weeks before or after Microsoft’s publicization. Verified MLPerf submissions from other cloud vendors and OEMs will likely surface quickly; once multiple vendors publish verified numbers, Azure’s “first” advantage is only a short‑lived time‑to‑market lead. AMD’s Instinct MI355X and other accelerator entrants also appeared in MLPerf v5.1 results, creating alternative price‑performance options to NVIDIA’s stack for some workloads. Competition is therefore both about raw throughput and about price, ecosystem, software support and supply chain access. MLPerf’s v5.1 results already broaden the vendor field and the set of viable architectures.

---

Engineering and operational implications for IT buyers​

Verify use cases, not headlines​

• Map your workload: quantify the proportion of requests that are batch versus interactive, and how much time‑to‑first‑token and tail latency matter.
• Run end‑to‑end tests: measure the complete pipeline (tokenizer, retrieval latency, network hops, orchestration) on ND_GB300_v6 instances, not just raw MLPerf‑style runs.
• Price the whole system: include facility costs, egress, availability zones, multi‑region redundancy and software licensing for optimized runtimes (TensorRT‑LLM).
• Validate multi‑tenant behavior: confirm sustained performance when nodes are shared or when background jobs execute concurrently.
• Plan portability: architect for model and runtime portability (model quantization options, alternative runtime stacks) so you avoid vendor lock‑in when supply or pricing changes.

Practical procurement checklist (short)​

• Confirm region availability and exact ND_GB300_v6 SKU names and quotas with your Azure account team.
• Request verified MLPerf v5.1 results for the relevant scenario(s) and ask for test harnesses or runbooks used by the provider.
• Check power and cooling assumptions for on‑prem deployments; expect ~100–140 kW per GB300 rack if considering private colocation OEMs.
• Insist on performance under real loads (interactive latency) and not just offline throughput metrics.

---

Deeper technical notes (for architects and engineers)​

Memory, precision, and throughput tradeoffs​

• FP4 quantization: FP4 implementations materially increase throughput by reducing memory bandwidth and arithmetic costs per token, with acceptable accuracy tradeoffs for many generation tasks. But model fidelity and hallucination risk must be validated per workload and safety constraints.
• HBM efficiency: Microsoft reports 92% HBM utilization during the benchmark — an indicator of a well‑tuned pipeline. But HBM efficiency in practice depends on model sharding strategies, NCCL configuration, and rendezvous coordination for token generation across devices.
• NVLink/NIC fabric: The GB300 NVL72’s denser NVLink domains reduce cross‑GPU synchronization overheads and keep attention‑layer gradient or K/V cache transfers fast; this is a core reason rack‑scale is advantageous for long‑context reasoning.

What the MLPerf v5.1 evolution means for buyers​

MLPerf v5.1 broadened the test set to include reasoning models (DeepSeek‑R1) and interactive scenarios. These workloads stress different system properties — latency, routing efficiency for mixture‑of‑experts (MoE), and cache management for long contexts — than traditional offline LLM throughput tests. A vendor’s v5.1 Llama 2 result is useful, but buyers should also look for verified DeepSeek‑R1 and interactive scenario submissions that more closely mirror production demands.

---

Risks, sustainability and the long view​

• Power and carbon: Racks consuming 100–140 kW dramatically increase cooling loads and PUE considerations. Liquid cooling mitigates thermal inefficiency, but lifecycle emissions and operational footprint must be factored into procurement choices.
• Upgrade cycles: Rapid hardware generation turns “best in class” into legacy quickly. The financial calculus for private data centers becomes sensitive: capex procurement timelines must anticipate substantial hardware churn to stay near the cutting edge.
• Vendor concentration: Many hyperscalers and cloud providers are dependent on a narrow set of accelerator vendors and DPU/network suppliers. That concentration raises supply‑chain, pricing and geopolitical risk.

---

How to assess the claim for your organization​

• Run or request an interactive benchmark that includes latency percentiles and cold‑start behavior. Offline throughput alone is insufficient for production chat services.
• Ask for verified MLPerf runs (not just unverified logs) and request the exact harness and command lines so you can reproduce in a staging environment.
• Evaluate the model family you plan to run — smaller, quantized models behave very differently from massive MoE architectures; ask vendors to demonstrate your target model family.
• Project end‑to‑end costs: compute, storage, networking, model licensing, and engineering for operationalization (routing, autoscaling, safety filters).

---

The bottom line​

Azure’s ND_GB300_v6 demonstration is an important milestone — a reflection of co‑optimized hardware and software delivering truly industrial‑scale inference throughput on a single rack, and a practical preview of what is technically possible in cloud AI. The test proves the GB300 NVL72 concept works at scale and that cloud providers can productize it for customers. Yet the win is measured in quarters, not decades. The result is built on NVIDIA’s GB300 rack architecture, validated in an offline MLPerf v5.1 configuration that remains unverified by MLCommons at the time of publication. The practical questions — latency under interactive load, scaling to larger or different model families (Llama 3.1, DeepSeek‑R1, MoE systems), and the economics of facility upgrades and power consumption — remain open and will determine who actually captures long‑term market share. Industry competition, verified benchmark submissions and multi‑vendor supply responses will blur the line between “first” and “standard.” For engineering teams and procurement leads, the sensible stance is pragmatic: treat ND_GB300_v6 as a powerful new tool, verify it against your real workloads (interactive and offline), and plan for the nontrivial facilities and financial implications of adopting rack‑scale GB300 infrastructure. The million‑token barrier has been broken — the next task is turning that headline capability into repeatable, secure and cost‑effective production services that survive competition, verification and the relentless march of hardware innovation.

---

Source: CTOL Digital Solutions Microsoft Breaks Million-Token Barrier in Cloud AI Race, But Victory May Be Fleeting