The hardware question facing most enterprise AI projects is not whether to buy NVIDIA H100s, H200s, or Blackwell systems. It is whether the proposed service needs a GPU fleet at all — and, if it does, how much GPU memory is required after model weights, context length, concurrent users, and serving overhead are accounted for.
Spiceworks’ August 4 guide gets the central procurement instinct right: size the workload before selecting the accelerator. But its clean ladder from H100 to H200 to B200 to GB200 NVL72 risks hiding the decision that produces the largest budget errors. A model fitting in GPU memory is not the same as a production service fitting in GPU memory. The difference is the key-value cache, the working memory retained for each active conversation, plus runtime buffers and the capacity reserved to prevent latency from collapsing during a traffic spike.
NVIDIA’s current TensorRT and Triton documentation makes that operational reality unusually plain. Their LLM serving stacks allocate a distinct KV-cache pool after model weights are loaded, and capacity is governed by a token limit or a fraction of the GPU memory that remains. vLLM similarly exposes maximum concurrency at a configured model length as a cache-sizing figure. An IT team that buys based on “our 70-billion-parameter model loads on one card” may discover it bought a system for one long conversation at a time rather than the 20, 50, or 200 simultaneous requests promised to the business.
That is the compute reality check: the right unit for planning is not GPU count or even model parameters. It is validated tokens per second at the stated latency target, context length, precision, and concurrent-request level.
The shorthand calculation for model weights is useful: parameter count multiplied by bytes per parameter. A 70-billion-parameter model stored at BF16 or FP16 requires roughly 140GB just for the weights before runtime overhead. Quantization can substantially reduce that footprint, which is why a model that does not fit in an 80GB H100 at BF16 may fit at lower precision.
But that is a starting point, not a sizing result.
For an interactive LLM, every generated or processed token adds data to the key-value cache, the memory structure used to avoid recomputing the entire prior conversation at every next-token step. NVIDIA’s TensorRT documentation defines the cache in terms of batch size, number of attention heads, maximum sequence length, and hidden dimension. In practice, that means a deployment’s context-window policy and concurrency target can consume the VRAM supposedly saved through quantization.
The immediate consequence is easy to see. A chatbot capped at 4,000 tokens with a modest concurrent-user target can have a radically different VRAM requirement from the same model exposed to document-heavy RAG prompts at 32,000 tokens. Calling both projects “a 70B chatbot” tells a reseller almost nothing useful.
This also changes how Windows and infrastructure teams should interpret retrieval-augmented generation. RAG is often presented as a lightweight alternative to training a bespoke model, and it usually is. Yet the retrieval layer does not eliminate the serving problem: documents retrieved from Microsoft SharePoint, file shares, Exchange archives, or a vector database become prompt tokens, which in turn create cache demand. A poorly controlled document-chunking policy can turn a modest internal assistant into a long-context inference workload without anyone changing the base model.
Before approving hardware, require the application team to provide these operating limits in writing:
NVIDIA’s H100 specifications include several form factors. An H100 SXM has 80GB of HBM3, while the PCIe H100 NVL configuration uses two GPUs linked with NVLink and provides 188GB of aggregate HBM3 across the pair. Treating “H100” as one fixed 80GB product can therefore make an existing estate appear less capable — or a proposed configuration more comparable — than it really is.
The H200’s defining change is also more consequential than “same compute ceiling as H100.” It remains a Hopper-generation architecture, but NVIDIA specifies 141GB of HBM3e and 4.8TB/s of memory bandwidth, compared with 80GB and 3.35TB/s for H100 SXM. For workloads constrained by model placement, cache capacity, or memory bandwidth, those changes can alter throughput as well as whether a deployment fits at all. The H200 should not be sold as a free compute upgrade; it should also not be dismissed as merely more expensive H100 memory.
Blackwell changes the calculation further. NVIDIA’s B200 is documented with 192GB of HBM3e, and MLCommons’ MLPerf Training results show why raw capacity can be valuable: NVIDIA reported that larger Blackwell memory enabled a Llama 2 70B LoRA fine-tuning submission to avoid model-parallel communication. The result is not a universal performance multiplier — MLPerf systems, software versions, precision choices, and benchmark rules matter — but it confirms the mechanism. Fitting a workload with fewer partitions can remove inter-GPU traffic that a simple “FLOPS per dollar” comparison misses.
The practical dividing line is therefore not “Hopper for inference, Blackwell for training.” An H100 or H200 can be a sound choice for mature, well-characterized inference. A B200 can be justified for an inference service where higher memory enables more useful concurrency, lower precision, or a simpler single-GPU placement. Conversely, Blackwell will not rescue an application whose actual bottleneck is a slow retrieval database, a CPU-bound document pipeline, or users waiting on a downstream approval workflow.
NVIDIA describes the GB200 NVL72 as a liquid-cooled rack containing 36 Grace CPUs and 72 Blackwell GPUs, joined by an NVLink domain. Its published specification lists 13.4TB of aggregate HBM3e across the rack and 372GB for each GB200 Grace Blackwell Superchip, which consists of two Blackwell GPUs plus a Grace CPU. Software still has to partition the model and coordinate work across devices; the NVLink fabric makes that distributed configuration far more capable than ordinary multi-server networking, but it does not erase the engineering requirement.
That distinction matters to IT operations. A GB200 NVL72 procurement brings rack-scale power distribution, liquid cooling, network design, facilities coordination, installation planning, and a support model that are qualitatively different from adding one or two GPU servers to an existing virtualization cluster. NVIDIA’s own product material frames the system around trillion-parameter inference and large-scale training, which is the appropriate workload class.
An internal policy assistant, help-desk summarization service, software-code assistant, or RAG portal does not automatically become a GB200 problem because executives call it strategic. The question is whether measured demand requires the model size, sustained throughput, and multi-GPU communication characteristics that justify a rack-scale system. Most pilots will not.
The story’s quoted lead-time ranges deserve equal skepticism. Publicly posted availability figures vary by geography, reseller allocation, OEM relationship, chassis configuration, and whether the buyer wants a standalone accelerator, an eight-GPU server, or an integrated rack. No independently verifiable industry source establishes a universal six-week, 26-week, or 52-week delivery promise for these products. Treat a vendor’s date as a commercial term to be written into the purchase order, with the exact system bill of materials attached — not as a planning assumption derived from a GPU-market blog.
Pre-training a foundation model is an unusually capital-intensive, multi-node training task. Few enterprise IT departments need it, and MLCommons’ recent large-language-model training submissions illustrate the scale: the Llama 3.1 405B benchmark was designed around systems with hundreds or thousands of accelerators. The sensible alternative for most organizations is to select an existing model and spend engineering effort on data controls, evaluation, retrieval quality, and deployment.
Fine-tuning is more plausible on owned hardware, particularly when data residency or repeated adaptation work makes external processing difficult. But it must be sized as a training job. Depending on the method, memory must accommodate weights, activations, gradients, optimizer state, and checkpointing, not merely the final model that will later be served. LoRA and related parameter-efficient methods can reduce the burden, but their resource profile must be measured against the particular model and training framework.
Inference is usually where enterprise systems spend their productive life. It is also the workload whose economics are most sensitive to utilization. Uptime Institute’s recent work on AI inference economics identifies infrastructure utilization and the dilution of fixed costs as primary drivers of cost per token. That is a less glamorous conclusion than choosing a GPU generation, but it decides whether owned equipment earns its place.
A team with continuous, predictable use and requirements for local data control may have a strong case for on-premises inference. A team with quarterly fine-tuning runs, uncertain adoption, or sharp seasonal peaks should price cloud or managed GPU capacity first. The correct answer can be hybrid: own the base capacity that is busy every day, then burst development or peak traffic to a provider.
Use the actual serving stack — whether TensorRT-LLM, vLLM, Triton, or another supported runtime — to run a representative model with representative prompts. Record GPU memory consumption, achieved concurrent requests, queue time, first-token latency, output token rate, power draw, and failure behavior after the cache fills. Repeat the test at normal context and at the longest context the product intends to allow.
Then take the measured baseline to at least two suppliers. Ask each one to quote the complete platform: GPU server, CPUs, memory, NVMe capacity, network adapters and switches, support, racks, power distribution, cooling changes, software subscriptions, installation, and lead time. A low accelerator price accompanied by an unbudgeted cooling retrofit is not a low-cost deployment.
The CFO-ready conclusion is more specific than “buy only what you need.” Do not approve an AI compute project until the owner can show the model’s precision, maximum token policy, concurrency target, latency service-level objective, and measured utilization curve. Those five figures will tell the organization whether one GPU server, a small cluster, reserved cloud capacity, or a rack-scale platform is warranted — and they will expose when the honest answer is that the AI application is not ready to justify any hardware purchase yet.
NVIDIA’s current TensorRT and Triton documentation makes that operational reality unusually plain. Their LLM serving stacks allocate a distinct KV-cache pool after model weights are loaded, and capacity is governed by a token limit or a fraction of the GPU memory that remains. vLLM similarly exposes maximum concurrency at a configured model length as a cache-sizing figure. An IT team that buys based on “our 70-billion-parameter model loads on one card” may discover it bought a system for one long conversation at a time rather than the 20, 50, or 200 simultaneous requests promised to the business.
That is the compute reality check: the right unit for planning is not GPU count or even model parameters. It is validated tokens per second at the stated latency target, context length, precision, and concurrent-request level.
Model weights are only the first line of the memory budget
The shorthand calculation for model weights is useful: parameter count multiplied by bytes per parameter. A 70-billion-parameter model stored at BF16 or FP16 requires roughly 140GB just for the weights before runtime overhead. Quantization can substantially reduce that footprint, which is why a model that does not fit in an 80GB H100 at BF16 may fit at lower precision.But that is a starting point, not a sizing result.
For an interactive LLM, every generated or processed token adds data to the key-value cache, the memory structure used to avoid recomputing the entire prior conversation at every next-token step. NVIDIA’s TensorRT documentation defines the cache in terms of batch size, number of attention heads, maximum sequence length, and hidden dimension. In practice, that means a deployment’s context-window policy and concurrency target can consume the VRAM supposedly saved through quantization.
The immediate consequence is easy to see. A chatbot capped at 4,000 tokens with a modest concurrent-user target can have a radically different VRAM requirement from the same model exposed to document-heavy RAG prompts at 32,000 tokens. Calling both projects “a 70B chatbot” tells a reseller almost nothing useful.
This also changes how Windows and infrastructure teams should interpret retrieval-augmented generation. RAG is often presented as a lightweight alternative to training a bespoke model, and it usually is. Yet the retrieval layer does not eliminate the serving problem: documents retrieved from Microsoft SharePoint, file shares, Exchange archives, or a vector database become prompt tokens, which in turn create cache demand. A poorly controlled document-chunking policy can turn a modest internal assistant into a long-context inference workload without anyone changing the base model.
Before approving hardware, require the application team to provide these operating limits in writing:
- The model name, revision, weight precision, and any LoRA adapters that must be resident in memory.
- The maximum input and output tokens per request, rather than an aspirational “supports long context” statement.
- The sustained and peak number of simultaneous requests, with a defined first-token and total-response latency target.
- The percentage of traffic expected to be embeddings, reranking, batch jobs, interactive generation, and fine-tuning.
H100, H200, and B200 are not interchangeable capacity tiers
Spiceworks describes the H100 as an 80GB inference workhorse, the H200 as a 141GB memory-focused upgrade, and the B200 as a 192GB Blackwell part for larger training and inference. The broad direction is fair, but the labels conceal important product and system distinctions.NVIDIA’s H100 specifications include several form factors. An H100 SXM has 80GB of HBM3, while the PCIe H100 NVL configuration uses two GPUs linked with NVLink and provides 188GB of aggregate HBM3 across the pair. Treating “H100” as one fixed 80GB product can therefore make an existing estate appear less capable — or a proposed configuration more comparable — than it really is.
The H200’s defining change is also more consequential than “same compute ceiling as H100.” It remains a Hopper-generation architecture, but NVIDIA specifies 141GB of HBM3e and 4.8TB/s of memory bandwidth, compared with 80GB and 3.35TB/s for H100 SXM. For workloads constrained by model placement, cache capacity, or memory bandwidth, those changes can alter throughput as well as whether a deployment fits at all. The H200 should not be sold as a free compute upgrade; it should also not be dismissed as merely more expensive H100 memory.
Blackwell changes the calculation further. NVIDIA’s B200 is documented with 192GB of HBM3e, and MLCommons’ MLPerf Training results show why raw capacity can be valuable: NVIDIA reported that larger Blackwell memory enabled a Llama 2 70B LoRA fine-tuning submission to avoid model-parallel communication. The result is not a universal performance multiplier — MLPerf systems, software versions, precision choices, and benchmark rules matter — but it confirms the mechanism. Fitting a workload with fewer partitions can remove inter-GPU traffic that a simple “FLOPS per dollar” comparison misses.
The practical dividing line is therefore not “Hopper for inference, Blackwell for training.” An H100 or H200 can be a sound choice for mature, well-characterized inference. A B200 can be justified for an inference service where higher memory enables more useful concurrency, lower precision, or a simpler single-GPU placement. Conversely, Blackwell will not rescue an application whose actual bottleneck is a slow retrieval database, a CPU-bound document pipeline, or users waiting on a downstream approval workflow.
GB200 NVL72 is a data-center design, not a bigger GPU
The biggest correction to the simple product table concerns NVIDIA’s GB200 NVL72. It is not a normal server option that happens to contain more accelerators, and its memory is not a generic rack-wide pool that applications can consume as if it were a single 13.4TB card.NVIDIA describes the GB200 NVL72 as a liquid-cooled rack containing 36 Grace CPUs and 72 Blackwell GPUs, joined by an NVLink domain. Its published specification lists 13.4TB of aggregate HBM3e across the rack and 372GB for each GB200 Grace Blackwell Superchip, which consists of two Blackwell GPUs plus a Grace CPU. Software still has to partition the model and coordinate work across devices; the NVLink fabric makes that distributed configuration far more capable than ordinary multi-server networking, but it does not erase the engineering requirement.
That distinction matters to IT operations. A GB200 NVL72 procurement brings rack-scale power distribution, liquid cooling, network design, facilities coordination, installation planning, and a support model that are qualitatively different from adding one or two GPU servers to an existing virtualization cluster. NVIDIA’s own product material frames the system around trillion-parameter inference and large-scale training, which is the appropriate workload class.
An internal policy assistant, help-desk summarization service, software-code assistant, or RAG portal does not automatically become a GB200 problem because executives call it strategic. The question is whether measured demand requires the model size, sustained throughput, and multi-GPU communication characteristics that justify a rack-scale system. Most pilots will not.
The story’s quoted lead-time ranges deserve equal skepticism. Publicly posted availability figures vary by geography, reseller allocation, OEM relationship, chassis configuration, and whether the buyer wants a standalone accelerator, an eight-GPU server, or an integrated rack. No independently verifiable industry source establishes a universal six-week, 26-week, or 52-week delivery promise for these products. Treat a vendor’s date as a commercial term to be written into the purchase order, with the exact system bill of materials attached — not as a planning assumption derived from a GPU-market blog.
Training, fine-tuning, and inference should have separate budgets
The most common architectural error is to buy one permanent cluster sized for the largest thing the organization might someday do. That joins three workloads with incompatible utilization patterns.Pre-training a foundation model is an unusually capital-intensive, multi-node training task. Few enterprise IT departments need it, and MLCommons’ recent large-language-model training submissions illustrate the scale: the Llama 3.1 405B benchmark was designed around systems with hundreds or thousands of accelerators. The sensible alternative for most organizations is to select an existing model and spend engineering effort on data controls, evaluation, retrieval quality, and deployment.
Fine-tuning is more plausible on owned hardware, particularly when data residency or repeated adaptation work makes external processing difficult. But it must be sized as a training job. Depending on the method, memory must accommodate weights, activations, gradients, optimizer state, and checkpointing, not merely the final model that will later be served. LoRA and related parameter-efficient methods can reduce the burden, but their resource profile must be measured against the particular model and training framework.
Inference is usually where enterprise systems spend their productive life. It is also the workload whose economics are most sensitive to utilization. Uptime Institute’s recent work on AI inference economics identifies infrastructure utilization and the dilution of fixed costs as primary drivers of cost per token. That is a less glamorous conclusion than choosing a GPU generation, but it decides whether owned equipment earns its place.
A team with continuous, predictable use and requirements for local data control may have a strong case for on-premises inference. A team with quarterly fine-tuning runs, uncertain adoption, or sharp seasonal peaks should price cloud or managed GPU capacity first. The correct answer can be hybrid: own the base capacity that is busy every day, then burst development or peak traffic to a provider.
Benchmark the service before the purchase order
Calculators can help convert assumptions into a first bill of materials, but they cannot establish whether the assumptions describe the application. The Spiceworks-recommended calculators may be useful for scenarios and total-cost comparisons, yet their output should be treated as a model to challenge, not an independent validation of a vendor quote.Use the actual serving stack — whether TensorRT-LLM, vLLM, Triton, or another supported runtime — to run a representative model with representative prompts. Record GPU memory consumption, achieved concurrent requests, queue time, first-token latency, output token rate, power draw, and failure behavior after the cache fills. Repeat the test at normal context and at the longest context the product intends to allow.
Then take the measured baseline to at least two suppliers. Ask each one to quote the complete platform: GPU server, CPUs, memory, NVMe capacity, network adapters and switches, support, racks, power distribution, cooling changes, software subscriptions, installation, and lead time. A low accelerator price accompanied by an unbudgeted cooling retrofit is not a low-cost deployment.
The CFO-ready conclusion is more specific than “buy only what you need.” Do not approve an AI compute project until the owner can show the model’s precision, maximum token policy, concurrency target, latency service-level objective, and measured utilization curve. Those five figures will tell the organization whether one GPU server, a small cluster, reserved cloud capacity, or a rack-scale platform is warranted — and they will expose when the honest answer is that the AI application is not ready to justify any hardware purchase yet.