The caveat is not a footnote—it is the product definition. TileRT v0.1.5 is designed to run one in-flight decode request per node. This is a speed tier for a user whose response must stream immediately, not a replacement for the high-concurrency serving stack that produces the cheapest possible token.
SemiAnalysis’ August 10 report puts the apparent win at roughly 1.9× over the best conventional result at the 1,000/1,000 test point and 3.0× over the best FP8 result. TileRT’s own public repository confirms that the project now supports prefill/decode disaggregation with vLLM, making it possible to reserve the specialized decode path for selected requests while ordinary traffic stays on a conventional vLLM decode pool.
The news is less that NVIDIA GPUs suddenly became Groq or Cerebras hardware. It is that a software runtime has found a way to turn part of an existing B200 fleet into a high-priced, low-latency lane—without requiring an operator to buy a separate accelerator fleet for every customer who wants a faster cursor.
The benchmark win is real, but it is not an apples-to-apples capacity comparison
SemiAnalysis measures interactivity as tokens per second per user—the inverse of time per output token—and distinguishes it from aggregate tokens per second per GPU. That distinction is the center of this story. A chat interface, coding agent, or full-duplex voice assistant is judged by the pauses between tokens; an API provider selling bulk generation is judged largely by aggregate capacity and cost.
At the 8,000-token input and 1,000-token output point, SemiAnalysis reports TileRT at 340 tokens per second per user and 160.4 total tokens per second per GPU. Its comparison GB300 NVL72 system, running FP4 with multi-token prediction at concurrency 12, reaches a lower 154 tokens per second per user—but delivers about 240 total tokens per second per GPU.
In other words, TileRT wins the individual experience while the conventional configuration completes roughly 50% more aggregate work per GPU. Neither number invalidates the other. They represent different operating points, and a provider choosing between them has to decide whether its scarce resource is GPU-hours or user patience.
This is also why the headline comparison between an eight-GPU B200 node and a 72-GPU GB300 NVL72 system needs care. The larger NVLink domain is built to help scale collective operations and serve many requests efficiently. At batch size one, much of that scale-out advantage cannot be amortized. SemiAnalysis explicitly says its comparison concerns per-user interactivity, not system-wide throughput or cost.
The result should therefore be read as an achievement in low-latency decode engineering, not evidence that a single B200 node replaces nine HGX-class nodes for a general AI service. The performance figures are from SemiAnalysis’ InferenceX testing, rather than an independently published reproduction of the exact 1,000/1,000 and 8,000/1,000 runs. TileRT’s source and package are public, but the deployment requirements are restrictive enough that most operators will not be in a position to reproduce them casually.
TileRT attacks the GPU overhead between tokens
The core technical claim is credible because it targets a known weak point in autoregressive inference: small-batch decode spends too much time crossing boundaries rather than doing useful math. A conventional serving engine dispatches many GPU kernels. CUDA Graphs reduce CPU-side launch overhead by replaying a captured sequence, but the individual GPU kernels, synchronization points, and state transitions remain.
TileRT instead compiles a model-specific decode schedule into a persistent engine kernel. The host launches it, then the GPU keeps executing through the decode lifecycle. Within that persistent work, different warps and blocks specialize in data movement, tensor computation, and communication, allowing loads, computation, reductions, and stores to overlap more aggressively.
That matters at batch size one because the GPU cannot hide fixed costs behind other users’ requests. HBM bandwidth has risen dramatically across accelerator generations, but memory latency and synchronization overhead do not improve in the same way. At ordinary serving concurrency, those penalties are shared across a batch. For the user waiting on every token, they become a material part of each token’s delivery time.
SemiAnalysis says TileRT’s strongest end-to-end advantage comes after the first token. In its 1,000/1,000 GLM-5.1 test, the reported decode tail was 3.01 seconds, compared with 6.54 seconds for its best NVFP4-plus-MTP conventional comparison. That is useful context for administrators evaluating a “fast mode”: time to first token remains important, but a long response that begins promptly and then trickles out still feels slow.
TileRT also assigns nonuniform jobs across GPUs. In the GLM-5.1 design described by SemiAnalysis, one GPU can perform sparse-indexing work while the other seven execute the main attention-side workload. Conventional tensor-parallel deployments often aim for symmetric work across ranks; TileRT is deliberately breaking that symmetry when the model architecture makes it wasteful.
The cost is obvious: this is highly model-specific work. A persistent kernel that knows the model’s attention pattern, routing, tensor layout, communication schedule, and buffer residency is fast because it has already made a long list of decisions in advance. Change the architecture and much of that compilation and tuning work must be done again.
The vLLM integration is the operationally significant part
The vLLM integration is what turns TileRT from a benchmark runtime into something an inference operator can plausibly deploy. TileRT v0.1.5 uses vLLM’s V1 KV connector interface and MultiConnector support, according to the TileRT repository and vLLM’s July integration post. It does not require a forked vLLM server.
The split is straightforward. vLLM handles prefill, the phase that processes a prompt and benefits from broad parallel work and high throughput. TileRT handles selected decode requests after their KV cache state has been transferred, using NIXL or Mooncake transport. A router marks latency-critical requests for the TileRT pool; normal requests can remain on vLLM’s native decode path behind the same OpenAI-compatible endpoint.
For an infrastructure team, that arrangement is more valuable than a raw tokens-per-second claim. It allows a provider to expose a premium “ultrafast” option without dedicating every B200 to one-user-at-a-time inference. A shared prefill fleet can feed a small TileRT decode pool for interactive coding, voice, or agent supervision, while bulk completions and background agent work continue to use a throughput-optimized configuration.
But the integration carries operational constraints that the marketing language can obscure. TileRT’s current public build is compiled for eight NVIDIA B200 GPUs, Linux x86_64, Python 3.12, a CUDA 13-era stack, and a specific PyTorch build. Its maintainers describe these as hard requirements rather than suggested minimums. The supplied Docker image is intended to prevent version drift.
That makes TileRT an interesting option for a tightly controlled Linux inference cluster, not a drop-in package for a heterogeneous GPU estate. There is no Windows Server deployment path in the public documentation. Windows-focused organizations looking to test it will be running a dedicated Linux host or container environment on B200 hardware; WSL is not a credible production substitute for the required eight-GPU, RDMA-connected serving topology.
A premium tier can use existing GPUs, but it cannot absorb ordinary traffic
TileRT’s addressable market is the part of inference where customers visibly notice a response delay and will pay to remove it. That includes real-time voice systems, interactive coding assistants, high-priority API classes, and agent workflows where humans are waiting for tool calls or intermediate results.
The project’s public repository says TileRT is already used for Xiaomi MiMo-V2.5-Pro-UltraSpeed and Z.ai’s GLM-5.1 HighSpeed service. Xiaomi’s July engineering paper independently documents extensive production optimization for the MiMo-V2.5 family—especially KV-cache management, routing, RDMA-oriented distributed caching, and prefill/decode pipeline work—though it does not itself establish the specific TileRT throughput figures. Z.ai’s documentation confirms GLM-5.1 supports streaming responses and is aimed at long-running agentic coding workloads, where decode responsiveness can materially affect the experience.
The stronger commercial implication is fleet flexibility. A purpose-built dataflow accelerator may still deliver superior performance for specific models and latency targets, especially where large on-chip SRAM and a hardware-native execution model matter. TileRT cannot add SRAM, remove GPU HBM latency, or erase the limits of SIMT hardware.
What it can do is reduce the need to commit capital to a separate hardware pool before an operator knows how much premium low-latency demand will exist. GPU capacity can be reassigned among prefill, conventional decode, and a TileRT speed tier as demand changes. Dedicated low-latency hardware remains attractive where utilization is predictable and the speed requirement is extreme; it is less attractive when the operator needs flexibility more than a record-setting single-user number.
The model catalog is the constraint that will decide its reach
TileRT’s public v0.1.5 documentation supports GLM-5/5.1 and DeepSeek-V3.2. It ships separate backend libraries for those paths and permits one backend per Python process. That is a far smaller compatibility footprint than vLLM or SGLang, and it is the price of squeezing out the kernel boundaries and dynamic behavior that conventional runtimes tolerate.
This is the most important limit on the claim that software can displace specialized inference hardware. A provider cannot route a general catalog through TileRT today. It needs a supported model, a precise B200 environment, converted and sharded weights, matching KV-cache formats on both sides of the prefill/decode boundary, suitable RDMA configuration, and a traffic class valuable enough to justify dedicating a decode node to one active request.
SemiAnalysis plans to test TileRT at batch sizes of two, four, and eight, as well as on longer multi-turn agent traces. Those results will matter more to buyers than the present batch-one peak. The practical question is where the runtime’s latency advantage begins to flatten once a node carries more than one passenger.
For now, TileRT is a compelling specialized decoder for B200 operators with GLM-5-family or DeepSeek-V3.2 traffic and a genuine premium-latency product. It is not general-purpose serving capacity, and treating its 494.2-token-per-second result as a fleet-wide performance figure would be a procurement mistake.
References
- Primary source: SemiAnalysis
Published: August 10, 2026 at 4:51 AM UTC
Loading…
newsletter.semianalysis.com - Related coverage: github.com
Loading…
github.com - Related coverage: llmrumors.com
Loading…
www.llmrumors.com - Related coverage: vllm.ai
Loading…
vllm.ai - Related coverage: github.com
Loading…
github.com