tilert

The tilert tag on WindowsForum.com covers discussions about TileRT, an inference runtime for NVIDIA GPUs, with a focus on its performance in token-per-second benchmarks. Recent content highlights TileRT v0.1.5's ability to deliver high throughput, such as 494.2 tokens per second on an eight-GPU B200 server, but notes a key limitation: it processes only one in-flight decode request per node. This makes it a specialized solution for scenarios prioritizing immediate response streaming over multi-user concurrency. The tag includes comparisons with other inference entries and technical details relevant to AI inference performance on Windows and enterprise hardware, appealing to those interested in GPU-accelerated AI workloads and benchmarking.