transformer engine

The transformer engine tag on WindowsForum.com covers discussion of Nvidia's Transformer Engine library and how its low-precision training paths perform in practice. A recent thread follows a week-long experiment training a ChatGPT-style model from scratch on Nvidia DGX Spark workstations, comparing Transformer Engine's FP8 mode, its newer four-bit NVFP4 format, and nanochat's own FP8 code. The reported finding was that fewer bits and newer tooling did not speed up training, and that attention and power limits mattered more than the numeric format. Coverage here is hands-on and measurement-focused, with results tied to specific hardware and software versions.