Nvidia DGX Spark Training Test: Native FP8 Beats NVFP4 as Attention and Power Bottlenecks Emerge

These are one author's measurements, not independent benchmarks. The comparisons were short runs on specific hardware and software versions.

The scoreboard: three recipes, one winner​

Here is what the author reports for the recipe comparison.

• Transformer Engine FP8 was the baseline in the two-Spark tests.
• NVFP4 used about 7% less memory than FP8 but ran about 7% slower per step.
• Nanochat native FP8 took 20.6% less time per update than Transformer Engine FP8 and used far less memory.

The head-to-head over updates 5 to 20 on the same pair of Sparks looked like this:

A repeat on a healthy pair later in the week landed at a 20.5% gap. The test covered only 20 steps, so the author can't say how many hours the native path would have saved over the full run.

Why NVFP4 isn't a free lunch​

Four-bit training is more than a smaller number format. NVIDIA's Transformer Engine documentation is not the source for this, but the docs the packet drew on describe NVFP4 as a 1-sign, 2-exponent, 1-mantissa (E2M1) format. It uses an FP8 scaling factor shared by each block of 16 values, plus a global FP32 scale. It also adds stochastic rounding for gradients and a Random Hadamard Transform to smooth out outliers. That extra machinery plausibly explains why the bit savings didn't turn into speed here. That is my inference, not something the author profiled. NVIDIA's docs also list NVFP4 training support on Blackwell-class devices.

NVIDIA's own research claims NVFP4 pretraining can match an FP8 baseline on loss and accuracy. That is a claim about quality, not about speed on a GB10 running nanochat.

Single Spark: busy is not productive​

The first run was a 20-layer, 896.5-million-parameter model trained on 5.2 billion tokens. It took six days and 21 hours.

• Weights & Biases showed GPU utilization at about 96% the whole time.
• Power draw sat mostly at 50 to 60 W and peaked at 90 W.
• The GPU clock held near 2,460 MHz, so this was not the power-brick throttling problem some owners have reported.
• Nanochat estimated roughly 15 quintillion floating-point operations, or about 25 TFLOPS sustained.
• A plain matrix-multiply test on the same Spark hit 98.6 TFLOPS at 81 to 85 W.

The author derived about 125 TFLOPS of dense BF16 from Nvidia's headline of up to 1 petaFLOP at FP4. Nvidia's product page confirms that headline figure is quoted at FP4 precision. The 125 figure is the author's own arithmetic, so treat it as a rough yardstick.

The attention fallback​

The software clue was in the startup log. FlashAttention 3 wasn't available, so nanochat fell back to PyTorch's SDPA attention. That path can't handle the model's sliding-window pattern, and the log warned that utilization would be terrible.

A separate nanochat discussion by another Spark owner explains why. It says FlashAttention 3 depends on Hopper-specific hardware features, so no driver update will bring it to the GB10. That user added FlashAttention 2 as a middle tier (FA3, then FA2, then SDPA). They report FA2 is about 28% faster than SDPA on GB10 and supports sliding windows natively. Rice-Jones used FlashAttention 2 for his two-Spark run.

Opinions on the attention path differ across the community. One setup guide for ML training on the Spark says PyTorch's native SDPA with cuDNN is faster than the upstream flash-attn package on Blackwell. That advice concerns Hugging Face-style training code, not nanochat's sliding-window model. If you're tuning your own run, benchmark the attention path rather than trusting any single recommendation.

Misleading dashboards​

Two readouts looked alarming but didn't mean the hardware was idle:

• Nanochat's bf16_mfu printed 0.00 on every step because it had no peak-compute figure for the GB10.
• The Weights & Biases "GPU Time Spent Accessing Memory" chart stayed at 0% because the GB10 doesn't report that metric.

The author also notes the restart math. At an assumed 20% speedup, restarting a run this long only pays off in roughly the first day.

Two Sparks: a new bottleneck​

The main run used two Sparks and a 24-layer, 1.38-billion-parameter model with FlashAttention 2 and Transformer Engine FP8. Over about 151 hours it completed 133,633 updates and 8.76 billion tokens, at 65,536 tokens per update. After every update the two machines must synchronize, so the pair moves at the speed of the slower one.

A post-run profile found one GPU at about 600 MHz while its twin ran near 2,190 MHz. The fast machine spent about 3.3 seconds of each 7.2-second update waiting. The author stresses this did not slow the six-day run, which averaged about 4.07 seconds per update, in line with earlier tests. The throttling appeared afterward.

The reported fix​

The symptom resembles owner reports of a stuck USB-C power-delivery controller in the power brick that pins the GPU near 611 MHz without throttle flags. A normal reboot didn't help. After a cold drain of both Sparks, the affected unit returned to about 2,190 MHz under load, and the test update dropped from 7.2 to 3.9 seconds. This is the author's diagnosis and remedy, not a confirmed root cause or an official Nvidia procedure.

Other two-Spark experiences​

The other owner's account shows the same theme from a different angle. That user describes a Spark bundle as two independent computers, each with its own CPU, memory and filesystem. The link between them is a 200 Gbps cable.

That user found problems that only appear across two nodes. Mismatched compile caches let one node race ahead until NCCL's watchdog killed the job. A large batch size exhausted unified memory and froze a machine instead of raising a clean out-of-memory error. They also report a roughly 12-fold memory-bandwidth gap between an H100 and the GB10's 273 GB/s as the likely reason for lower efficiency. Rice-Jones raises the same bandwidth suspicion but can't confirm it, because the GB10 exposes no bandwidth counters.

A checklist before you commit a week of GPU time​

These steps come from the author's experience, supplemented by the other owner's report. They are not official Nvidia guidance.

1. Read the startup warnings. Resolve any attention-backend fallback before launching.
2. Pilot the exact recipe. Run a few dozen steps for each candidate (Transformer Engine FP8, NVFP4, native FP8) and compare seconds per update and peak memory.
3. Check every node. Compare GPU clocks, drivers and power draw on both Sparks before and during a distributed run.
4. Watch power and MFU, not just utilization. A 96% utilization graph can sit alongside low compute use.
5. Leave memory headroom. The Spark's unified memory means exhaustion can freeze the machine, according to the other owner's report.
6. Warm up compilation on both nodes. Mismatched caches can desynchronize ranks.

What the model could do​

The model was modest. Early in training it continued "If 5x + 3 = 13, then x is" with "5x 5x 5x". By the end it answered 5. The author describes it as roughly GPT-2-level, and it did better at haiku. Another owner's finished d24 model wrote working code but got 15 × 37 wrong. These are anecdotes, not standardized evaluations.

Bottom line​

For anyone building AI workstations, the useful point is that utilization is a poor proxy for efficiency on a new platform. The newest recipe wasn't the fastest, and a second machine adds a failure mode. A 20-step pilot run told this author more than six days of waiting did, and it costs almost nothing to repeat.