Why this matters now: llama.cpp 1.0
On November 13, the ggml-org team released llama.cpp v1.0.0 together with ggml v1.0.0. The team has committed to semantic versioning for both projects: a breaking change to the public API or ABI now means a major version bump. That makes it practical to pin a version when you compare runs. Pinning is the point of a benchmark harness.
Two details matter for testers:
- The release removes 35 deprecated functions, plus the warmup and defragmentation-threshold settings, from llama.h. That affects code linking against libllama. It does not affect a harness that only calls command-line tools such as llama-bench.
- The version number is not a performance claim. The headline is a promise rather than a speed jump. Benchmark your own card to see what changed for you.
The project also publishes numbered nightly builds, such as b12359. The v1.0.0 release notes point to nightly build b12359. Don't mix nightlies and the stable tag within one comparison. Doing so tests the code, not your hardware.
What each tool measures
No single program answers "is my GPU fast?" Use one tool per question:
| Tool | Measures | Best use |
|---|---|---|
| 3DMark Steel Nomad | A fixed graphics scene | Comparing against public results |
| Unigine Superposition (1080p Extreme) | Sustained rendering load | Thermal soak and stability |
| llama-bench | Tokens per second for local AI | Judging a card for local models |
| nvidia-smi | Power, clocks, temperature, VRAM, utilization | Explaining why a result moved |
UL describes Steel Nomad as a DirectX 12 benchmark for high-end gaming PCs running Windows 10 and 11. UL's overview lists a graphics test that measures GPU performance, rendered at 3840 × 2160. It is a graphics test, not an inference test. A high Steel Nomad score does not tell you how fast a card generates tokens.
llama-bench reports two separate numbers:
- pp512 is prompt processing on a 512-token prompt. It is compute-heavy.
- tg128 is token generation of 128 tokens. It leans on memory bandwidth.
This split is a useful rule of thumb, but it isn't universal for every model or backend. Report both numbers and don't average them.
NVIDIA's nvidia-smi documentation supports the logging approach. It offers a loop mode with a millisecond interval (--loop-ms). It also lets you select a GPU with -i/--id. The docs caution that enumeration order isn't guaranteed to stay the same across reboots, so on multi-GPU systems use a UUID or PCI bus ID instead of assuming index 0.
Prerequisites
- An NVIDIA GeForce RTX card and a current driver. nvidia-smi installs with the driver.
- Windows 11 64-bit or Ubuntu 24.04 LTS.
- Python 3.11 or newer. The scripts use only the standard library.
- CMake and a compiler if you build llama.cpp yourself. That means Visual Studio 2022 Build Tools on Windows or GCC on Linux. You also need the CUDA Toolkit for the CUDA backend. Prebuilt Windows release archives are a shortcut.
- One GGUF model, 7B to 8B parameters, in Q4_K_M quantization. It should fit fully in VRAM, so keep the exact filename in your notes.
- About 20 GB of free disk space.
The graphics tools are separate downloads. 3DMark is paid software, so check the current price and edition on its store page before buying. Check Unigine's download page for licensing terms.
AMD and Intel owners can still use the graphics tests and llama.cpp's Vulkan backend (-DGGML_VULKAN=ON). You will need a different telemetry source.
The 12 steps
Plan on roughly 80 minutes for the first pass and under 20 for each re-test. Treat those as estimates. Cooldown intervals and your hardware change the real time.
Steps 1–3: Plan, prepare, scaffold
- Define the test plan. Write down the one question you're answering. Examples: did a driver change anything, did an undervolt cost speed, or does a power limit save energy at an acceptable cost? Label each run with the single variable you changed, such as
baseline,driver-updateorpl-80. Keep labels short and free of spaces. - Prepare the PC.
- Close browsers, launchers, chat apps and overlays.
- Set the high-performance power mode.
- On a laptop, plug in the charger.
- Idle for five minutes after boot.
- Don't change drivers mid-session, and reboot once after any install.
- Run
nvidia-smito confirm the GPU is visible, and note the room temperature.
- Scaffold the project. Create a
gpubenchfolder with arunssubfolder, then runpython -m venv .venv. Activate it with.venv\Scripts\Activate.ps1in PowerShell, orsource .venv/bin/activateon Linux.
Steps 4–5: Inventory and telemetry
- Capture a system inventory. A score is nearly useless a month later without its context. Record these items:
- the OS, Python and CPU
- the GPU name, driver, VBIOS, memory size, power limit and PCIe link generation and width
- the active power profile
- the output of
llama-cli --version
--query-gpuwith--format=csv,noheader,nounits. An idle GPU often drops to a lower PCIe generation to save power. NVIDIA's docs say the current link can be reduced when the GPU isn't in use. Re-read the link state under load if you suspect a slot problem. - Build the telemetry logger. It starts nvidia-smi in loop mode at 500 ms and writes one CSV row per sample. Query these fields:
timestamppower.drawtemperature.gpuutilization.gpumemory.usedclocks.smclocks.memfan.speed
[N/A]for fields such as fan speed or power. NVIDIA notes that unsupported fields can show as N/A. Check the exact query on your own card before you trust a full run.
Step 6: Graphics benchmarks
- Start the logger in a second terminal, then run each graphics test three times with about 90 seconds between runs.
- 3DMark Steel Nomad: use default settings and record the overall score, graphics score and frame rate. Custom settings make the result incomparable with public results.
- Unigine Superposition: use the 1080p Extreme preset. Record the score and average FPS.
Treat the first run as a warm-up. A cold card boosts highest. A large gap between run 1 and run 3 shows how much your cooling limits sustained performance.
One caveat applies here. The harness does not launch 3DMark or Superposition. You enter those scores by hand with --score flags, and you start graphics telemetry separately. The telemetry script only runs automatically around the AI test.
Steps 7–8: Build llama.cpp and run llama-bench by hand
- Build llama.cpp and get a model. Pin the stable tag:
Confirm the tag exists before checking it out. On Windows, run the same commands from a Developer PowerShell for Visual Studio 2022. Binaries land inCode:git clone [GitHub - ggml-org/llama.cpp: LLM inference in C/C++ · GitHub](https://github.com/ggml-org/llama.cpp) cd llama.cpp git fetch --tags git tag --list "v1*" git checkout v1.0.0 cmake -B build -DGGML_CUDA=ON cmake --build build --config Release -jbuild\bin\Releaseon Windows andbuild/binon Linux. Put your GGUF file in amodelsfolder and verify the build withllama-cli --version. - Run llama-bench by hand first. Never automate a command you haven't seen work:
./build/bin/llama-bench -m ../models/model-8b-q4_k_m.gguf -ngl 99 -p 512 -n 128 -r 5
Check three things in the output table:- The backend column should say CUDA, or Vulkan if you built that. A CPU result means the GPU wasn't used.
- The
nglcolumn should show your requested offload. - The plus-minus spread should be small. A spread above a few percent suggests background load or throttling.
Step 9: Automate the AI run
- The harness script
runbench.pydoes the following:- It creates a timestamped folder.
- It saves the inventory.
- It runs one unrecorded warm-up pass.
- It starts telemetry and runs three rounds of five repetitions each, with a 90-second cooldown between rounds.
- It calls llama-bench with
-o jsonand writes the raw output.
--score steel_nomad=VALUE.
Steps 10–12: Read, compare, archive
- Read the results. The report script filters telemetry to samples where GPU utilization is at least 50%. It takes the median across rounds and reports the spread. It also divides median tokens per second by average power to get tokens per joule. Treat that figure as a rough comparison metric. The power input is GPU telemetry sampled only during busy rows, not wall power. NVIDIA's docs say average power draw for the board is supported on Ampere (except GA100) or newer devices, so older cards may not report it.
What you see Likely meaning Spread under about 2% Stable test conditions Round 1 much faster than rounds 2 and 3 Thermal or power throttling Minimum SM clock far below average Clocks dropping under load Peak VRAM near total memory Model plus cache nearly fills VRAM Low tg128 but normal pp512 Memory-bandwidth limit, or VRAM spill Average power near the power limit Power-limited, not temperature-limited - Compare two runs after one change. Rerun with a new label, then compare the two folders with
compare.py. It calls a delta "real" when it exceeds the larger of the two spreads. That is a filter for what deserves a second look, not a significance test. A single-digit gain with a 3% spread is not evidence. - Archive and share. Copy the
runsfolder to cloud storage or a Git repository after each session. When you post results, include these fields:- the card and driver version
- the llama.cpp version
- the model filename and quantization
- the exact llama-bench arguments
- the ambient temperature
Pitfalls that ruin results
- Mixing models or quantizations. A Q4_K_M file and a Q8_0 file differ in size and speed. If you swap files, you're measuring the file.
- Testing on battery or a balanced power plan. Laptops cut GPU power on battery.
- Publishing only the first run. A cold card posts its best case.
- Leaving overlays and background apps running. Even a small background load can add several percent of spread.
- Changing the driver mid-comparison. Turn off automatic driver updates during a test series.
- Letting the model spill out of VRAM. Token generation collapses when layers move to system memory.
- Mixing nightlies with the stable build.
- Trusting a score without its telemetry. A higher score at much higher power draw is a different statement from a higher score at the same power.
Troubleshooting
- nvidia-smi not found. This usually means a broken or partial driver install. Reinstall the driver and reboot. In WSL, call the tool by its full path if it isn't on the path.
- Telemetry CSV has a header but no rows. The logger drops any line that doesn't split into exactly eight fields. Run a small query by hand, such as
nvidia-smi --query-gpu=power.draw,fan.speed --format=csv. Remove any field your card rejects from the field list in both the logger and the report. [N/A]values. These are normal on some laptop and workstation GPUs. The run still works, but the tokens-per-joule line will be missing if power isn't reported.- llama-bench says the backend is CPU. The GPU backend wasn't built. Delete the build folder, confirm the CUDA toolkit is visible, and reconfigure with
-DGGML_CUDA=ON. AMD and Intel owners should use the Vulkan flag. - JSON parse error in the harness. Run llama-bench by hand and read the raw output. A mismatched binary is a common cause, so pin v1.0.0 and rebuild.
- Model fails to load or runs out of memory. Use a smaller quantization or model, or lower
--ngl. A partial offload is a CPU/GPU mix. Label it clearly and never compare it to a full-offload run. - nvidia-smi fails after a driver change. Reboot and reinstall cleanly using one kernel-module variant. If the error says the GPU "fell off the bus," suspect power delivery, a loose PCIe connection or an unstable overclock.
- Scores swing more than 5% between rounds. Suspect thermals or background load. Lengthen the cooldown to around 150 seconds and close everything on the taskbar.
- A benchmark crashes or shows a black screen. Remove any overclock or undervolt, then retest the driver with a game.
ModuleNotFoundError. Keep the scripts in the same folder and launch them from there, with the virtual environment active.
Advanced extensions
- Context sweeps. Pass a list such as
-p 512,2048,8192to see how speed changes as the cache grows. - Power-limit sweeps. Use
nvidia-smi -pl(administrator rights required) and label runspl-80,pl-70and so on. Reset the limit afterward. - Flash attention and batch size. Flags such as
-fa 1and-bchange speed and memory use. Change one flag per run. - Throttle reasons. Log the clocks event reasons field if your driver supports it, or read
nvidia-smi -q -d PERFORMANCEduring a long run. A power cap and a thermal cap need different fixes. - Other vendors. Only the telemetry script and the report's field names need to change for AMD or Intel, since llama-bench and the comparison logic are vendor neutral.
- A CPU control run. One run with
-ngl 0confirms the GPU delivers a real speedup. - A weekly scheduled run. It builds a history of how drivers and cooling affect your card.
Bottom line
The value of this setup is the discipline, not any one tool. Pick one graphics test, one AI test and one logger. Hold the driver, model, build and ambient conditions constant. Change one variable, then compare against the baseline and the noise floor.
The sample tables in the guide use placeholder numbers, not real measurements. The "real versus noise" verdict is a heuristic. After a month of runs you'll hold a dataset about your own hardware, which is more useful than any leaderboard screenshot.
References
- GPU Benchmark Software: 12 Steps, 80-Min Test Plan - shattered.io shattered.io · 2026-11-18T09:19:25+00:00
- NAME docs.nvidia.com
- UL Benchmarks Overview of 3DMark Steel Nomad support.benchmarks.ul.com