A laptop displays local AI performance benchmarks and a diagram routing models from system RAM to a compact GPU.
An 8GB laptop GPU is not supposed to run a 35-billion-parameter model at conversational speed. XDA Developers writer Abhinav Raj says his Lenovo Legion 5 did exactly that. The laptop has a mobile RTX 3070 with 8GB of VRAM. A single llama.cpp setting, --n-cpu-moe, took generation on Qwen3.6-35B-A3B from about 4 tokens per second to about 33. These are one writer's numbers on one machine, not an independent replication. Still, the technique behind them is real, documented, and worth understanding if you run local AI on a Windows PC.

A laptop displays local AI performance benchmarks and a diagram routing models from system RAM to a compact GPU. What was tested​

Raj used llama-bench, the performance tool bundled with llama.cpp. He compared two models on the laptop:

  • Qwen3.6-27B, a dense model, at IQ4_XS quantization (14.37 GiB). He used it as a reference point, since a dense model has no expert layers to move.
  • Qwen3.6-35B-A3B, a mixture-of-experts (MoE) model, in an Unsloth Dynamic UD-IQ4_XS quantization (16.50 GiB).

His command was:

.\llama-bench.exe -m Qwen3.6-27B-IQ4_XS.gguf -m Qwen3.6-35B-A3B-UD-IQ4_XS.gguf -ngl 99 -ncmoe 0,28 -p 512 -n 128 -r 3 -ctk q8_0 -ctv q8_0

The settings break down like this:

  • -ngl 99 asks llama.cpp to place all layers on the GPU.
  • -ncmoe 0,28 runs each model with no expert offload and with expert weights for 28 layers moved to system RAM.
  • -p 512 -n 128 sets a 512-token prompt and 128 generated tokens.
  • -r 3 repeats each test three times.
  • -ctk q8_0 -ctv q8_0 stores the KV cache at 8-bit precision.

The reported results​

Model and setupPrompt processing (tokens/sec)Generation (tokens/sec)
Qwen3.6-27B dense, IQ4_XS38.830.75
Qwen3.6-35B-A3B MoE, no expert offload109.664.02
Qwen3.6-35B-A3B MoE, n-cpu-moe 28341.3732.82

The key comparison is the same MoE model with and without offload. Generation rose from 4.02 to 32.82 tokens per second, which works out to roughly 8.2 times faster. Prompt processing roughly tripled. The dense model's 0.75 tokens per second shows the other side of the story. A dense model has no cheap parts to push into RAM, so overflowing VRAM hurts every token.

Why the trick works​

In a mixture-of-experts model, only a fraction of the weights run for each token. For this model, Raj says roughly 3 billion of the 35 billion parameters are active per token. That is what the "A3B" in the name refers to.

Several independent guides describe the flag the same way. It keeps the expert feed-forward tensors of the first N layers in system RAM and runs them on the CPU, while attention, shared weights and the KV cache stay on the GPU. A separate explainer puts the trade-off simply: higher N means less VRAM used and more CPU work per token, while lower N keeps more experts on the GPU and generates faster.

That framing matters for reading the result. The flag does not make an 8GB card act like a bigger one. It decides which part of the model uses the GPU's fast memory and which part uses slower system RAM. Without offload, the 16.5 GiB model overflowed an 8GB card, which is the likely reason for the 4 tokens per second. Moving the experts out left the memory-hungry, always-used parts on the GPU.

Where the evidence stops​

This is a promising result, but it has limits.

  • It is a single-writer benchmark. Raj's account does not give the laptop's CPU, system RAM capacity, llama.cpp build, CUDA version, power mode or thermals. It also doesn't report run-to-run variance. Treat the numbers as indicative, not as a guarantee.
  • "28" is not a magic number. The right value depends on your model, quantization, context length and card. One guide advises that there is no universally good value. Another recommends lowering N until VRAM runs out, then stepping back by one.
  • It only helps MoE models. One guide states that a dense model doesn't have routed expert tensors, so the flag does nothing for them. Raj's dense-model run reflects that.
  • It can backfire. If a model already fits in VRAM, offloading experts just means reading them over PCIe for no gain.
  • Short tests are not long sessions. A 512-token prompt and 128 generated tokens say little about long-context work, sustained thermals or answer quality. Raj's desktop experiment used a 32K context, but the laptop benchmark command sets no context length. Don't carry that figure over to the laptop.
  • "Faster than I can read" is subjective. It refers to the 32.82 tokens per second of generation, not to prompt processing or response quality.

One technical claim from the XDA piece is only partly confirmed by independent sources. Raj says the attention layers, embeddings, shared weights and KV cache all stay in VRAM. Guides agree that attention and the KV cache stay on the GPU. One adds a nuance: the input embedding stays on the CPU regardless of -ngl. That is a detail of the model's tensor layout, not a flaw in the result. Don't assume everything non-expert lives in VRAM.

Trying it on your own PC​

This is a general approach based on the flag documentation, not a guarantee for your hardware.

  1. Check the model type. It must be a mixture-of-experts model. If the model card doesn't say so, this flag won't help.
  2. Check your RAM. The offloaded experts live in system RAM, so you need enough free memory for them. Raj's report doesn't say how much RAM his laptop has.
  3. Start with a benchmark. Run llama-bench with several values, for example -ncmoe 0,14,28, so you can compare them in one table. The tool's README says multiple values can be separated by commas for each parameter, and it lists -ncmoe as the short form of --n-cpu-moe. I could not confirm this from the documentation text I reviewed, so check the short aliases (-ncmoe, -ctk, -ctv) against your installed build before relying on them.
  4. Tune downward. Find the lowest N that loads without running out of memory, because every extra layer moved to RAM costs some speed.
  5. Record your settings. The benchmark output captures the build, GPU and KV-cache types. Keep those with your numbers so comparisons stay honest.

A moving target​

llama.cpp's expert handling is changing quickly. Recent reporting describes a newer --moe-cache-mib option, a GPU cache for experts kept in RAM, disabled by default. One write-up still recommends keeping a hand-tuned --n-cpu-moe for small cards where only a sliver of the experts fits, which describes an 8GB laptop GPU well. Newer builds also try to fit models automatically, so your results may differ from Raj's depending on your version.

The bottom line​

The headline overstates things slightly. It took -ngl 99 plus a tuned expert-offload count, not one flag alone. The underlying idea holds up, though. With a MoE model and enough system RAM, an 8GB gaming laptop can run a model much larger than its VRAM at a usable speed. The cost is that your RAM speed, CPU and tuning now matter as much as the GPU. The practical advice is to benchmark your own machine and treat any single set of numbers as a starting point.

 

References

  1. I changed one llama.cpp flag, and my 8GB laptop GPU now runs a 35B model faster than I can read XDA 2026-10-22T20:30:21+00:00
  2. How to pick --n-cpu-moe in llama.cpp: Qwen3.6 35B-A3B on 12, 16 and 24 GB GPUs - DEV Community dev.to
  3. llama.cpp's MoE Expert Offload: --n-cpu-moe and Streaming, Explained shawnmayzes.com