How to Set Up a Local AI Server on Windows with llama.cpp: GPU Offloading and Security

What was built​

Raj's hardware is an RTX 4070 Ti Super with 16GB of VRAM, a Ryzen 5 7600X and 32GB of DDR5. The model is Qwen3.6-35B-A3B in an Unsloth UD-IQ4_XS quantization, a GGUF file of about 17.7GB. That is larger than the card's VRAM. He used llama.cpp build b11927, and the project's releases page showed newer builds, including b12138, on the day of publication. Treat his build number as a snapshot, not a recommendation.

The pitch is that the model never leaves the PC. The host does all the inference on the GPU and streams plain text to each client. Phones, laptops and tablets only need a browser.

The one-command version​

The basic command, run in PowerShell, is:

llama-server -m "C:\path\to\your-model.gguf" --host 0.0.0.0 --port 8080

Swap in the real path to your GGUF file. The XDA article's printed example has a stray space (- m). The correct flag is -m with no space.

The two network flags do the work:

• --host 0.0.0.0 tells the server to listen on all network interfaces. The official docs say the default is 127.0.0.1, meaning only the PC itself can reach it. The llama.cpp Web UI documentation says this flag is what lets other devices on your network reach the Web UI.
• --port 8080 pins the port. Raj says llama.cpp is moving its default to 9931. The current documentation already lists 9931 as the default. An explicit port avoids surprises either way.

Clients then browse to the host PC's local IP address and the chosen port. That's an inference from how the flags work, not something Raj spelled out. Windows Firewall may prompt you or block the connection the first time, so allow the port on your private network profile if devices can't connect.

The tuned command​

For his own testing, Raj used this longer command:

llama-server -m "C:\llama.cpp\models\Qwen3.6-35B-A3B-UD-IQ4_XS.gguf" --host 0.0.0.0 --port 8080 -ngl 99 --n-cpu-moe 8 -c 32768 -np 2 -kvu -fa on --load-mode none

Here is what he says each part does, with the documentation's view where it adds something:

• -ngl 99 keeps the other layers resident in VRAM.
• --n-cpu-moe 8 keeps the mixture-of-experts weights of the first eight layers in system RAM. The official option text confirms it keeps MoE weights of the first N layers on the CPU. This is how a 17.7GB file fits on a 16GB card.
• -c 32768 sets a roughly 32K-token context window.
• -np 2 creates two server slots, so two people can chat at once. The docs describe slots as parallel decoding with multi-user support. A related llama.cpp doc page says each slot holds one conversation and that the context is shared across slots.
• -kvu is described by Raj as letting both slots share the context. The README describes it as a single unified KV buffer shared across all sequences.
• -fa on and --load-mode none appear in his tested command, but he doesn't explain them. Check llama-server --help on your own build before copying them. Flag names change between releases.

Why does a 35B model run this fast? Raj says only about 3 billion of its parameters are active per token. That makes splitting the experts between VRAM and RAM workable. This is his explanation, and I haven't verified it against the model publisher.

The numbers, with caveats​

Raj sent the same prompt, five paragraphs on how a CPU works, from a fresh chat on each device. He ran it first one device at a time, then both at once.

Some context on reading this table:

• The client numbers can exceed the host number. The host figure comes from llama-bench, a different tool than the chat-based client runs. The two aren't strictly comparable.
• Two simultaneous users cost roughly a third of each user's speed. That's the practical price of sharing one GPU. It's still a comfortable reading speed.
• The test is thin. Raj doesn't give run counts, warm-up details, client software or exact llama-bench settings. These are one author's results on one machine, not a benchmark to plan around.
• The network isn't the bottleneck. Text is tiny compared with video. Raj contrasts this with Sunshine and Moonlight game streaming, which has to send every frame. His claim that 90 tokens per second is a fraction of a megabit is an estimate, not a measured network test.

Power and VRAM​

Raj reports his figures from nvidia-smi:

• Idle with the model loaded: about 14W.
• While generating a reply: 130–140W, dropping back afterward.
• VRAM use: about 15.4GB of 16GB, or 96%. That leaves roughly 0.6GB free.

The power figures are GPU-only readings from one card. They aren't a whole-PC electricity measurement. The VRAM figure matters more for gamers. With the model resident, your game will have almost no video memory left. Raj notes lighter models, such as Gemma 4 E4B, or offloading more expert layers to RAM would ease this. If you want to game on the same card, stop the server first. The README also documents a --sleep-idle-seconds option that unloads the model after inactivity and reloads it on demand. That could help, though I haven't seen it tested on this setup.

The security gap​

"The whole house" assumes everyone in it is trusted. Binding to 0.0.0.0 doesn't limit access to your home. It listens on every network interface the PC has. An older llama.cpp community discussion warns that this exposes the server to other clients on the network, with no password protection. The documentation notes that by default the server responds to every request.

Sensible precautions, based on the documentation and third-party guides:

1. Set an API key. The docs list --api-key (comma-separated for several keys) and --no-webui as options. One third-party test found the key protects API routes but leaves the Web UI page itself public. That's one tester's observation on one build, so verify it on yours.
2. Keep the port off the internet. Don't forward it on your router. One guide recommends keeping the port behind a firewall or VPN.
3. Limit the firewall rule to the Private network profile. Do this if you use Windows Defender Firewall.
4. Don't enable file or shell tools casually. A third-party guide notes that the server's experimental tools expose file and shell operations over the API. The official README says these tools expose file access, which is why CORS defaults to localhost when they're on.
5. Put guests on a separate Wi-Fi network. If your router offers a guest network, it keeps visitors off the AI server.

Is this a good idea?​

Raj himself concedes the ideal setup is a dedicated, headless box, ideally with several GPUs. Using your main PC is a compromise. The server competes with games for VRAM, and the machine has to stay on. The upside is real, though. A model that no phone or thin laptop could run is available to all of them, and the idle cost reads as small on his card.

Whether it replaces a $20-a-month cloud subscription depends on your tasks and how much you trust a quantized 35B model compared with the hosted alternatives. The XDA article doesn't test output quality. It only measures speed, so don't read the table as a quality comparison.

Quick start checklist​

• Confirm your GPU's VRAM and the model's file size. Plan CPU offloading if the file is bigger than the card.
• Download a current llama.cpp Windows build that matches your GPU backend. The release assets include CUDA 12, CUDA 13 and Vulkan builds for Windows x64.
• Start with the basic command, then add -ngl, --n-cpu-moe, -c and -np as needed.
• Add --api-key before you share the address with anyone.
• Open the host's IP and port from another device on the same network. A working chat page is your success check.
• If devices can't connect, check that the server is bound to 0.0.0.0, that Windows Firewall allows the port, and that both devices are on the same subnet.

The setup is genuinely simple. Just remember that "one command" ends where your network security begins.