Open WebUI Desktop on Windows: NVIDIA CUDA Setup Problems and Local LLM Performance vs Jan

A note on dates: the metadata for the review says October 20, 2026. The StorageReview page itself is bylined October 19, 2026. Every number below is StorageReview's own test result, not an independent measurement.

What was tested​

The reviewer used the same hardware and model files from an earlier Jan v0.8.6 review, so the two apps could be compared under similar conditions. The system looked like this:

Desktop and server have separate version numbers, which can confuse people at first. I could not independently confirm the v0.12.2 server version. The Desktop v0.0.20 release is listed in the project's release page, and its notes describe a Linux webview rendering fix. Treat the server version as part of the reviewer's test setup.

Setup problems on Windows​

The review says most of the time went into the llama.cpp and CUDA setup, before normal testing could start. The installer offers optional llama.cpp and Open Terminal components and lets you change the install folder. The reviewer enabled llama.cpp and left Open Terminal off. Each problem below is the reviewer's own report from one machine.

1. "Latest" failed. Installing llama.cpp with the Version field on latest and CUDA 12.4 selected failed almost immediately. The desktop log showed latest resolving to llama.cpp v0.6.0, with a Windows CUDA archive that wasn't available under that release. Pinning build b11612 got past this.
2. CUDA DLLs were missing. llama.cpp still wouldn't start because the separate CUDA runtime package hadn't been downloaded. The reviewer copied cudart64_12.dll, cublas64_12.dll and cublasLt64_12.dll from the official CUDA 12.4 package into the b11612 folder. After that, llama-server started on port 18881.
3. A reinstall wiped the runtime arguments. Without --ctx-size 8192, llama.cpp fell back to the models' native context sizes. That meant 169,984 tokens per slot for V2 VL High and 192,000 for V3.5. VRAM climbed close to the card's 32GB limit, and the remote desktop session slowed until the 8,192 setting was restored.

The missing-DLL problem is not unique to this app. One Windows guide for pairing OpenWebUI with llama.cpp notes that CUDA builds often need runtime DLLs that aren't on your PATH, and that placing them next to the executables avoids "missing DLL" errors. Separately, a GitHub discussion from April 2026 shows another user on an RTX 5090 whose model ran on the CPU instead of the NVIDIA GPU with the built-in llama.cpp. That suggests GPU setup is a known rough edge, though it does not prove your install will hit the same problems.

Practical takeaways:

• If latest fails, pin a specific llama.cpp build.
• Check that the CUDA runtime DLLs sit beside llama-server.
• After any reinstall, re-check your context size and GPU-layer settings.
• Open WebUI's llama.cpp documentation shows --ctx-size and --n-gpu-layers as explicit server parameters. That is why leaving them unset can have large effects.

Performance versus Jan​

The measured runs used an 8,192-token context, a 2,048 batch size and 100 GPU layers. Prompt caching was disabled so each run measured real prompt processing. The prompt asked for a roughly 400-word explanation of RAID 5 and RAID 6. The first response after loading was discarded as a warm-up, then three fresh-chat runs were recorded per model.

Jan v0.8.6 reached 123 and 82 tokens/sec on the same models. That puts Open WebUI about 6.2% and 8.5% slower. The reviewer cautions that the two apps used different llama.cpp builds, so this compares complete setups and not the inference engine alone. The V2 model's much longer reasoning output also explains much of its longer total time.

My read: a gap under 10% is unlikely to matter in daily chat. The setup effort is the bigger cost.

Tools can eat your context window​

This is the most useful finding for anyone running small context windows.

• Built-in tools. Using V2 VL High as a raw model, Open WebUI attached tool instructions and attempted a query_knowledge_files call. That pushed the request past 8,192 tokens. A Workspace preset with Builtin Tools set to None cut the actual generation prompt to 41 tokens.
• Presets. One GGUF can back several presets, each with its own tools, system prompt and parameters. So you can have a stripped-down chat profile and a web-enabled one without duplicating the model file.
• Web search. The reviewer used DDGS, which needed no extra API key. The 4B model found the Open WebUI releases page but didn't open it. It then wrongly claimed October 15, 2026 was in the future.
• The fix. The reviewer told the model explicitly to use fetch_url. The fetched page ballooned the request to 17,689 tokens. Setting Fetch URL Content Length Limit to 8,000 characters fixed it, and the model then correctly reported v0.12.2.

The lesson for small local models is to budget context for tool output. Don't assume the model will chain tool steps on its own.

Image input and local API​

Jan V2 VL High found its multimodal projection file automatically. Given an nvidia-smi screenshot, it correctly returned the GPU model, driver, VRAM, utilization, temperature and power draw. That is one qualitative test, not a measure of general accuracy.

With API keys enabled, a PowerShell query to /api/models on 127.0.0.1:8080 returned both raw GGUF models and all three presets. A POST to /api/chat/completions using a preset returned a correct answer. One Unicode dash displayed garbled in the console, but the generated answer itself was fine. So presets are usable from scripts and not only from the chat window.

Who should care​

Open WebUI Desktop suits people who want chat, tools, search, files, presets and an API in one local install. It is a heavier tool than a basic chat app. The reviewer's verdict is that the performance difference was small while the setup difference was large. Once configured, it supported a much broader range of workflows.

If you just want fast local chat on an NVIDIA card, Jan was easier in this test. If you want the extra features, plan for some manual llama.cpp and CUDA work. Pin the runtime, check the DLLs, set the context size explicitly, and use presets to keep tools out of the way when you don't need them.

The caveat is that this is one workstation with a high-end 32GB GPU, two models and a single prompt. Your results on consumer hardware may differ. StorageReview says these numbers will feed its Best Local LLM Tools leaderboard, so more comparisons should follow.