Jan v0.8.6 on Windows: Local LLM Benchmarks, GPU Fixes and OpenAI-Compatible API

What makes this particular update cycle worth a closer look for Windows power users and system administrators isn't just the friendly chat interface—it is what changed under the hood. With the back-to-back releases of Jan v0.8.5 on October 8, 2026, and the v0.8.6 hotfix on October 9, 2026, the developers fundamentally altered how the app packages its inference engine, manages GPU memory offloading, and secures its local server.

Under the Hood of Jan v0.8.5 and v0.8.6: What Actually Changed?​

Before diving into the benchmark numbers, it helps to understand why StorageReview had to scrap its initial test pass and re-run its benchmarks on v0.8.6. According to Jan’s official release notes for v0.8.5 and v0.8.6, this update cycle introduced sweeping architectural changes to how the desktop client runs local models:

• Bundled llama.cpp Engine: Earlier versions of Jan downloaded a llama.cpp backend separately during first run or updates. Starting with v0.8.5, llama.cpp (version 0.5.0, build b11146) is bundled directly inside the installer and runs in its own isolated worker process. If the inference engine crashes, it no longer drags the entire desktop app down with it. On Windows, NVIDIA cards use a bundled CUDA 13 build, while AMD, Intel, and older NVIDIA cards (such as Pascal GTX 10-series GPUs that cannot run CUDA 13) fall back to Vulkan.
• Automatic GPU Offload & Parallel Sequences: When upgrading from earlier builds, v0.8.5 strips out old default n_gpu_layers: 100 entries in model.yml so GPU offloading can be decided automatically, and changes Parallel Sequences from 1 to 0 so llama.cpp can choose its own slot count.
• Conversation Prompt Caching: By default, Jan now enables Remember Conversations Between Sessions, storing each thread's prompt cache on disk inside llamacpp/thread-cache (capped at 8,192 MiB). Users who want clean, uncached runs or want to save SSD space can toggle this off under Settings > Model Providers > Llama.cpp.
• Windows CLI & PowerShell Changes: The desktop installer no longer bundles the jan command-line tool inside the app's resources\bin folder (moving it to a separate install in %USERPROFILE%\.local\bin), and the agent's bash tool was renamed to shell, running natively in a sandboxed Windows PowerShell environment rather than requiring Git Bash.
• Why v0.8.6 Followed Immediately: Released just a day after v0.8.5, the v0.8.6 hotfix resolved several regressions. For Windows users on older hardware, the most critical fix restored CUDA engine support for NVIDIA Turing GPUs (GTX 16xx and RTX 20xx series) running drivers older than the R595 branch, which had failed with PTX toolchain errors in v0.8.5. It also fixed a severe slowdown on 12 GB GPUs when running Qwen3.8 with Multi-Token Prediction (MTP) by forcing Parallel Sequences to reserve one slot instead of four when set to auto, and ensured images returned by Model Context Protocol (MCP) tools are passed to vision models as actual images rather than raw base64 text.

Section Summary: The v0.8.5/v0.8.6 updates bake llama.cpp 0.5.0 directly into the Windows installer, introduce disk-based prompt caching, and fix critical GPU regressions for Turing cards and multi-slot MTP models.

Windows 11 Installation, Hardware Requirements, and Model Hub​

StorageReview conducted its testing on an HP Z4 Rack G5 workstation running Windows 11 Pro for Workstations, outfitted with a 16-core Intel Xeon w5-2465X, 128 GB of RAM, and an NVIDIA RTX 5000 Ada Generation GPU carrying 32 GB of VRAM (running NVIDIA driver 597.06 with CUDA 13.2 compatibility).

While that is serious workstation iron, you don't need an enterprise rack to get started. According to Jan’s official Windows documentation, the baseline system requirements are much more modest:

• OS: Windows 10 or higher
• CPU: x64 processor with AVX2 support (Intel Haswell 2013+ or AMD Excavator 2015+) or Windows on ARM ( via the dedicated ARM64 installer)
• RAM: 8 GB minimum (16 GB recommended)
• GPU: 6 GB VRAM minimum across NVIDIA, AMD, or Intel Arc GPUs
• Storage: At least 10 GB of free space, with local data, models, and logs stored by default in ~\Users\<YourUsername>\AppData\Roaming\Jan\data

On the HP test rig, installation via the standard Windows .exe installer required no user account creation to run local models. Because the workstation already had current NVIDIA drivers installed, Jan v0.8.6 automatically recognized the Xeon CPU, 128 GB of system memory, and the RTX 5000 Ada GPU in its Hardware settings page, enabling GPU acceleration right out of the box without manual CUDA configuration.

Upon first launch, Jan prompts users regarding anonymous telemetry and offers a one-click download of its recommended starter model, Jan V3.5 (a 2.8 GB download). Beyond the starter prompt, Jan’s built-in Hub connects directly to Hugging Face repositories, displaying file sizes, quantization formats, capability tags (such as Multimodal or Tools), and a color-coded hardware-fit badge before you waste bandwidth downloading a massive file. On the 32 GB RTX 5000 Ada, both the Jan V3.5 4B (Q4_K_M listed at 2.5 GB in the Hub) and the vision-capable Jan V2 VL High (Q4_K_M, 4.7 GB) earned green Fits badges. For hybrid workflows, Jan also lets you plug in cloud API keys for OpenAI, Anthropic, Gemini, Groq, and Mistral right alongside your local llama.cpp models.

Section Summary: Jan installs cleanly on Windows 10 and 11 (both x64 and ARM64) without requiring an account, automatically detects existing GPU drivers, and uses an integrated Hugging Face Hub to flag whether models will fit into your system's VRAM.

Local Inference Benchmarks: Speed, VRAM, and Caching Quirks​

To see how the bundled llama.cpp 0.5.0 engine performs, StorageReview tested two models on the RTX 5000 Ada: Jan V3.5 4B (Q4_K_XL) and Jan V2 VL High (Q4_K_M).

One important testing nuance: because StorageReview upgraded an existing installation to v0.8.6, their saved model profiles retained 100 GPU layers (n_gpu_layers), an 8,192-token context size, and a batch size of 2,048. Rather than wiping those profiles, they left the 100-layer setting intact, disabled Web Search, discarded one warm-up run per model, and recorded the median across three fresh threads using the prompt: "Explain RAID 5 and RAID 6 to someone building their first NAS. Include the main advantages and disadvantages of each, in about 400 words."

A few practical takeaways jump out from these numbers:

1. UI Metrics Match the Engine: Jan’s on-screen token-per-second counter aligned almost identically with the raw llama.cpp timing logs (123 tok/s vs. 122.55 tok/s, and 82 tok/s vs. 82.30 tok/s).
2. Watch Out for Reasoning Tokens in Totals: Why did Jan V2 VL High report 1,728 output tokens when asked for a 400-word essay? Because V2 VL High is a reasoning-capable model that thinks through problems before outputting final prose. Its visible answer was roughly 400 words, meaning Jan combines internal reasoning tokens and visible response tokens into a single total—so you cannot compare raw output token counts across reasoning and non-reasoning models.
3. Prompt Caching Skews Prefill Benchmarks: StorageReview omitted prefill (prompt processing) speed medians because v0.8.6’s new disk-based conversation prompt caching was active; on repeated runs, the engine recognized the cached prompt and evaluated only a single token. That is great for real-world responsiveness, but anyone benchmarking hardware on Jan v0.8.6 needs to disable Remember Conversations Between Sessions first.

Section Summary: On an RTX 5000 Ada, Jan v0.8.6 delivered 123 tokens/sec on a 4B parameter model using under 5 GB of VRAM, and 82 tokens/sec on a multimodal reasoning model using roughly 8.2 GB of VRAM.

Multimodal Vision, Web Search, and the Local API Server​

Can a local desktop app actually replace everyday utilities beyond simple text generation? StorageReview tested three of Jan's headline power features:

1. Multimodal Image Extraction​

In an earlier build of Jan, feeding an nvidia-smi terminal screenshot to Jan V2 VL High resulted in the model burning excessive time analyzing the ASCII layout of the terminal box. Retesting on v0.8.6 showed a much cleaner visible result: when asked to extract six specific metrics from a fresh nvidia-smi capture, V2 VL High accurately pulled the GPU name (NVIDIA RTX 5000 Ada Generation), driver (597.06), total VRAM (32,760 MiB), utilization (1%), temperature (35C), and power draw (4W / 250W). It still chewed through 21 seconds of internal reasoning and 1,979 total output tokens under the hood, but the final answer presented in the chat window was crisp and accurate.

2. Web Search and MCP Quotas​

When Web Search was toggled on in the chat bar and V2 VL High was fed the same NAS RAID prompt, the model autonomously decided—without being told to browse the web—that it should verify current NAS best practices online. However, the search failed to execute because the configured Exa search integration had hit its free Model Context Protocol (MCP) rate limit. The lesson here is straightforward: Jan’s local models know how to call external tools on their own, but if you rely on live web fetching, you will need to configure a search provider with an API key and sufficient quota.

3. OpenAI-Compatible Local API Server​

For developers and tinkerers who want to hook local models into VS Code extensions, PowerShell scripts, or custom local agents, Jan includes a built-in OpenAI-compatible API server accessible from the Settings menu. Starting the server with Jan V3.5 4B set as the default model spins up an endpoint at:

[url]http://127.0.0.1:1337/v1[/url]

StorageReview verified the endpoint from Windows PowerShell by querying /v1/models (which properly returned the installed model list) and firing a chat-completions request to Jan-v3.5-4B-Q4_K_XL asking for a one-sentence definition of RAID 6. The plumbing worked flawlessly, returning a response directly into the PowerShell console—though the 4B model stumbled on storage terminology, claiming RAID 6 uses two parity blocks "per data block" rather than per stripe. Small four-billion-parameter models are fast and light on VRAM, but they will still hallucinate technical minutiae if you don't keep an eye on them.

Furthermore, if you plan to expose Jan’s API server beyond 127.0.0.1 by binding it to 0.0.0.0 so other PCs on your network can reach it, note a major security hardening change introduced in v0.8.5 (tracked as GHSA-x6p8-7cp8-c3p6): Jan now enforces a Trusted Hosts allowlist. Requests arriving with a Host header that isn't localhost, host.docker.internal, a private LAN IP, or an explicitly listed hostname under Settings > Local API Server > Trusted Hosts will be rejected with a 403 error.

Section Summary: Jan v0.8.6 accurately extracts text data from screenshots and serves models over a local OpenAI-compatible port (127.0.0.1:1337/v1), though users must watch out for third-party web search rate limits and small-model hallucinations.

Known Windows Caveats and Final Verdict​

While StorageReview’s six measured test runs completed without a single crash or model-load failure, Windows administrators should keep a few documented v0.8.5/v0.8.6 release notes caveats in mind:

• Sandbox Restrictions on the shell Tool: On Windows, Jan runs its agent shell tool inside an app-container sandbox using PowerShell. Because of those sandbox permissions, git currently cannot read the workspace, MSI-installed tools like Node.js that block app-container access may fail to launch, and enterprise PCs enforcing PowerShell Constrained Language Mode will block the shell tool entirely.
• Silent Update Failures: If an in-app update fails, the update dialog currently closes without displaying an on-screen error message (#9137), forcing you to check the application logs to see what went wrong.

Even with those minor growing pains, Jan v0.8.6 hits a sweet spot for Windows 10 and 11 users. By bundling llama.cpp 0.5.0 directly into the installer, offering clear VRAM fit indicators in the model Hub, and exposing granular controls like KV-cache offload, context length, and GPU layers right in the GUI, it strips away the command-line friction of local AI without dumbing down the toolkit. Whether you are testing vision models on an RTX workstation or wiring a lightweight 4B model into a local PowerShell script, Jan makes running local LLMs on Windows remarkably painless.