Local LLM Web Search on Windows: Jan and LM Studio Setup for 8GB GPUs

For Windows power users and local AI enthusiasts, an obvious question arises: if we can already run capable Large Language Models (LLMs) on consumer graphics cards to summarize documents or replace sluggish desktop file indexing, why not hand our daily web searches over to a local model as well? A recent week-long experiment documented by Nolen Jonker at XDA Developers tested exactly that premise—routing everyday questions through a local LLM running on an 8GB GPU with live web access instead of defaulting to Google. The results highlight both how remarkably far desktop AI tooling has come and the sharp technical ceilings you hit when a compact model tries to make sense of the live internet.

Why Traditional Search Feels Like a Treasure Hunt​

Frustration with mainstream search engines isn't merely nostalgia; it is backed by empirical research. Jonker points to a year-long study out of Germany that tracked 7,392 product-review queries across Google, Bing, and DuckDuckGo. That academic paper—Is Google Getting Worse? A Longitudinal Investigation of SEO Spam in Search Engines—monitored "best [product category]" searches between August 2022 and September 2023. The researchers found that the majority of top results in that product-review genre relied on affiliate marketing, and that all three search engines were vulnerable to large-scale affiliate-link spam campaigns. While the researchers noted that search updates occasionally improved conditions temporarily, low-complexity SEO spam remained a persistent fixture. (It is worth keeping the study's scope in mind: the authors specifically examined product-review queries and cautioned against treating that single commercial battleground as a measurement of every query type on Google.)

Layered atop the SEO arms race are AI Overviews. While built-in AI summaries can be convenient, they frequently synthesize personal opinions from community forums as objective fact, forcing careful readers to scroll down and verify the underlying links anyway. Worse yet, the commercial pressure hasn't stayed out of the summaries. According to Google's own May 2025 Ads announcement, Search and Shopping ads within AI Overviews expanded to desktop users in the United States, alongside tests placing ads below and integrated into AI Mode responses. When the top of the page is dominated by commercial placements and questionable summaries, users start looking for an exit ramp.

Giving a Local Model Eyes on the Web​

Running an LLM locally on your PC is fantastic for privacy and offline explanation, but an unassisted model is frozen in time at its training cutoff. In Jonker's testing, asking a local Gemma E4B model for the current date resulted in a confident declaration that the year was 2016. And when you are working with an 8GB graphics card, you are restricted to relatively small, quantized models that lack massive encyclopedic parameter counts.

To turn a small local model into a viable research assistant, you have to give it tool-calling access to the web. There are two primary ways PC users are accomplishing this right now:

• Jan Desktop: In the XDA experiment, Jonker used Jan, noting that enabling web access in their setup involved flipping a settings toggle and installing the Jan browser extension, which defaulted to searching through Exa on a free keyless endpoint. Looking at Jan's current official desktop documentation, the workflow has become even more streamlined: built-in web search is now native and turned on by default with no browser extension, API key, or Model Context Protocol (MCP) server required. Users can simply toggle the globe icon in the chat input toolbar.
• LM Studio with MCP: For users who prefer LM Studio, tool use can be wired up through the Model Context Protocol. Jonker configured LM Studio to query Brave Search—which maintains its own independent web index rather than relying on Google or Bing—by registering for a Brave Search API key and editing the mcp.json configuration file. LM Studio's developer documentation confirms that versions 0.4.0 and newer support both per-request ephemeral MCP servers and persistent servers defined in mcp.json, even allowing users to restrict specific tools via an allowed_tools parameter to save prompt processing time.

How Search and Fetch Actually Work​

Whether you use Jan or LM Studio, giving a local LLM "web access" actually relies on two distinct operations. In Jan's architecture, these are advertised to tool-capable models as web_search and web_fetch:

According to Jan's documentation, while Exa serves as the default keyless provider out of the box, users can also switch the backend in Settings > Web Search to Tavily (API key required), You.com (keyless with a daily cap, or keyed for extracted content), or a self-hosted SearXNG instance pointed at a local JSON API URL.

A Crucial Privacy Caveat: Based on how this client-server architecture works, Windows users should remember that "local LLM web search" does not mean your query stays entirely inside your bedroom PC. While the neural network inference runs locally on your GPU, the search terms themselves are still transmitted to whatever external search provider (Exa, Brave, You.com) you configure, and fetching a webpage makes a network request to pull that site's text. If you want to minimize external tracking, pointing Jan or an MCP server at a self-hosted SearXNG container is the way to go.

Life on an 8GB GPU: Model Selection and Hallucinated Decades​

When your hardware budget is capped at 8GB of VRAM, model choice dictates whether your local assistant actually uses the tools you give it. A model must natively support tool calling, and it needs the discipline to invoke a search automatically when asked about current events.

In Jonker's week-long test, Google's Gemma struggled to initiate searches without explicit prompting. By contrast, Alibaba's Qwen family proved reliable at autonomous tool calling: Jonker settled on Qwen 3.5 4B (Q4_K_M quantization) for quick, lightweight queries and Qwen 3.5 9B (Q4_K_M) for heavier lifting. When asked a question, Qwen automatically triggered Jan's search tool, selected relevant pages to fetch, and displayed a progress bar as it digested the text—a process that typically took just a few seconds and clearly showed whether the model read the full page or relied solely on snippets.

However, giving a 4-billion or 9-billion parameter model live web text does not magically cure hallucinations. In fact, it can create brand-new failure modes when the model tries to synthesize messy search results:

• The Time-Traveling Changelog: When Jonker asked Qwen a vague question—"what Obsidian released this month"—the model pulled ten different sources, nine of which were third-party community plugin changelogs and only one of which came from Obsidian's official site. Blending those loosely related pages together, Qwen confidently dated the update to October 2035, placed 2045 in a subheading, and cited a version number that did not exist anywhere.
• The Context Window Crunch: Fetching real web pages devours memory. Even with Jan truncating individual pages, Jonker observed that a single multi-source web search consumed roughly 70% of a 30,000-token context window. On an 8GB GPU, allocating a 30k context window alongside model weights is a tight balancing act between quantization level, parameter count, and VRAM limits.

Because Jan neatly stacks the fetched source URLs at the bottom of every response, recovering from that Obsidian hallucination was simple: Jonker could immediately spot that the model had ingested third-party plugin pages and click directly through to the single official source.

How to Optimize a Local LLM Search Workflow on Your PC​

If you want to test this setup on your own Windows rig without blowing up your VRAM or getting tricked by synthetic hallucinations, the lessons from this experiment translate into a few practical ground rules:

1. Pick a Tool-Trained Model First: Verify that your chosen GGUF model explicitly advertises tool-calling support in Jan or LM Studio. Small Qwen builds (4B to 9B at Q4_K_M) are a practical starting point for 8GB GPUs.
2. Write Specific, Bounded Prompts: Small models struggle to filter out noisy search results when given vague prompts. Instead of asking "What did Obsidian release this month?", specify the exact product, target timeframe, and source type: "Search for the official Obsidian desktop app changelog and list the core features added in the latest stable release."
3. Watch Your Context Allocation: If you set a 24k or 32k context window to accommodate web_fetch payloads, monitor your GPU memory usage in Windows Task Manager. If inference slows to a crawl because context spills into system RAM, drop down to a smaller parameter model (like a 4B variant) or limit the number of pages the model fetches per turn.
4. Treat the LLM as a Router, Not an Oracle: Always inspect the stacked citation links under the generated answer for version numbers, dates, and technical procedures.

The Verdict After One Week​

Did a local LLM completely kill off Google after seven days? Not quite. Conventional search engines remain unbeaten for pure navigational tasks—like finding an account login page—or hyper-local, real-time checks like seeing which neighborhood restaurants are open right now.

What changed was the starting line for informational questions. Because a small local model has limited context and requires clear direction to avoid blending bad sources, using it forces you to slow down and formulate thoughtful, complete questions rather than lazily typing fragmented keywords and clicking the first link below the ads. For PC enthusiasts tired of wading through sponsored clutter, trading a bit of prompt engineering for an ad-free, source-linked local workspace is a bargain well worth testing.