Lemonade and Crush Test Exposes Local AI Freshness Limits

XDA’s experiment was small—13 questions—and its measurements describe one home-lab deployment, not local language models as a class. No independent reproduction of these measurements is established by the available reporting. Nevertheless, the experiment separates several failures that can look identical in a terminal: a model that does not know, a model that confidently recalls obsolete information, and an assistant whose supposedly available search tool never produces a usable answer.

That distinction has also appeared in engineering commentary. In “Knowledge Cutoff Is a Silent Production Bug,” Tian Pan distinguishes fabricated information from temporal staleness: information that once described the world correctly but is presented as current after circumstances have changed. That is useful framing for this case, although the commentary does not independently confirm XDA’s tests.

Lemonade’s 13-question test exposed a freshness problem​

The initial setup paired Lemonade, which served the local model, with Crush, the terminal-based interface through which the author worked. XDA tested questions both directly against Lemonade and through a self-hosted AI-search interface, using the same system instruction to acknowledge uncertainty. The questions covered recent releases, changing facts, exact version numbers, stable knowledge and deliberately invented products or commands.

Without search, the model refused or acknowledged uncertainty on 11 of the 13 questions, XDA reports. It did not recognize Lemonade Server or Crush, despite those being components of the environment serving and displaying its answers. That is a particularly clear illustration of the gap between running inside a software stack and possessing information about that stack: in this test, the surrounding applications did not make their identities or release histories available as usable knowledge.

The conspicuous fabrication concerned qm autoscribe, a Proxmox subcommand invented for the test. According to XDA, the model described a helper that read a virtual machine’s configuration and generated a human-readable description in Markdown, HTML or plain text. The answer was detailed and formatted, but the command was a test fabrication. By contrast, the model rejected the invented Firewalla router and Crush flag.

Those outcomes should not be flattened into an accuracy score. Refusing a nonexistent command is a useful result; refusing a legitimate question about current software is an honest but unsuccessful answer. Both avoid fabrication, but only one completes the intended task. For an administrator choosing an assistant, the distinction determines whether the tool can provide practical help rather than merely avoid obvious mistakes.

XDA also reports that the model identified its knowledge cutoff as June 2024 when asked about a later phone launch. That is a reported model response, not independent verification of its training history. The stronger evidence of staleness comes from the actual behavior: unanswered questions about current tooling and a later response that described Proxmox VE 8.2 as the current stable release.

The test therefore supports a narrow conclusion with broad practical relevance. This deployment’s main obstacle was access to current information, while fabricated commands remained a separate hazard. It does not establish that hallucination is rare across local models, or that every unanswered question can be traced to one precise cutoff date.

Vane added retrieval, but the installation was only one boundary​

XDA chose the self-hosted answering engine now called Vane, previously Perplexica, to supply web retrieval. The report dates the rename to March 2026. A discussion in the project’s GitHub repository contains a community member’s reproduction of a March announcement explaining that the repository and codebase had been renamed while documentation, branding and Docker images were being updated. That supports the existence of a transition, although it is not an independently inspected original announcement.

There is firmer first-party documentation for the integration’s earlier foundations. The maintainer’s October 21, 2025, announcement for version 1.11.0 introduced a setup wizard, single-command Docker installation, a redesigned configuration system and Lemonade as a supported provider. The historical record matters here because these were established capabilities before the reported rename, rather than features that can safely be attributed to the name change itself.

For the September experiment, XDA created a Debian LXC container on a Proxmox host, installed Docker and ran Vane with persistent storage and port 3000 exposed for its interface. The report says that this deployment bundled SearXNG, the metasearch component that queried external search engines. A January project discussion also contains startup logs showing SearXNG launching inside the earlier Perplexica deployment, corroborating that bundled architecture.

Connecting the answering engine to the local model required another boundary to be correct. According to XDA, Vane’s Lemonade provider needed an endpoint ending in /api/v1; without that suffix, the request reached Lemonade’s web interface, and the wizard saved an unusable connection. That is an observation about this integration, not a universal endpoint recipe for every model provider. It explains why successfully opening a setup page did not demonstrate that model requests would work.

The terminal workflow added a further complication. XDA reports that community Perplexica MCP servers supplied ordinary provider and model names, while the Vane API it was using expected a provider UUID and a model key. MCP—the Model Context Protocol—was the tool-integration boundary here, but the specific identifier mismatch was inside the connection to the answering engine. A tool can be exposed to a client and still send requests that the service behind it cannot use.

The author wrote a small Python bridge exposing a web_search tool, then encountered what XDA describes as an MCP Python SDK compatibility problem involving a 2.x release and the FastMCP interface. Pinning the dependency below 2.0 reportedly restored compatibility. The precise package version and bridge code are not supplied, so this is insufficient evidence for a general instruction to downgrade the SDK. The supported lesson is more specific: the client, bridge, SDK and Vane request format all had to agree before terminal search could function.

SearXNG made external search availability the bottleneck​

The next failure occurred after the components were connected. XDA reports that searches initially returned nothing because the external engines queried through SearXNG were blocking or failing requests. The author inspected SearXNG’s engine configuration and tested engines individually, separating upstream search availability from the behavior of the language model.

The reported outcomes were uneven:

These are observations from the author’s host, not established availability rules for each service. An HTTP 200 response in the Mojeek comparison also does not, by itself, demonstrate that Vane received a complete, usable search result. The comparison was useful diagnostically because request handling changed with the user agent; it is not a supported recommendation to bypass a provider’s restrictions.

Bing was the only working engine in that initial configuration, according to XDA. The author then observed suspensions lasting about three minutes after a small number of queries, while Vane expanded a question into approximately three searches. That multiplication makes the end-to-end constraint easy to understand: one user question can consume several upstream requests before an answer is attempted.

This is a concrete Microsoft dependency, but it should not be confused with Microsoft’s documented Bing connector limits. The connector documentation describes a separate, authenticated integration with its own throttling rules. Those limits do not establish the cause or threshold of SearXNG’s reported public-search failures. The relevant evidence here remains the behavior XDA observed, rather than a rate limit borrowed from a different Microsoft service.

The experiment also changes the meaning of “local.” The model could remain locally served while the search component sent queries to external providers. Thus, adding web retrieval did not preserve the initial setup’s description of “nothing leaving my network.” That conclusion follows from the reported architecture; it does not establish that complete conversations or unrelated local files were transmitted. For an organization, the actual search queries become a data-flow boundary to evaluate.

XDA eventually supplied a Brave API key and reports that search then returned 26–55 sources per query in about 12 seconds. The author judged those results more useful than the earlier Bing results because they included Proxmox’s wiki downloads page and an upgrade guide rather than a marketing homepage. This was a successful change in that deployment, not a general search-quality ranking. It removed the observed retrieval bottleneck without resolving the model’s ability to use what came back.

Lemonade performance faults looked like failed search​

Several local problems produced the same visible symptom as blocked engines: an unanswered or apparently stalled query. XDA reports that Lemonade had automatically loaded the model using Vulkan rather than ROCm, with throughput of 10.7 tokens per second instead of approximately 33. Reported VRAM usage also differed sharply, at about 0.2GB versus 6.7GB.

Those measurements identify a meaningful configuration difference in the author’s setup, but they are not a general Vulkan-versus-ROCm benchmark. The hardware, complete model configuration and measurement method are not established sufficiently to turn them into a platform recommendation. In particular, a reader using a different GPU or running a Windows installation should not treat this Linux home-lab result as a universal instruction to switch back ends.

What the measurements do show is the danger of diagnosing only from the final interface. A search request can appear to stall because the upstream engine rejected it, or because the model handling the task is running much more slowly than expected. The fact that both end with no timely answer does not make them the same fault.

Even after improving inference speed, the author encountered excessive output. XDA reports that a reasoning model ran beyond 15,000 tokens without an output cap, causing a timeout. Tokens are the units used to account for model input and generated text; in this experiment, an unusually long generation consumed time without delivering the expected bounded answer. Faster generation alone did not make the request reliably complete.

The report also mentions an environment variable that reverted when a new terminal opened, but does not identify it or provide a verified persistence fix. That missing detail prevents a responsible shell-specific repair procedure. It still adds an important boundary to the diagnosis: a connection that works in one terminal session may not demonstrate that the next session has the same configuration.

Taken together, these failures argue for checking the stages separately. XDA’s experience distinguishes model execution, search-engine responses and terminal-session state, even though the interface made their failures look similar. The practical success condition is a completed answer through the intended client—not a running container, a reachable setup wizard or a model that responds when queried directly.

Vane’s search tool was available more often than it was used​

The most revealing comparison changed the prompt rather than the infrastructure. XDA asked a Proxmox-version question without search, with search available and with search explicitly requested. The three reported outcomes were substantially different.

The third row is evidence of a changed answer after retrieval, not independent verification here of Proxmox’s current release. XDA described it as correct, but an authoritative Proxmox release record is not established in the available corroboration. An administrator should not use a model’s quoted ISO filename in this experiment as an upgrade instruction.

The behavioral result is better supported within the experiment. Across the 13 questions, XDA reports that the model searched on seven when the tool was merely available and on 12 when told to use web search. Explicit wording increased observed retrieval, but did not make it universal. That makes “use web search” a useful prompt intervention in this case, rather than an enforceable guarantee that every time-sensitive question will be checked.

Nor did successful retrieval settle the integration problem. After adding the Brave API key, XDA tested four models and reports that Vane’s research agent still could not obtain consistently usable structured tool calls. These are machine-readable requests that the surrounding software must interpret as actions; printing text that resembles such a request is not the same as successfully executing it.

This is not a model leaderboard. The report does not establish enough about hardware, quantization, prompt templates, model revisions or repeatability to rank the models beyond this workflow. It shows four distinct failure modes that survived the repair of upstream search availability. The earlier successful Proxmox response and the later research-agent failures also caution against treating one successful query—or a working web interface—as validation of every execution path.

One Coder-model response was useful precisely because it stopped short of certainty. XDA reports that it found conflicting Perplexica and Vane image references and declined to identify the current image conclusively. That was a reasonable handling of conflicting evidence, although the existence of old and new names would not make every confident answer wrong. Establishing which image is current requires an authoritative record; counting search results that repeat either name cannot settle the issue.

Local AI assistants need acceptance tests for current work​

Treat a local assistant intended for current administration as unproven until it completes representative tasks through the interface you actually use. The XDA experiment supplies a useful evaluation pattern: test recent release information, changing names, exact commands, stable facts and deliberate fakes separately. That reveals whether a model is missing knowledge, declining appropriately, inventing functionality or failing to invoke retrieval.

The decision is not simply whether to enable search. A user asking stable conceptual questions may not encounter the same obstacles as an administrator requesting a current release number. For the latter task, this experiment shows why freshness and evidence handling belong alongside answer quality when deciding whether the assistant is ready to rely on.

The following checks follow directly from the reported failures, without assuming an undocumented fix:

• Include both genuine current-version questions and deliberately nonexistent commands in an evaluation, and score useful answers separately from correct refusals.
• For freshness-sensitive questions, explicitly request retrieval and establish whether it occurred; XDA’s prompt change improved tool use but did not guarantee it.
• Test the complete Crush-to-search-to-model path if that is the intended workflow, rather than accepting a working Vane web interface as equivalent.
• When a query stalls, distinguish external search rejection from model execution speed, excessive output and session-dependent configuration before changing unrelated components.
• Match bridge code to the actual Vane API and SDK versions in use; the reported below-2.0 workaround is not sufficient evidence for a blanket dependency downgrade.
• Treat externally submitted search queries as an outbound data flow, and do not assume that locally hosted inference makes web-assisted answers wholly private.

The strongest result of this home-lab experiment is a more demanding definition of a working assistant. It must obtain relevant evidence, invoke the right tool, interpret the result and finish within a useful time—not merely avoid fabricating an answer. For current software administration, the next concrete decision is whether the exact deployed workflow passes those checks; until it does, its release numbers and command recommendations remain leads to verify, not instructions to execute.