The Berkeley- and UT Austin-affiliated team behind the August 17 preprint has published the engine under Apache-2.0 through its FlashML GitHub organization. Its paper describes a different approach to local AI serving: store the full expert pool in host memory, use VRAM as a changing cache rather than a complete model store, and decide on every generation step whether missing experts should be copied over PCIe or computed by the CPU.
That approach targets a genuine problem with modern MoE models. A model may activate only a small fraction of its parameters for each token, but its inactive experts still need to be available somewhere. Traditional local runtimes tend to pick a fixed split between CPU RAM and VRAM when the model is loaded. FreeToken instead treats the GPU, host memory, CPU cores, and PCIe bus as one pool of uneven resources whose best division changes by machine and workload.
The headline is a memory-management claim, not a cheap-GPU miracle
The paper’s largest demonstration is GLM-5.2, described as a 753B-parameter MoE model with roughly 40B active parameters per token. FreeToken reports 14.9 tokens per second on an RTX PRO 6000, compared with 7.3 tokens per second for llama.cpp in its test. That is an impressive result if it holds across independent tests, but it should not be read as proof that a 96 GB graphics card alone can host a 753B model.
The full model checkpoint must remain accessible in host memory. The reported workstation setup pairs its GPU with 512 GB of DDR5 memory; the paper’s 284B-model desktop configuration uses 192 GB. Those are workstation-class system-memory requirements even if only one GPU is installed. Storage also matters: loading and pinning a checkpoint measured in hundreds of gigabytes is a different operational task from downloading a 7B or 14B local model onto a gaming PC.
This is still a useful shift in capability. For years, the practical constraint for local model serving has been “can the entire model fit in VRAM?” FreeToken’s answer is that a sparse MoE model can run without that condition, provided the model is arranged in host memory and the machine has enough bandwidth to retrieve or execute the routed experts quickly enough.
The hardware bill has moved from a multi-GPU requirement toward an unusually memory-heavy single-workstation design. That will matter to teams deciding whether to buy one large accelerator with 256 GB or 512 GB of system RAM, or several commodity GPUs. It does not turn a standard 16 GB desktop into a 753B-model server.
FreeToken’s main idea is to avoid wasting either PCIe or CPU bandwidth
FreeToken calls its policy bandwidth-adaptive execution, or q-star. When the model router asks for experts that are not already in the GPU cache, the runtime can do one of two things: transfer the missing experts to VRAM and execute them on the GPU, or run those experts directly on the CPU from system memory.
Both paths contend for host-memory bandwidth. Saturating PCIe can starve CPU-side expert execution; sending all work to the CPU leaves GPU compute and the PCIe path underused. FreeToken profiles the machine with ft bench bw, then splits cache misses between the two paths based on measured CPU and PCIe bandwidth instead of relying on a predetermined offload percentage.
That mechanism is technically more consequential than the headline model-size number. PCIe bandwidth varies widely across systems. A desktop RTX 5090 connected at full PCIe bandwidth behaves very differently from a laptop GPU wired through fewer lanes, or a desktop with a second device sharing chipset lanes. A static placement policy that performs acceptably on one configuration can fall apart on another.
The project also uses a shared least-recently-used expert cache spanning MoE layers during decode. Rather than reserving a “hot” set of experts at load time, it allows the cache to follow real router activity. For long prompts, FreeToken double-buffers layers during prefill so the next layer’s weights can stream while the GPU processes the current layer.
These are engineering choices aimed at the two forms of latency local-agent users actually notice: time to first token after sending a large prompt, and pauses during multi-step tool use. FreeToken additionally checkpoints selected agent-state boundaries around items such as tool calls, enabling a changed suffix of context to be processed without replaying all prior work.
The performance figures are promising, but mostly remain author-reported
In the paper, FreeToken reports 77–83 tokens per second for Qwen3.6-35B-A3B on an RTX 5090 and 22–25 tokens per second for DeepSeek-V4-Flash in MXFP4 format on the same GPU. Across the tested comparisons with llama.cpp, Ollama, and KTransformers, the authors report a 1.5× to 2.3× decode-throughput improvement and worst-case time-to-first-token below 44 seconds.
Those figures are detailed enough to be meaningful, and the paper’s reported speedup arithmetic is internally consistent. But the preprint is less than a week old, has not been peer reviewed, and has not accumulated a broad set of outside benchmarks. Performance claims should therefore be treated as the project team’s measurements, not as established rankings for local inference engines.
One early exception is a report from PureTensor AI, which said it reproduced the GLM-5.2 setup on matching RTX PRO 6000 and 512 GB DDR5 hardware. PureTensor reported a warm average of 16.9 tokens per second across three 512-token runs, above FreeToken’s reported 14.9 tokens per second. That is encouraging, but it is one reproduction on closely matched hardware rather than a cross-platform validation.
The KTransformers comparison also needs context. FreeToken’s paper limits baseline runs on rented servers to six CPU threads to better approximate smaller edge systems. That is a defensible design choice for an “edge” comparison, but KTransformers is specifically designed to exploit many-core CPU execution and AMX acceleration. Its number in this paper should be read as performance under FreeToken’s edge-host constraint, not as KTransformers’ best possible configuration.
Windows users should separate the desktop application from the documented server path
FlashML offers FreeToken downloads for Windows and Linux, and its GitHub documentation says the project supports OpenAI-compatible and Anthropic-compatible local API endpoints. The ft serve command listens on port 1919 by default, while ft launch can configure supported coding-agent clients including Claude Code, Codex, OpenCode, and OpenClaw to use the local service.
For developers wanting to point an existing client at their own machine, that API compatibility may be the most immediately useful feature. It means an agent harness designed around a remote provider endpoint can, in principle, be redirected to a locally hosted MoE model without rewriting the client integration.
However, the documented command-line installation path is currently Linux x86_64 with an NVIDIA GPU, driver r580 or later, CUDA 13 tooling, Python 3.10 or newer, and JIT-compiled CUDA kernels. The official install instructions do not document the same CLI setup for native Windows. The Windows offering is presented as a separate desktop installer with a graphical interface, while the primary server and developer workflow remains Linux-focused.
That distinction matters for WindowsForum readers. A Windows desktop download is not the same thing as documented support for every Windows workstation, WSL configuration, CUDA setup, scripting workflow, or enterprise deployment model. Administrators should not assume that a Windows app release makes FreeToken ready for unattended Windows Server deployments or that it has been validated across consumer laptop switchable-graphics configurations.
The realistic first use case is a private local agent server
FreeToken makes the strongest case for developers and small teams that already own an NVIDIA workstation and are paying recurring API costs for coding agents, batch evaluations, code review, or document analysis. A local endpoint also keeps prompts, source code, tool outputs, and generated data on hardware under the operator’s control, which can be valuable in regulated or IP-sensitive work.
It is less compelling as an all-purpose replacement for a datacenter serving stack. Large host-memory requirements, Linux-centric CLI support, limited independent benchmarking, first-use kernel compilation, and the operational burden of storing massive checkpoints all remain substantial barriers. Organizations that need high concurrency, multi-user isolation, audited model provenance, or service-level guarantees will need more than a fast single-machine runtime.
FreeToken’s contribution is practical: it turns host RAM and CPU capacity into active parts of an MoE inference system instead of treating them as slow overflow behind a GPU. On a properly provisioned NVIDIA workstation, that can make models previously dismissed as “too large for local” usable for interactive work. For Windows users, the next concrete test is whether FlashML’s desktop app matures into a fully documented Windows serving path—or whether the serious deployment target remains a Linux box beside the Windows workstation.