NVIDIA Nemotron 3.5 Bolt 74B: Run a 24GB Local AI Agent on Windows 11

The announcement targets three distinct tiers of computing, stretching from heavy-duty enterprise infrastructure down to local GeForce RTX desktops, workstations, and the newly arriving RTX Spark laptops. While marketing headlines inevitably lump "RTX PCs" into a single bucket, the technical reality—detailed across Windows Report's coverage, NVIDIA's official release notes, and NVIDIA Research's technical documentation—comes down to strict memory footprints, aggressive quantization math, and Mixture-of-Experts (MoE) routing.

If you are wondering whether your current Windows rig is about to host an autonomous digital coworker—or if you will need at least 24GB of memory just to get through the door—here is a breakdown of what NVIDIA actually shipped, how the compression works under the hood, and what it means for local Windows workflows.

The Three-Tier Lineup: From Data Center to Desktop​

Rather than pretending a single massive model can scale down to a laptop without melting the chassis, NVIDIA split the Nemotron 3.5 release into three specific models joining its existing Lightning (high-throughput) and Ultra (heavy reasoning) families.

Section Summary: NVIDIA's October 2026 Nemotron 3.5 rollout separates enterprise multimodal workloads (Super VL) from local hardware execution (the Bolt series), with Super VL and the 24GB Bolt model available immediately and the lightweight 4GB Bolt E4B arriving on October 23.

Nemotron 3.5 Super VL: Multimodal Muscle for Always-On Agents​

At the top of the stack sits Nemotron 3.5 Super VL, an open vision-language and reasoning model built with 120 billion total parameters, though its Mixture-of-Experts architecture activates just 12 billion parameters (120B-A12B) per token. Built in collaboration with the Nemotron Coalition, Super VL is designed for what NVIDIA characterizes as specialized, always-on enterprise agents.

Historically, building an AI agent that could read a messy PDF invoice, interpret a financial chart, inspect a live webpage, and then execute a database query required stitching together separate optical character recognition (OCR) pipelines, vision models, and text-based reasoning models. Every handoff between separate models risked losing context or introducing latency. According to NVIDIA's official blog, Super VL combines text and visual understanding inside a single system, ingesting images, webpages, PDFs, tables, and graphs directly so context stays intact from document ingestion to tool execution.

The Performance Claims and Open Ecosystem​

In its launch materials, NVIDIA claims that Super VL delivers up to five times faster throughput than the next-best open model in its class on the Artificial Analysis Intelligence Index leaderboard, while achieving accuracy comparable to NVIDIA's largest reasoning model at roughly one-fifth the parameter count. NVIDIA also claims that deploying the model via NVIDIA Dynamo allows organizations to serve up to eight times more users at 85% lower latency. As any seasoned systems administrator knows, vendor-supplied throughput and latency figures represent best-case scenarios on tuned hardware; your actual mileage will depend heavily on batch sizes, context lengths, and serving topology.

What makes Super VL particularly compelling for enterprise IT and security-conscious developers is its licensing and transparency. Unlike closed frontier APIs that lock organizations into proprietary cloud ecosystems, NVIDIA released Super VL with:

• Open weights available via Hugging Face, OpenRouter, the ModelScope community, and packaged as an enterprise-supported NVIDIA NIM microservice (which NVIDIA notes is refreshed every two weeks with security patches and optimizations).
• Training recipes and post-training pipelines accessible through NVIDIA NeMo and standard Python workflows.
• Open datasets and reinforcement learning environments (via NeMo Gym) on Hugging Face, covering agentic tool use, finance, long-context processing, and vision-language understanding.

NVIDIA reports that Super VL was trained on roughly 26 trillion tokens during pretraining and continued pretraining, followed by post-training on approximately 3.8 million samples and reinforcement learning across dozens of simulated environments. Major software platforms and enterprises are already customizing the model for domain-specific agents, including CrowdStrike (for agentic blue-teaming in enterprise security), Atlassian (for agentic search across Jira and Confluence), Salesforce (for secure government workflows), Perplexity, Synopsys, and Dassault Systèmes.

Section Summary: Nemotron 3.5 Super VL combines vision and text processing in a 120B-parameter (12B active) open-weight model, giving enterprises full control over weights, training recipes, and on-premises data residency for document-heavy agentic workflows.

Under the Hood of Bolt 74B-A9B: Fitting 74 Billion Parameters into 24GB​

For Windows power users, local AI tinkerers, and developers who prefer running models on their own metal, the standout announcement is Nemotron 3.5 Bolt 74B-A9B 24GB.

Anyone who has spent an evening calculating VRAM budgets in LM Studio or Ollama knows the brutal arithmetic of large language models: a standard 70-billion-parameter model quantized to 4-bit typically requires upwards of 40GB of memory just to load the weights, before you even allocate memory for the KV cache and desktop apps. That effectively locks out single-card desktop PCs, even those sporting flagship 24GB GeForce RTX 3090 or RTX 4090 GPUs.

How did NVIDIA cram a 74-billion-parameter model into a 24GB footprint while leaving room for Windows 11 and user applications to breathe? According to NVIDIA Research's technical report for Nemotron 3.5 Bolt 74B-A9B 24GB, the engineering team took a counterintuitive route: instead of training a smaller model from scratch and quantizing it to 4-bit, they started with the massive 120B-A12B Super VL model and crushed it down using structured architectural pruning and extreme 2-bit quantization.

1. Star Elastic Architecture Compression​

NVIDIA used an internal router-guided architecture search and compression framework called Star Elastic to prune the parent 120B-A12B hybrid Mamba-Transformer Mixture-of-Experts model down to 74 billion total parameters and 9 billion active parameters (reducing the active routed expert top-K from 22 down to 9). After pruning the network, NVIDIA ran additional post-training and end-to-end reinforcement learning to recover vision functionality and sharpen local tool execution.

2. Mixed Precision and 2-Bit (IQ2_XXS) Quantization​

Normally, dropping a model down to 2-bit precision turns its reasoning capabilities into digital soup. To prevent that, NVIDIA applied Quantization-Aware Distillation (QAD) alongside a mixed-precision scheme tailored to the model's hybrid Mamba-Transformer anatomy:

• FC1 and FC2 Expert Layers: Quantized to IQ2_XXS at 2.0625 bits per element. Because these routed and shared feed-forward layers make up the vast majority of the model's total parameter weight, shrinking them to ~2 bits yields massive memory savings.
• Mamba Linear Layers: Quantized to NVFP4 (4-bit floating point).
• Attention, Embeddings, LM Head, and Latent MoE Weights: Kept at a much higher Q8_0 (8-bit) precision to preserve accuracy and routing coherence.

3. LiLiCorr Speculative Decoding on Windows 11​

Because local MoE execution is heavily bottlenecked by memory bandwidth, NVIDIA also trained a 1.13-billion-parameter parallel speculative decoding drafter called LiLiCorr. Rather than drafting tokens independently or correcting them one by one, LiLiCorr scores token candidates across an entire draft in a single lightweight pass.

In NVIDIA Research's published benchmarks—tested using llama.cpp on Windows 11 ARM x64 running on an RTX Spark system (specifically a Surface Laptop Ultra with 128GB of memory)—Bolt 74B-A9B with the LiLiCorr drafter hit 83.99 tokens per second (TPS) on the AIPerf SPEED-Bench-Coding test. On public agentic leaderboards comparing open models that fit inside a 24GB memory envelope, NVIDIA states that Bolt 74B-A9B ranks #1 on PinchBench, #2 on WebArena, and #2 on BrowseComp.

Section Summary: By pruning Super VL down to 74B total (9B active) parameters and using Quantization-Aware Distillation to compress expert layers to 2.0625 bits (IQ2_XXS), NVIDIA squeezed a high-accuracy hybrid Mamba-Transformer model—complete with a speculative decoding drafter—into a 24GB memory budget.

Nemotron 3.5 Bolt E4B: The 4GB Edge Contender​

Of course, as industry observers at The Decoder rightly pointed out, a 24GB memory requirement is still a lofty hardware bar for mainstream PC users. The vast majority of consumer gaming laptops and mid-range desktop graphics cards ship with 8GB, 12GB, or 16GB of VRAM.

That is where Nemotron 3.5 Bolt E4B 4GB enters the picture. Scheduled for release on October 23, 2026, Bolt E4B is engineered specifically to fit inside a compact 4GB memory footprint. Rather than trying to tackle sprawling multi-day enterprise coding projects, Bolt E4B is built for low-latency, multi-turn conversational and physical agents.

NVIDIA targets this model at mainstream GeForce RTX GPUs, RTX/DGX Spark machines, and Jetson edge modules. It is positioned to power fast local voice assistants, embedded system interfaces, and interactive AI characters (such as NPCs powered by NVIDIA ACE) without starving a game or primary desktop application of video memory.

Section Summary: Arriving October 23, Nemotron 3.5 Bolt E4B targets a modest 4GB memory footprint, making local conversational agents and voice assistants viable on mainstream GeForce RTX cards without requiring flagship VRAM capacities.

Practical Guide: How to Evaluate and Run Nemotron Bolt on Your Windows PC​

If you are ready to test local agentic workflows—such as delegating code debugging, local file organization, or web synthesis—without sending prompts to a cloud provider, Nemotron 3.5 Bolt 74B-A9B is live now on Hugging Face.

Hardware Reality Check​

Before downloading tens of gigabytes of weights, match your local system to the right checkpoint:

1. Desktop GPUs with 24GB+ VRAM (e.g., GeForce RTX 3090, RTX 4090, RTX 5090, or RTX PRO Workstations): You can target the Nemotron 3.5 Bolt 74B-A9B 24GB quantized checkpoint. Keep in mind that on a strict 24GB card, Windows 11's Desktop Window Manager (DWM), web browsers, and multi-monitor setups already consume 1GB to 2.5GB of VRAM. You may need to close background GPU-accelerated apps or allow minor system-RAM spillover depending on context length.
2. Unified / High-Memory Systems (RTX Spark laptops, DGX Spark, or high-RAM Windows workstations): The 24GB footprint is specifically tuned so the model can reside comfortably in memory alongside your IDE, browser, and productivity tools.
3. Mainstream GPUs with 8GB–16GB VRAM: Hold off on the 74B model unless you plan to offload heavily to system RAM at lower token speeds; instead, watch for Bolt E4B 4GB on October 23.

Supported Local Software Stack​

NVIDIA has ensured day-one compatibility with the standard tools Windows enthusiasts already use:

• Runtimes & UIs: llama.cpp, Ollama, LM Studio, Unsloth, and vLLM.
• Agent Harnesses: Directly accessible via local agent frameworks such as Hermes and OpenClaw, alongside NVIDIA PAIR (which just exited beta with added llama.cpp support).

Picking the Right Artifact on Hugging Face​

When browsing NVIDIA's repository for the 74B-A9B model, you will see multiple distinct checkpoints released by NVIDIA Research. Do not grab the wrong file for your runtime:

• Nemotron 3.5 Super VL 74B-A9B 24GB GGUF: Choose this checkpoint if you are running LM Studio, Ollama, or llama.cpp on Windows.
• Nemotron 3.5 Super VL 74B-A9B 24GB LiLiCorr GGUF: Grab this companion speculative drafter GGUF if your llama.cpp setup supports speculative decoding to boost token generation speeds.
• Nemotron 3.5 Super VL 74B-A9B BF16: This is the massive, unquantized 16-bit checkpoint intended strictly for enterprise developers doing custom fine-tuning or distillation—avoid downloading this for local desktop inference.

A Quick Security Note on "Local" Agents​

While running Bolt 74B-A9B locally ensures your model inference happens on your own silicon rather than a third-party cloud server, remember that an agent is only as private as the tools you grant it. If you hook a local Nemotron model up to a web-browsing plugin, an external API, or an unrestricted PowerShell terminal, the agent will still transmit queries over the network or modify local files according to its permissions. Always sandbox local coding and terminal agents before letting them loose on production directories.

Section Summary: Windows users with 24GB+ memory setups can deploy the GGUF versions of Nemotron 3.5 Bolt 74B-A9B today using LM Studio, Ollama, or llama.cpp, alongside the LiLiCorr speculative drafter for faster local token generation.