The problem: one slot, every job
Jonker's starting point is an 8GB GPU, which in their telling has room for one decent model at a time. Whatever is loaded that day handles every request, suited or not. Small models respond quickly but struggle with heavy or multi-step work. Bigger ones fix that but make quick questions slower as a session grows.
The weekly swapping didn't help. The real fix was to look at what the prompts actually were. Many of them had little in common, so they didn't need the same model.
This is the writer's own experience on an RTX 3070, not a controlled test. It's still a useful way to think about constrained hardware: match the model to the task instead of looking for one that wins everywhere.
The three-model stack
| Role | Model | What it handles (per the writer) |
|---|---|---|
| Everyday and vision | Qwen3.5-4B | Quick questions, screenshot-led tasks, web research through search and fetch tools |
| Long reasoning chats | Qwythos-9B-v2 (Q4_K_M) | Back-and-forth conversations, research documents, portfolio redesign |
| Light coding | FluentlyQwen3-Coder-4B | YAML, JSON, regex, small Python utilities, explaining scripts |
Anything that needs serious horsepower still goes to a cloud model. The writer names Claude for that.
Qwen3.5-4B: the generalist with eyes
According to Alibaba's Qwen repository, Qwen3.5-9B, 4B, 2B and 0.8B became available on Hugging Face Hub and ModelScope on 2026-03-02. That date matches the writer's "March" release. The vision claim also holds up. The official Qwen3.5-4B model card describes a causal language model with a vision encoder. NVIDIA's documentation separately lists both the 4B and 9B as vision-language models. So the screenshot claim is plausible for the official checkpoints.
The writer's memory argument is that a 4-bit 4B needs roughly 3GB, about half of what the 9B takes. That leaves more room for context than the roughly 30K they normally use. Public file listings broadly agree. An Unsloth GGUF listing shows a Q4_K_M file of 2.74 GB for the 4B, and the Q4_K_M Qwythos 9B file is about 5.74 GB (see below). Treat these as file sizes, not total VRAM needs. Runtime overhead, the KV cache that grows with context, and any vision projector all take extra memory. A separate community model card shows how vision is packaged in GGUF form. That one pairs a quantized model with a separate vision projector file for image input. If screenshots don't work in your setup, check whether your runner is loading that projector.
One caution from the writer is worth repeating. They cite Artificial Analysis as finding that the 4B guessed about 80% of the time when it didn't know a factual answer. I couldn't locate that test or its methodology, so treat the figure as the writer's claim. The practical advice holds either way. Small models can bluff, so pair them with a search tool and make them read sources before answering. That is how the writer runs it, using Brave Search and a page-fetch tool.
Qwythos-9B-v2: the reasoning workhorse
Qwythos is a community fine-tune of Qwen3.5-9B from Empero AI. Its Hugging Face page says v2 is the original Qwythos with the looping behavior fixed, with repetition under greedy or low-temperature decoding dropping from 6.7% to 0%. That matches the writer's "loop rate at zero." Empero's own notes say all training was text-only and that the vision tower was never fine-tuned. The page says image behavior inherits from base Qwen3.5-9B and has not been independently evaluated for this release.
That qualifies the writer's claim that Qwythos is weak on images. The model has a vision path when you load the separate projector. The writer's habit of sending images to the Qwen 4B is still reasonable, mostly because that is the model Alibaba tests and documents for vision.
The file-size claim checks out. The Q4_K_M GGUF is listed at 5.34 GiB (5.74 GB), which is close to the writer's "around 5.7GB." Whether it fits fully on your card depends on context length and other VRAM use. On an 8GB GPU, a ~5.7GB model file leaves limited headroom for a long context.
The benchmark numbers are developer-reported, as the writer notes. They are 93.6% against 80.6% on GSM8K, and 49% against 32.8% on GPQA-Diamond. These are Empero's own comparisons against the base 9B. They show direction, not a ranking across all tasks. The same goes for the writer's comparison to Claude Sonnet and the improvement from a Claude-style system prompt. Those are subjective impressions.
Two practical details from the model page matter if you try it. First, every response opens with a think block before the answer, so expect extra tokens and some latency. Second, the page recommends Q4_K_M as the default and lists sampling settings, including temperature 0.6, top-p 0.95 and top-k 20. The listing also describes the model as intentionally uncensored. That is a reason to think about where you deploy it, especially in a shared or work setting.
FluentlyQwen3-Coder-4B: the small code helper
Project Fluently's model listing shows FluentlyQwen3-Coder-4B-0909 as a 4B text-generation model, updated Sep 9, 2025. Its model card says it came from SFT and GRPO training plus step-by-step merging. It lists a native 32,768-token context, extendable to 131,072 with YaRN, and an Apache-2.0 license. It is built on the older Qwen3 generation, which fits the writer's description, and it is text-only. Don't expect it to read screenshots.
The writer uses a 4-bit quant from mradermacher at around 2.4GB. I couldn't confirm that specific file or its size. Quantized builds of the model exist on the Hub, but check the listing before you download.
The card also advises against greedy decoding for thinking mode, citing performance degradation and endless repetition. It suggests temperature 0.6, top-p 0.95 and top-k 20. That is the opposite of the Qwythos v2 advice, where greedy decoding is described as safe. If you run both in one app, set sampling per model instead of sharing one preset.
Safety notes the original skips
Two parts of the workflow deserve a harder look.
Explaining scripts before you run them. Asking a local model what a GitHub script will do is a sensible first pass. It is not a security review. A 4B model can miss obfuscated behavior, misread what a script does, or confidently describe something wrong. For untrusted code, read it yourself, test it in a disposable VM or sandbox, and check what network calls and file operations it makes. Treat the model's summary as a hint.
Private documents. The writer queries health results and bank statements locally on a phone with the 4B and 2B models. Running inference locally avoids sending the prompt to a cloud model, but it doesn't guarantee privacy end to end. The app, its logs, backups and any enabled tools all matter. In particular, if a search or fetch tool is switched on, the model could send query text outward. For sensitive documents, use a setup with those tools disabled, and know where your runner stores chat history.
How to try this yourself
- List your last two weeks of prompts and sort them into buckets such as quick questions, multi-step reasoning, code and config, and image work.
- Pick one small general model with vision, one reasoning-leaning model and one code-tuned model that fit your VRAM.
- Check file size against your card, then budget for the KV cache. A model that barely loads will choke once the context grows.
- If you want screenshot input, confirm your runner supports vision and has the projector file.
- Add a search and fetch tool to the smallest model, and test whether it cites what it read.
- Set sampling parameters per model and keep a cloud fallback for the hard stuff.
Analysis: why the approach makes sense
Splitting tasks across models is a familiar idea in the broader AI world, where routing between models is common. On a single consumer GPU it matters more, because VRAM is the scarce resource. A 4B model that takes about half the memory of a 9B leaves room for a bigger context window. For research with long fetched pages, that room can matter more than a few benchmark points. The writer makes the same argument.
There are fair counterpoints. Juggling three models adds overhead, because loading and unloading takes time and each model needs its own settings. The gains come from one person's workload, and results will differ on other hardware, runtimes and quantizations. Community fine-tunes also vary in quality and maintenance. The code model hasn't been updated on Hugging Face since September 2025, and Qwythos is a small-lab project with mostly self-reported evaluations.
The takeaway isn't that these three models are the best ones. It's that matching tasks to models is a cheaper upgrade than buying a bigger GPU. If you spend your weekends swapping a default model and blaming it, this is worth a try.