Most of the speed and quality claims below are the author's personal testing. They are not controlled benchmarks, and the article gives no reproducible protocol or runtime versions. Treat them as informed hints, not verdicts.
The quick rundown
| Model | Size | Pitched for | Key caveat |
|---|---|---|---|
| Qwen3.8-9B-Distill | ~9B | General use, tools, vision | Vision needs an extra projector file |
| Ornith-1.5-9B | 9B dense | Agentic coding | Token- and context-hungry |
| Qwythos-9B-v2 | ~9B | Uncensored creative writing | Slower; needed KV-cache trimming |
| Llama-3.2-3B-Instruct | ~3.2B | CPU-only, always-on servers | Weaker reasoning than larger models |
Qwen3.8-9B-Distill: the all-rounder
The author's favorite is Empero's distillation. Its Hugging Face card says it is a full-parameter distillation of Qwen3.8 2.4T A95B into the Qwen3.5-9B architecture. In practice that means the larger "teacher" model's reasoning style is squeezed into a 9B student.
The card's own evaluation, run with lm-evaluation-harness, shows a mixed result. GSM8K (flexible match) slipped from 0.885 to 0.870, a 1.5-point drop. MMLU with chain-of-thought rose from 0.546 to 0.751, about 20 points. Those are developer-run numbers on specific test protocols, so don't read them as a guarantee for your documents.
Two details matter for planning:
- Vision is inherited, not tested. The card says the fine-tune is text-only and the base model's vision behavior wasn't evaluated. The author reports using it for image classification and getting better results than a conventional ONNX model on ambiguous images, though slower. That is his experience, not an evaluated feature.
- Expect long "thinking." Every answer opens with a reasoning block, and the card warns of occasional over-long deliberation on easy questions. It recommends temperature 0.6, top-p 0.95 and top-k 20, and says greedy decoding on long outputs can loop.
Will it fit in 8 GB?
An unofficial community GGUF conversion lists the Q4_K_M file at 5.78 GB. The vision projector adds 0.92 GB. That conversion's own guidance for 8 GB cards is to pick IQ4_XS or Q3_K_M, and it budgets about 1 GB extra if you want vision plus KV cache. So the Q4_K_M build is plausible on 8 GB for text with modest context. It is not a blanket promise once you add vision or a long context window. The conversion is third-party, and the author says nothing about which exact build he used.
Ornith-1.5-9B: a coding agent in a small package
Ornith-1.5-9B comes from the Ornith team and is a dense 9B model. Its "self-improvement" is a training method. According to the Ornith blog, the loop has the model propose new tasks, generate task-specific scaffolds, and produce solution rollouts for reinforcement learning. The installed model does not retrain itself while you code. It was trained to plan, test and revise.
The developer-reported numbers for the 9B are 47.0 on Terminal-Bench 2.1 using Claude Code and 70.6 on SWE-bench Verified. The model card lists Ornith-1.0-9B at 69.4 and Qwen3.5-9B at 53.2 on the latter. The same table shows Qwen3.6-35B-A3B scoring 73.4 on SWE-bench Verified, so "matches or exceeds much larger models" holds for some comparisons and not others. These runs used large context windows (up to 256K for SWE-bench) and big hardware, including 32 CPU cores and 48 GB RAM per Terminal-Bench run. Your 8 GB GPU setup will not reproduce them.
The author paired it with the OpenCode harness and built landing pages and small CLI tools in one pass. He warns the plan-test-revise behavior burns more tokens and context than a plain chat model, which limits project size. The model card says the same harness-dependence applies in general, since it recommends different sampling for general tasks (temperature 1.0) and precise coding (temperature 0.6).
For local use, the card shows an Ollama route (ollama run ornith-1.5:9b) and a llama.cpp route. It also gives an OpenCode provider config for pointing the agent at a local endpoint. Its GGUF list starts at Q4_K_M, 5.78 GB.
Qwythos-9B-v2: creative writing without the guardrails
Qwythos-9B-v2 is also from Empero AI, and it is intentionally uncensored. The developer says v2 targets repetition. Its card reports repetition under greedy decoding dropping from 6.7% to 0%, measured on its internal harness. It also says the same fix leaves benchmark scores roughly where v1 was. Those results are not independent, and the card itself admits HumanEval is a few points below the Qwen3.5-9B base.
The author prefers it for roleplay and fiction, saying action scenes and character interactions felt livelier than the Gemma models he tried. That is subjective. On his machine he saw close to 30 tokens per second with a context of around 68K tokens, and he had to drop KV-cache precision to Q5_1 to fit. Those figures apply to his hardware and settings only.
The card advertises a 1M-token context using YaRN scaling over a 262,144 native window. It notes that static YaRN carries a small short-context trade-off. Don't expect anything near a million tokens on an 8 GB card, because the cache is what runs out first.
Because the model is intentionally uncensored, deploy it responsibly. The card says it will engage sensitive technical topics.
Llama-3.2-3B-Instruct: the always-on workhorse
Meta's small Llama is the oldest pick, released September 25, 2024. It has about 3.2 billion parameters. Hugging Face lists a model release date of Sept 25, 2024. Third-party model pages and the LM Studio community GGUF card both give a 128K context length. The LM Studio card also lists officially supported languages: English, German, French, Italian, Portuguese, Hindi, Spanish and Thai.
The author's use case is a low-power home server running Project NOMAD, answering questions from a local knowledge base of Kiwix content and forum archives. He says 3B instruct models are roughly the cutoff before hallucination and instruction-following trouble set in. He doesn't give CPU, RAM or token-rate figures, so "runs on an old PC" should be tested on your own hardware.
The license is worth a line. Meta's Hugging Face page says use is governed by the Llama 3.2 Community License, a custom commercial license agreement. That is open weights, not an unrestricted open-source license. Check the terms before building a product on it.
For retrieval setups, always check answers against the retrieved passages. A small model can summarize confidently and still be wrong.
Settings that matter more than model choice
The author's tuning advice is a sensible starting point, not a universal recipe:
- Start with Q4_K_M where it's offered.
- Need more context? Try Q8_0 for both the K and V cache to save memory.
- Test MTP decoding (3 draft tokens) against MTP off instead of assuming it's faster. The Qwythos card says its multi-token-prediction head was restored but not co-trained with the fine-tuned weights, so speculative-decoding gains may be modest.
- Change one setting at a time and test on your real workload.
Weights are only part of memory use. KV cache grows with context, and vision files add more. A model that loads can still run out of memory once you ask for a long context.
Availability: check before you download
The author says all four are available in both LM Studio and Ollama. We could only confirm part of that. The Ornith card shows an Ollama command and a llama.cpp command. The Qwen3.8 card points to llama.cpp, Ollama and LM Studio for GGUF quantizations, but the GGUF we inspected was an unofficial conversion. A Llama-3.2-3B-Instruct GGUF is in the LM Studio community catalog. Features such as vision, tool calling and MTP can vary by runtime, build and quantization, so check the specific file.
Bottom line
For most 8 GB PC owners, the sensible weekend order is:
- General assistant, documents, tools: Qwen3.8-9B-Distill.
- Coding with an agent harness: Ornith-1.5-9B.
- Fiction and roleplay: Qwythos-9B-v2.
- Old hardware or a CPU-only server: Llama-3.2-3B-Instruct.
The caveat is that this is one enthusiast's testing on one GPU. The developer benchmarks come from the model makers themselves. The practical lesson holds: match the quantization and context size to your VRAM, and treat the first download as an experiment.