Benchmarks are not your workload
Abidi's starting point is fair to local models. OpenAI says gpt-oss-20b can run on edge devices with just 16 GB of memory in the sense of its own release notes. Those notes also report results similar to o3-mini on common benchmarks. OpenAI says the larger gpt-oss-120b model runs on a single 80 GB GPU and gets near-parity with o4-mini on core reasoning benchmarks.
Both figures are vendor-reported. A benchmark shows how a model performs on a defined test. It does not show that the model will handle your hard questions consistently, with enough context and at a tolerable speed. A "runs in 16 GB" figure is a minimum to load the model. It does not promise a given response speed or context length.
Abidi also makes a point that is easy to miss. Running a model locally does not change how smart it is. What changes is which models your hardware can host, and what compromises you accept to host them.
The 43 GB model and the 32 GB card
The article's main example is concrete. Ollama lists Llama 3.3 70B at Q4_K_M quantization as a 43 GB download with a 128K context window. Nvidia's RTX 5090 carries 32 GB of GDDR7 and a $1,999 launch price. The weights alone cannot fit entirely in that card's VRAM, even before any working memory is counted.
A note on that price. $1,999 is Nvidia's announced starting price. It is not a verified street price or an availability guarantee for today. Abidi himself says you will almost certainly not find the card at that figure.
Context is a second memory bill
Loading the weights is only the first cost. Ollama's documentation says that setting a larger context length will increase the amount of memory required to run a model. Ollama also sets default context lengths by VRAM:
- Under 24 GiB of VRAM: 4k context
- 24 to 48 GiB: 32k context
- 48 GiB or more: 256k context
Ollama also says tasks that need large context, such as web search, agents and coding tools, should be set to at least 64000 tokens. That is why a model's advertised 128K window can mislead. On a typical consumer GPU, the default you actually get may be far smaller.
This has a practical consequence for the tasks people most want to move off the cloud. Third-party guides note that input beyond the active context is truncated. A long document or a big codebase may therefore never be fully seen by the model unless you raise the context and have the memory for it. Those guides are secondary sources, but they match Ollama's own documentation on the defaults.
Quantization and offload are trade-offs, not fixes
Abidi calls quantization "clever, not magic." Lower-precision weights shrink the footprint, and aggressive settings can cost accuracy. The llama.cpp documentation backs up the wider picture. It lists KV cache types such as q8_0 and q4_0. It says Flash Attention is required for certain combinations, such as quantized V cache. It also gives the escalating fixes for running out of memory:
- Lower the context size. The KV cache is roughly proportional to it.
- For llama-server, lower the number of parallel slots, since each concurrent sequence allocates its own cache.
- As a last resort, reduce GPU layers. The remaining layers run on the CPU and inference gets much slower.
Ollama similarly advises that for best performance you should avoid offloading to the CPU. In short, software can make a workload fit, but the cost shows up as accuracy, context or tokens per second.
Efficient models still need somewhere to live
The mixture-of-experts design of the gpt-oss models reduces compute per token. According to OpenAI, gpt-oss-120b activates 5.1B parameters per token and gpt-oss-20b activates 3.6B. They have 117B and 21B total parameters respectively. Fewer active parameters speed things up, but the full set of weights still has to be stored. That is why the 120b model is aimed at an 80 GB-class GPU.
Abidi also lists the power supply, cooling, case space and DDR5 prices as extra costs. Nvidia does specify 1,000 W of required system power for its RTX 5090 reference configuration. The remaining claims, about DDR5 pricing, cooling and chassis needs, are the author's general argument. I have no independent figures for them, so treat them as directional.
Windows angle
For Windows users, the same trade-offs apply to Microsoft's own tooling. OpenAI's release notes say Microsoft brought GPU-optimized gpt-oss-20b builds to Windows devices through ONNX Runtime. They are available via Foundry Local and the AI Toolkit for VS Code. That is a sign that the 20b class is the realistic target for mainstream PCs, while the 120b class stays workstation territory.
A practical checklist before you buy
- Check the quantized file size. Compare the actual download, such as 43 GB, against your VRAM, not the parameter count.
- Decide the context you need. Coding agents and long-document work need far more than a 4k default.
- Inspect the split. Ollama's
ollama pscommand shows the PROCESSOR split and the allocated context. Any CPU share means slower generation. - Set context deliberately. Ollama lets you set it via an app slider or the
OLLAMA_CONTEXT_LENGTHenvironment variable. - Test your real task. Try a smaller model on your actual workload first. If it does the job, you may not need the expensive GPU.
Where the article is strongest and weakest
It is strongest on the distinction between "loads" and "works well." It is more rhetorical on price, and it does not benchmark local models against Claude or any other hosted service on real tasks. It also does not weigh the alternatives. These include multi-GPU setups, which llama.cpp supports, unified-memory machines, and renting cloud GPUs. The core claim still stands. Free software widens your options, but it does not lift the hardware ceiling.
For many users, privacy, offline access and control justify a smaller model. As Abidi puts it, a small model that handles your workload is a successful tool, not a consolation prize. The expensive hardware only becomes unavoidable if you want a local replacement for the largest hosted assistants.