Some links are affiliate links: if you buy through them we may earn a commission, at no extra cost to you. It never changes our assessments. Affiliate disclosure
The first question for any local model is not speed but whether it fits. If the model does not fit in the memory your GPU can use, it either refuses to load or crawls along in system RAM. The good news: the calculation is simple once you know the three parts.
The three parts
1. The weights. The model file. Its size depends on the number of parameters and the quantisation, the number of bits used per weight:
| Quantisation | Bits per weight (approx.) | Quality |
|---|---|---|
| Q4_K_M | ~4.8 | The usual choice: small quality loss |
| Q8_0 | ~8.5 | Nearly identical to the original |
| FP16 | 16 | Original precision |
A rule of thumb for 4-bit is 0.6 GB per billion parameters, but real files differ, so we always use the actual GGUF file size. gpt-oss-120b is a good example: its experts ship in 4-bit already, so even its "8-bit" file is about 63 GB.
2. The context cache (KV cache). Every token in the conversation stores keys and values for each attention layer. It grows with the context length and varies enormously between models. Newer models are much more frugal here:
| Model (4-bit) | 8K context | 32K context | 128K context |
|---|---|---|---|
| Qwen3.6 27B | 18.9 GB | 20.5 GB | 26.9 GB |
| Qwen3 32B | 23.4 GB | 29.8 GB | 32.0 GB (max 40K) |
| Llama 3.3 70B | 46.7 GB | 54.8 GB | 87.0 GB |
| Qwen3.6 35B A3B | 23.8 GB | 24.3 GB | 26.3 GB |
Look at Qwen3 32B: at 32K tokens its cache alone is 8.6 GB, so it no longer fits a 24 GB card, while the newer Qwen3.6 27B does with room to spare. Qwen3.5 and 3.6 use linear attention in most layers, which keeps the cache tiny.
3. Runtime overhead. Buffers and the GPU runtime take about 1.5 GB more.
How much of your memory can a model use?
- Graphics cards: practically all of it (close other GPU programs).
- Macs: by default macOS lets the GPU use about two thirds of memory on machines up to 36 GB and three quarters above that. A 128 GB Mac Studio gives a model about 96 GB.
- Ryzen AI Max and other unified-memory PCs: on Linux almost all of it; we count about 90%.
Practical sizes
- 12–16 GB: the best small models (Qwen3.5 9B, Gemma 4 12B, gpt-oss-20b) with long contexts.
- 24–32 GB: the sweet spot for 27–35B models such as Qwen3.6 27B and Gemma 4 31B. A used NVIDIA GeForce RTX 3090 is still the cheapest way there.
- 48–64 GB: 70B dense models at 4-bit, usually with two cards.
- 96–128 GB: 100B-class mixture-of-experts models like gpt-oss-120b and Qwen3.5 122B. This is where AMD Ryzen AI Max+ 395 (Strix Halo) and the Mac Studio shine.
- 256 GB and up: the largest open models, realistically only on a Apple Mac Studio (M5 Ultra).
Try it
Pick any model on the models page and choose your quantisation and context: every platform shows whether it fits, with a memory bar and a speed ceiling.