Cheapest GPUs to run Qwen3.5 397B-A17B locally
Qwen3.5 397B-A17B needs about 245 GB of VRAM at Q4 with an 8K-token context (429 GB at Q8). A mixture of experts: 17B active. The cheapest way to hold it right now: 4 × A16 for $9,171, about 23–35 tokens/s.
The cheapest setups right now
Live on eBay| Setup | VRAM | Speed (est.) | Price now | Power | Software |
|---|---|---|---|---|---|
| 4 × A16 | 256 GB6.7 GB spare | 23–35 tokens/s | $9,171the 4 cheapest listings | 1,000 W | CUDA 11.1 and newer |
| 4 × Instinct MI210 | 256 GB6.7 GB spare | 33–57 tokens/s | $15,249the 4 cheapest listings | 1,200 W | ROCm (on AMD's support list) |
| 4 × RTX PRO 5000 Blackwell 72 GB | 288 GB38.7 GB spare | 39–59 tokens/s | $37,244the 4 cheapest listings | 1,200 W | CUDA 12.8 and newer |
| 3 × RTX PRO 6000 Blackwell | 288 GB39.7 GB spare | 52–78 tokens/s | $43,480the 3 cheapest listings | 1,800 Wover a 15 A circuit | CUDA 12.8 and newer |
| 4 × A100 PCIe 80 GB | 320 GB70.7 GB spare | 56–84 tokens/s | $55,498the 4 cheapest listings | 1,200 W | CUDA 11.0 and newer |
| 2 × H200 NVL | 282 GB34.7 GB spare | 140–210 tokens/s | $92,300the 2 cheapest listings | 1,200 W | CUDA 11.8 and newer |
| 3 × H100 NVL | 282 GB33.7 GB spare | 114–170 tokens/s | $107,599the 3 cheapest listings | 1,200 W | CUDA 11.8 and newer |
| 4 × H100 PCIe | 320 GB70.7 GB spare | 58–87 tokens/s | $156,612the 4 cheapest listings | 1,400 Wover a 15 A circuit | CUDA 11.8 and newer |
Per question: scaled from the model's reported speed on an RTX PRO 6000 by each card's compute and memory bandwidth, within about half either way. Licenses, scores and sources: decision models.
A set of cards is priced at that many of the cheapest live listings, shipping included. Speed: an estimate from memory bandwidth for one user chatting, within a third either way.
How it works
The memory a model needs is its weights at the quantisation you pick, plus the attention cache for the context and a little for working buffers. Q4 is what most people run at home, close to full quality at a third of the size. Each card keeps about 1 GB for itself, so two 24 GB cards hold 46 GB.
Writing each token reads every active weight once, so the speed is about the memory bandwidth divided by the model's size, times what the software gets out of the card (less off CUDA). Cards sharing a model layer by layer take turns, so a set runs about as fast as one card would. Treat it as a rough figure for one person chatting; a long prompt takes longer to read in.
Power is the cards' rated power. Over about 1,440 W with the rest of the PC, a set needs more than one 15 A circuit (or a 240 V one), and older cards often need a power cable adapter, which each card's page mentions.