How much VRAM do you need to run a 70B model?
About 46 GB at Q4 (the usual 4-bit download) with an 8K context: two 24 GB cards or one 48 GB card. At Q8 it needs about 78 GB. The cheapest setup listed now is 2 × Tesla P40 for $540.00.
A model's memory need is its weights at the chosen quantisation, plus the cache for the conversation (it grows with the context) and a little working space. Llama 3.3 70B has 70.6 billion parameters:
| Quantisation | Memory with an 8K context |
|---|---|
| Q4 (Q4_K_M) | 46 GB |
| Q5 (Q5_K_M) | 53 GB |
| Q6 (Q6_K) | 61 GB |
| Q8 (Q8_0) | 78 GB |
| FP16 (unquantised) | 144 GB |
Q4 is what most people run: the quality loss against the full model is small. Each card keeps about 1 GB for itself, so two 24 GB cards give 46 GB to the model; a longer context needs more. The cards don't need to be the newest: memory bandwidth sets the writing speed, and older 24 GB cards like the RTX 3090 have plenty.
The cheapest setups right now
| Setup | VRAM | Price | Writing speed |
|---|---|---|---|
| 2 × Tesla P40 | 48 GB | $540.00 | 4.0–6.1 tokens/s |
| 4 × Tesla P100 PCIe 16 GB | 64 GB | $721.00 | 8.6–13 tokens/s |
| 4 × Radeon VII | 64 GB | $1,023 | 8.4–14 tokens/s |
| 4 × Instinct MI50 16 GB | 64 GB | $1,087 | 8.4–14 tokens/s |
| 4 × Tesla V100 PCIe 16 GB | 64 GB | $1,327 | 11–16 tokens/s |
| 2 × Instinct MI50 32 GB | 64 GB | $1,329 | 8.4–14 tokens/s |
| 4 × Radeon Pro VII | 64 GB | $1,630 | 8.4–14 tokens/s |
| 4 × Radeon RX 6800 | 64 GB | $1,658 | 4.2–7.2 tokens/s |
Live eBay prices, shipping included; a set of cards at that many of the cheapest listings. Speeds are estimates for one user. Every model: What can I run?
Prices and picks update every time the site does (live listings checked within the last few hours).