PerGB

How much VRAM do you need to run a 70B model?

About 46 GB at Q4 (the usual 4-bit download) with an 8K context: two 24 GB cards or one 48 GB card. At Q8 it needs about 78 GB. The cheapest setup listed now is 2 × Tesla P40 for $540.00.

A model's memory need is its weights at the chosen quantisation, plus the cache for the conversation (it grows with the context) and a little working space. Llama 3.3 70B has 70.6 billion parameters:

QuantisationMemory with an 8K context
Q4 (Q4_K_M)46 GB
Q5 (Q5_K_M)53 GB
Q6 (Q6_K)61 GB
Q8 (Q8_0)78 GB
FP16 (unquantised)144 GB

Q4 is what most people run: the quality loss against the full model is small. Each card keeps about 1 GB for itself, so two 24 GB cards give 46 GB to the model; a longer context needs more. The cards don't need to be the newest: memory bandwidth sets the writing speed, and older 24 GB cards like the RTX 3090 have plenty.

The cheapest setups right now

SetupVRAMPriceWriting speed
2 × Tesla P4048 GB$540.004.0–6.1 tokens/s
4 × Tesla P100 PCIe 16 GB64 GB$721.008.6–13 tokens/s
4 × Radeon VII64 GB$1,0238.4–14 tokens/s
4 × Instinct MI50 16 GB64 GB$1,0878.4–14 tokens/s
4 × Tesla V100 PCIe 16 GB64 GB$1,32711–16 tokens/s
2 × Instinct MI50 32 GB64 GB$1,3298.4–14 tokens/s
4 × Radeon Pro VII64 GB$1,6308.4–14 tokens/s
4 × Radeon RX 680064 GB$1,6584.2–7.2 tokens/s

Live eBay prices, shipping included; a set of cards at that many of the cheapest listings. Speeds are estimates for one user. Every model: What can I run?

Prices and picks update every time the site does (live listings checked within the last few hours).