PerGB

Cheapest GPUs to run Llama 3.1 8B locally

Llama 3.1 8B needs about 6 GB of VRAM at Q4 with an 8K-token context (10 GB at Q8). The cheapest way to hold it right now: Tesla P100 PCIe 12 GB for $128.00, about 57–85 tokens/s.

The cheapest setups right now

Live on eBay
SetupVRAMSpeed (est.)Price nowPowerSoftware
Tesla P100 PCIe 12 GB12 GB4.6 GB spare57–85 tokens/s$128.00250 WCUDA 8.0 to 12.9 (not CUDA 13)
GeForce RTX 2070 Super8 GB0.6 GB spare46–69 tokens/s$169.99215 WCUDA 10.0 and newer
Tesla P100 PCIe 16 GB16 GB8.6 GB spare75–113 tokens/s$170.00250 WCUDA 8.0 to 12.9 (not CUDA 13)
GeForce RTX 20808 GB0.6 GB spare46–69 tokens/s$175.00215 WCUDA 10.0 and newer
Quadro RTX 40008 GB0.6 GB spare43–64 tokens/s$175.00160 WCUDA 10.0 and newer
RTX A2000 8 GB8 GB0.6 GB spare30–45 tokens/s$175.6170 WCUDA 11.1 and newer
GeForce RTX 20708 GB0.6 GB spare46–69 tokens/s$180.00175 WCUDA 10.0 and newer
GeForce RTX 2060 Super8 GB0.6 GB spare46–69 tokens/s$191.99175 WCUDA 10.0 and newer
GeForce RTX 30708 GB0.6 GB spare46–69 tokens/s$199.99220 WCUDA 11.1 and newer
GeForce RTX 2060 8 GB8 GB0.6 GB spare35–52 tokens/s$200.00160 WCUDA 10.0 and newer
RTX 2000 Ada 8 GB8 GB0.6 GB spare23–35 tokens/s$213.7570 WCUDA 11.8 and newer
GeForce RTX 3050 8 GB8 GB0.6 GB spare23–35 tokens/s$230.00130 WCUDA 11.1 and newer
GeForce RTX 2080 Super8 GB0.6 GB spare51–77 tokens/s$247.95250 WCUDA 10.0 and newer
Radeon VII16 GB8.6 GB spare74–127 tokens/s$249.00300 WNot on ROCm's support list (older ROCm or Vulkan)
GeForce RTX 3060 Ti8 GB0.6 GB spare46–69 tokens/s$257.99200 WCUDA 11.1 and newer

A set of cards is priced at that many of the cheapest live listings, shipping included. Speed: an estimate from memory bandwidth for one user chatting, within a third either way.

How it works

The memory a model needs is its weights at the quantisation you pick, plus the attention cache for the context and a little for working buffers. Q4 is what most people run at home, close to full quality at a third of the size. Each card keeps about 1 GB for itself, so two 24 GB cards hold 46 GB.

Writing each token reads every active weight once, so the speed is about the memory bandwidth divided by the model's size, times what the software gets out of the card (less off CUDA). Cards sharing a model layer by layer take turns, so a set runs about as fast as one card would. Treat it as a rough figure for one person chatting; a long prompt takes longer to read in.

Power is the cards' rated power. Over about 1,440 W with the rest of the PC, a set needs more than one 15 A circuit (or a 240 V one), and older cards often need a power cable adapter, which each card's page mentions.