PerGB

Can you run big AI models on a CPU instead of GPUs?

Yes, for the big mixture-of-experts models: they read only a small part of their weights for each word, so a server with many memory channels runs them at a usable speed for far less than the GPUs that would hold them. Dense models (Llama 70B) are much faster on GPUs.

Writing speed is set by memory bandwidth: every token reads the weights it uses once. A CPU platform's bandwidth is its memory channels × speed × 8 bytes; llama.cpp gets about 35-60% of it. An EPYC 9004 with all 12 DDR5-4800 channels filled has about 460 GB/s, a desktop with two DDR5 channels about 90 GB/s, an RTX 3090 936 GB/s.

A dense 70B model reads about 43 GB per token, so even a big server writes 2-5 tokens a second. A mixture of experts like gpt-oss 120B (5.1B active) or DeepSeek R1 (37B active) reads a fraction of that, while needing far more memory than GPUs hold affordably: that's where a server with lots of memory wins.

GPUs or a server, model by model

ModelMemory at Q4Cheapest GPUs (up to 8 cards)Best value CPU platform (chip, board, memory)
gpt-oss 120B MoE72 GB5 × Tesla P100 PCIe 16 GB$910.98 · 71–107 tokens/sAMD EPYC 7351P + 8 × 16 GB DDR4-2666 RDIMM$818.05 · about 26.2 tokens/s
Qwen3 235B-A22B MoE145 GB7 × Tesla P40$1,907 · 7.8–12 tokens/sAMD EPYC 9124 + 12 × 16 GB DDR5-4800 RDIMM$2,825 · about 16.4 tokens/s
DeepSeek R1 / V3 671B MoE408 GB7 × A16$16,670 · 11–16 tokens/sAMD EPYC 7351P + 8 × 64 GB DDR4-2666 RDIMM$2,663 · about 3.6 tokens/s
Llama 3.3 70B46 GB2 × Tesla P40$540.00 · 4.0–6.1 tokens/sAMD EPYC 7252 + 8 × 8 GB DDR4-3200 RDIMM$714.30 · about 2.3 tokens/s

Live prices; CPU platforms from the build page (the build with the most tokens/s per dollar that holds the model). Speeds are estimates; reading a long prompt is much slower on a CPU than on a GPU. With GPUs doing the work: does the CPU still matter?

Prices and picks update every time the site does (live listings checked within the last few hours).