Arrakis / Guides / GPU memory
How much GPU memory does your model need?
Count bytes per parameter. Serving needs 2 bytes per parameter in 16-bit, 1 in 8-bit and half a byte in 4-bit, plus about 20% for the KV cache and activations. Fine-tuning needs far more: about 16 bytes per parameter with Adam, unless LoRA or QLoRA freezes the base model.
Serving a model
Memory to run a model for inference, with the smallest GPU setup that holds it and the lowest price on Arrakis right now.
| Model size | 16-bit | 8-bit | 4-bit |
|---|---|---|---|
| 8BLlama 3.1 8B | 19 GB: one RTX 4090 or RTX 3090, RTX 3090 from $0.22 per GPU-hour | 9.6 GB: one RTX 4090 or RTX 3090, RTX 3090 from $0.22 per GPU-hour | 4.8 GB: one RTX 4090 or RTX 3090, RTX 3090 from $0.22 per GPU-hour |
| 14BQwen2.5 14B | 34 GB: one L40S, RTX 6000 Ada or RTX A6000, RTX A6000 from $0.42 per GPU-hour | 17 GB: one RTX 4090 or RTX 3090, RTX 3090 from $0.22 per GPU-hour | 8.4 GB: one RTX 4090 or RTX 3090, RTX 3090 from $0.22 per GPU-hour |
| 32BQwen2.5 32B | 77 GB: one H100 or A100, A100 from $0.65 per GPU-hour | 38 GB: one L40S, RTX 6000 Ada or RTX A6000, RTX A6000 from $0.42 per GPU-hour | 19 GB: one RTX 4090 or RTX 3090, RTX 3090 from $0.22 per GPU-hour |
| 70BLlama 3.3 70B | 168 GB: one B200, from $5.60 per GPU-hour | 84 GB: one RTX PRO 6000, from $1.75 per GPU-hour | 42 GB: one L40S, RTX 6000 Ada or RTX A6000, RTX A6000 from $0.42 per GPU-hour |
| 405BLlama 3.1 405B | 972 GB: 4 × B300, from $6.99 per GPU-hour | 486 GB: 2 × B300, from $6.99 per GPU-hour | 243 GB: one B300, from $6.99 per GPU-hour |
Fine-tuning a model
QLoRA trains adapters on a 4-bit frozen model, LoRA on a 16-bit one, and full fine-tuning updates every weight. Figures are before activations, which grow with batch size and sequence length.
| Model size | QLoRA | LoRA | Full fine-tuning |
|---|---|---|---|
| 8BLlama 3.1 8B | 5.6 GB: one RTX 4090 or RTX 3090, RTX 3090 from $0.22 per GPU-hour | 19 GB: one RTX 4090 or RTX 3090, RTX 3090 from $0.22 per GPU-hour | 128 GB: one H200, from $3.40 per GPU-hour |
| 14BQwen2.5 14B | 9.8 GB: one RTX 4090 or RTX 3090, RTX 3090 from $0.22 per GPU-hour | 34 GB: one L40S, RTX 6000 Ada or RTX A6000, RTX A6000 from $0.42 per GPU-hour | 224 GB: one B300, from $6.99 per GPU-hour |
| 32BQwen2.5 32B | 22 GB: one RTX 4090 or RTX 3090, RTX 3090 from $0.22 per GPU-hour | 77 GB: one H100 or A100, A100 from $0.65 per GPU-hour | 512 GB: 2 × B300, from $6.99 per GPU-hour |
| 70BLlama 3.3 70B | 49 GB: one H100 or A100, A100 from $0.65 per GPU-hour | 168 GB: one B200, from $5.60 per GPU-hour | 1,120 GB: 8 × H200, from $3.40 per GPU-hour |
| 405BLlama 3.1 405B | 284 GB: 4 × H100, from $1.77 per GPU-hour | 972 GB: 4 × B300, from $6.99 per GPU-hour | 6,480 GB: more than one 8-GPU machine |
The rules behind the numbers
Rules of thumb, close enough to choose a GPU. Measure before a long run.
- Weights
- Parameters times bytes per parameter: 2 in 16-bit (BF16 or FP16), 1 in 8-bit (FP8 or INT8), 0.5 in 4-bit.
- Serving headroom
- About 20% more for the KV cache and activations at moderate context and batch. Long contexts and large batches need more.
- Full fine-tuning
- About 16 bytes per parameter with mixed-precision Adam: weights, gradients, optimizer states and a 32-bit master copy (ZeRO, arXiv:1910.02054). Split it across GPUs with FSDP or ZeRO.
- LoRA
- The 16-bit base model stays frozen; small adapters train. Plan for the weights plus about 20%.
- QLoRA
- The frozen base model is stored in 4-bit. Its authors fine-tuned a 65B model on one 48 GB GPU (QLoRA, arXiv:2305.14314), about 0.7 bytes per parameter.
- GPU memory
- From NVIDIA's pages for each GPU, linked from its page here.
Questions
Live prices for every GPU are on the GPU prices page.
How much GPU memory does a 70B model need?
To serve it, about 168 GB in 16-bit, 84 GB in 8-bit, 42 GB in 4-bit, including about 20% for the KV cache and activations. To fine-tune it, about 49 GB with QLoRA, 168 GB with LoRA and 1,120 GB for full fine-tuning with Adam, before activations.
Can I run a 70B model on a single GPU?
Yes. It needs about 168 GB in 16-bit, which fits on one B200; about 84 GB in 8-bit, which fits on one RTX PRO 6000; about 42 GB in 4-bit, which fits on one L40S, RTX 6000 Ada or RTX A6000.
Can I fine-tune an 8B model on an RTX 4090?
Yes, with LoRA (about 19 GB) or QLoRA (about 6 GB), which fit in the RTX 4090's 24 GB. Full fine-tuning needs about 128 GB before activations, so one H200.
Why does fine-tuning need so much more memory than serving?
Full fine-tuning keeps 16-bit weights and gradients plus 32-bit Adam optimizer states and a 32-bit master copy of the weights, about 16 bytes per parameter (Rajbhandari et al., ZeRO, arXiv:1910.02054). LoRA freezes the base weights and trains small adapters; QLoRA also stores the frozen weights in 4-bit, which let its authors fine-tune a 65B model on one 48 GB GPU (Dettmers et al., arXiv:2305.14314).
How much does it cost to rent the GPUs in these tables?
Each suggestion shows the lowest price per GPU-hour on Arrakis right now. Right now (as of 13 Oct 2026, 17:09 UTC), 147 GPUs are available on Arrakis across 44 machines. The lowest H100 price is $1.77 per GPU-hour and the lowest RTX 4090 price is $0.49 per GPU-hour. You pay the hourly price shown when you rent, from a prepaid balance.