# How much GPU memory does your model need?

Count bytes per parameter. Serving needs 2 bytes per parameter in 16-bit, 1 in 8-bit and half a byte in 4-bit, plus about 20% for the KV cache and activations. Fine-tuning needs far more: about 16 bytes per parameter with Adam, unless LoRA or QLoRA freezes the base model.

## Serving a model

| Model size | 16-bit | 8-bit | 4-bit |
|---|---|---|---|
| 8B (Llama 3.1 8B) | 19 GB: one [RTX 4090](/gpus/rtx-4090.html.md) or [RTX 3090](/gpus/rtx-3090.html.md), RTX 3090 from $0.22 per GPU-hour | 9.6 GB: one [RTX 4090](/gpus/rtx-4090.html.md) or [RTX 3090](/gpus/rtx-3090.html.md), RTX 3090 from $0.22 per GPU-hour | 4.8 GB: one [RTX 4090](/gpus/rtx-4090.html.md) or [RTX 3090](/gpus/rtx-3090.html.md), RTX 3090 from $0.22 per GPU-hour |
| 14B (Qwen2.5 14B) | 34 GB: one [L40S](/gpus/l40s.html.md), [RTX 6000 Ada](/gpus/rtx-6000-ada.html.md) or [RTX A6000](/gpus/rtx-a6000.html.md), RTX A6000 from $0.42 per GPU-hour | 17 GB: one [RTX 4090](/gpus/rtx-4090.html.md) or [RTX 3090](/gpus/rtx-3090.html.md), RTX 3090 from $0.22 per GPU-hour | 8.4 GB: one [RTX 4090](/gpus/rtx-4090.html.md) or [RTX 3090](/gpus/rtx-3090.html.md), RTX 3090 from $0.22 per GPU-hour |
| 32B (Qwen2.5 32B) | 77 GB: one [H100](/gpus/h100.html.md) or [A100](/gpus/a100.html.md), A100 from $0.65 per GPU-hour | 38 GB: one [L40S](/gpus/l40s.html.md), [RTX 6000 Ada](/gpus/rtx-6000-ada.html.md) or [RTX A6000](/gpus/rtx-a6000.html.md), RTX A6000 from $0.42 per GPU-hour | 19 GB: one [RTX 4090](/gpus/rtx-4090.html.md) or [RTX 3090](/gpus/rtx-3090.html.md), RTX 3090 from $0.22 per GPU-hour |
| 70B (Llama 3.3 70B) | 168 GB: one [B200](/gpus/b200.html.md), from $5.60 per GPU-hour | 84 GB: one [RTX PRO 6000](/gpus/rtx-pro-6000.html.md), from $1.77 per GPU-hour | 42 GB: one [L40S](/gpus/l40s.html.md), [RTX 6000 Ada](/gpus/rtx-6000-ada.html.md) or [RTX A6000](/gpus/rtx-a6000.html.md), RTX A6000 from $0.42 per GPU-hour |
| 405B (Llama 3.1 405B) | 972 GB: 4 × [B300](/gpus/b300.html.md), from $8.00 per GPU-hour | 486 GB: 2 × [B300](/gpus/b300.html.md), from $8.00 per GPU-hour | 243 GB: one [B300](/gpus/b300.html.md), from $8.00 per GPU-hour |

## Fine-tuning a model

Figures are before activations.

| Model size | QLoRA | LoRA | Full fine-tuning |
|---|---|---|---|
| 8B (Llama 3.1 8B) | 5.6 GB: one [RTX 4090](/gpus/rtx-4090.html.md) or [RTX 3090](/gpus/rtx-3090.html.md), RTX 3090 from $0.22 per GPU-hour | 19 GB: one [RTX 4090](/gpus/rtx-4090.html.md) or [RTX 3090](/gpus/rtx-3090.html.md), RTX 3090 from $0.22 per GPU-hour | 128 GB: one [H200](/gpus/h200.html.md), from $4.50 per GPU-hour |
| 14B (Qwen2.5 14B) | 9.8 GB: one [RTX 4090](/gpus/rtx-4090.html.md) or [RTX 3090](/gpus/rtx-3090.html.md), RTX 3090 from $0.22 per GPU-hour | 34 GB: one [L40S](/gpus/l40s.html.md), [RTX 6000 Ada](/gpus/rtx-6000-ada.html.md) or [RTX A6000](/gpus/rtx-a6000.html.md), RTX A6000 from $0.42 per GPU-hour | 224 GB: one [B300](/gpus/b300.html.md), from $8.00 per GPU-hour |
| 32B (Qwen2.5 32B) | 22 GB: one [RTX 4090](/gpus/rtx-4090.html.md) or [RTX 3090](/gpus/rtx-3090.html.md), RTX 3090 from $0.22 per GPU-hour | 77 GB: one [H100](/gpus/h100.html.md) or [A100](/gpus/a100.html.md), A100 from $0.65 per GPU-hour | 512 GB: 2 × [B300](/gpus/b300.html.md), from $8.00 per GPU-hour |
| 70B (Llama 3.3 70B) | 49 GB: one [H100](/gpus/h100.html.md) or [A100](/gpus/a100.html.md), A100 from $0.65 per GPU-hour | 168 GB: one [B200](/gpus/b200.html.md), from $5.60 per GPU-hour | 1,120 GB: 8 × [H200](/gpus/h200.html.md), from $4.50 per GPU-hour |
| 405B (Llama 3.1 405B) | 284 GB: 4 × [H100](/gpus/h100.html.md), from $1.77 per GPU-hour | 972 GB: 4 × [B300](/gpus/b300.html.md), from $8.00 per GPU-hour | 6,480 GB: more than one 8-GPU machine |

## The rules behind the numbers

- Weights: parameters times bytes per parameter: 2 in 16-bit, 1 in 8-bit, 0.5 in 4-bit.
- Serving headroom: about 20% more for the KV cache and activations at moderate context and batch.
- Full fine-tuning: about 16 bytes per parameter with mixed-precision Adam ([ZeRO, arXiv:1910.02054](https://arxiv.org/abs/1910.02054)).
- LoRA: the 16-bit base model stays frozen; plan for the weights plus about 20%.
- QLoRA: a 4-bit frozen base; its authors fine-tuned a 65B model on one 48 GB GPU ([arXiv:2305.14314](https://arxiv.org/abs/2305.14314)), about 0.7 bytes per parameter.

## Questions

**How much GPU memory does a 70B model need?** To serve it, about 168 GB in 16-bit, 84 GB in 8-bit, 42 GB in 4-bit, including about 20% for the KV cache and activations. To fine-tune it, about 49 GB with QLoRA, 168 GB with LoRA and 1,120 GB for full fine-tuning with Adam, before activations.

**Can I run a 70B model on a single GPU?** Yes. It needs about 168 GB in 16-bit, which fits on one B200; about 84 GB in 8-bit, which fits on one RTX PRO 6000; about 42 GB in 4-bit, which fits on one L40S, RTX 6000 Ada or RTX A6000.

**Can I fine-tune an 8B model on an RTX 4090?** Yes, with LoRA (about 19 GB) or QLoRA (about 6 GB), which fit in the RTX 4090's 24 GB. Full fine-tuning needs about 128 GB before activations, so one H200.

**Why does fine-tuning need so much more memory than serving?** Full fine-tuning keeps 16-bit weights and gradients plus 32-bit Adam optimizer states and a 32-bit master copy of the weights, about 16 bytes per parameter (Rajbhandari et al., ZeRO, arXiv:1910.02054). LoRA freezes the base weights and trains small adapters; QLoRA also stores the frozen weights in 4-bit, which let its authors fine-tune a 65B model on one 48 GB GPU (Dettmers et al., arXiv:2305.14314).

**How much does it cost to rent the GPUs in these tables?** Each suggestion shows the lowest price per GPU-hour on Arrakis right now. Right now (as of 13 Oct 2026, 17:57 UTC), 125 GPUs are available on Arrakis across 43 machines. The lowest H100 price is $1.77 per GPU-hour and the lowest RTX 4090 price is $0.45 per GPU-hour. You pay the hourly price shown when you rent, from a prepaid balance.


Live prices for every GPU: [/gpus](/gpus.html.md).