phi-2 GPU requirements
phi-2 has 2.8B parameters. To serve it, plan for about 6.7 GB of GPU memory in 16-bit, 3.3 GB in 8-bit or 1.7 GB in 4-bit, including about 20% for the KV cache and activations.
What it takes to serve it
GPU memory to run it for inference, with the smallest GPU setup that holds it and the lowest price on Arrakis right now.
| Precision | Memory and the smallest setup |
|---|---|
| 16-bit | 6.7 GB: one RTX 4090 or RTX 3090, RTX 3090 from $0.19 per GPU-hour |
| 8-bit | 3.3 GB: one RTX 4090 or RTX 3090, RTX 3090 from $0.19 per GPU-hour |
| 4-bit | 1.7 GB: one RTX 4090 or RTX 3090, RTX 3090 from $0.19 per GPU-hour |
What it takes to fine-tune it
QLoRA trains adapters on a 4-bit frozen model, LoRA on a 16-bit one, and full fine-tuning updates every weight. Figures are before activations, which grow with batch size and sequence length.
| Method | Memory and the smallest setup |
|---|---|
| QLoRA | 1.9 GB: one RTX 4090 or RTX 3090, RTX 3090 from $0.19 per GPU-hour |
| LoRA | 6.7 GB: one RTX 4090 or RTX 3090, RTX 3090 from $0.19 per GPU-hour |
| Full fine-tuning | 44 GB: one L40S, RTX 6000 Ada or RTX A6000, L40S from $0.44 per GPU-hour |
About the model
From its card and files on Hugging Face. Memory figures here follow the GPU memory guide.
- Parameters
- 2.8B
- Context window (tokens)
- 2,048
- Published in
- BF16
- License
- mit
- Released
- 2023-12-13
- Weights
- microsoft/phi-2
Deploy it on Arrakis
Open Models in the console, choose vLLM, SGLang or llama.cpp, and rent a GPU it fits on. You get an OpenAI-compatible endpoint with its own API key, and root SSH to the machine. Or run it yourself on any machine with enough GPU memory:
vllm serve microsoft/phi-2 --tensor-parallel-size 1Questions
Live prices for every GPU are on the GPU prices page.