# Deploy a model

Serve any Hugging Face model (text, image, video or speech) with vLLM, SGLang, vLLM-Omni or llama.cpp and your own engine options: an OpenAI-style endpoint with its own API key, and root SSH.

## In the console

1. Open **Models** in the [console](/console/#/models).
2. Name the model as Hugging Face does, such as `Qwen/Qwen3-8B`, or paste its link.
3. Pick the engine: **vLLM** or **SGLang** for text models, **vLLM-Omni** or **SGLang** for image and video models, **vLLM-Omni** for speech and sound, **llama.cpp** for a GGUF file. For llama.cpp, add the quantization after a colon, such as `unsloth/Qwen3-8B-GGUF:Q4_K_M`.
4. Paste your own engine options if you have them (see below).
5. For a gated or private model, add a Hugging Face token with read access.
6. **Find GPUs** reads the model's size from Hugging Face, shows the start line the machine will run, and lists every GPU the model fits on now, with the price. Pick one and rent it.

The engine downloads the model when the machine starts, so the endpoint answers a few minutes later. **Models** shows when it is ready, its address, its API key, code to call it, a try for chat and image models, and the engine's log.

## Your engine options

Paste the options you tuned, in whatever shape you keep them:

```sh
VLLM_USE_DEEP_GEMM=1 vllm serve Qwen/Qwen3-8B \
  --max-model-len 32768 --enable-prefix-caching \
  --speculative-config '{"method": "eagle3", "num_speculative_tokens": 3}'
```

Bare flags, the whole command and a `docker run` line all work, with variables in front and JSON values in quotes. You choose them and keep them current with your engine's releases. Arrakis reads them without a shell and hands them to the engine word for word, so:

- Shell syntax (`;`, `|`, `&&`, `$(...)`, `$VARIABLES`) is refused: nothing would run it.
- Where the engine listens (`--host`, `--port`), its key (`--api-key`) and the model stay Arrakis's, so the endpoint keeps its address and key. `--served-model-name` (`--alias` for llama.cpp) renames the model requests use.
- Your parallelism wins: with `-tp`, `--tp` or `-dp` in your options, Arrakis adds none of its own.
- Settings that choose GPUs (`NVIDIA_*`) and loader settings (`LD_*`) are refused: a machine uses the GPUs it rents.
- Your Hugging Face token goes in its own field, never in the options, so it isn't saved with them.

## From the API

```sh
curl -X POST $ARRAKIS_URL/v1/deploy \
  -H "Authorization: Bearer ark_..." \
  -H "Content-Type: application/json" \
  -d '{"model": "black-forest-labs/FLUX.1-schnell", "engine": "omni", "termination_hours": 4,
       "extra_args": "--enforce-eager"}'
```

Without `node_id`, Arrakis rents the best-ranked machine that holds the model, on as few GPUs as it allows. `POST /v1/deploy/plan` lists every GPU it fits on first, for free, and with `extra_args` it checks your options and returns the start line. The answer carries the endpoint and its `api_key`; `GET /v1/instances/{id}/endpoint` gives them again, with `ready` once the model answers, and `GET /v1/instances/{id}/endpoint/log` the end of the engine's log. Agents use the `deploy_model` tool over [MCP](/docs/agents).

## Call it

The endpoint speaks the OpenAI API, so any OpenAI client works. The model's name in a request is its Hugging Face id, without the quantization, unless you renamed it.

```python
from openai import OpenAI

client = OpenAI(base_url="http://<address>/v1", api_key="sk-...")
reply = client.chat.completions.create(
    model="Qwen/Qwen3-8B",
    messages=[{"role": "user", "content": "Hello"}],
)
print(reply.choices[0].message.content)
```

| The model makes | Ask it at |
|---|---|
| Text | `POST /v1/chat/completions`, `/v1/completions`, `/v1/embeddings` |
| Images | `POST /v1/images/generations`, `/v1/images/edits` |
| Video | `POST /v1/videos`, then `GET /v1/videos/{id}` until it is completed and `GET /v1/videos/{id}/content` |
| Speech | `POST /v1/audio/speech` |
| Sound and music | `POST /v1/audio/generate` |
| Text from speech | `POST /v1/audio/transcriptions` |

## The machine is yours

- It is an ordinary rental: billed by the hour, removed at its removal time, and listed under **Instances**.
- It boots an Arrakis image built on the engine's official release, with root SSH. Run `arrakis-serve status`, `arrakis-serve logs` or `arrakis-serve restart` on it.
- To try other options without a new machine, edit `/etc/arrakis/serve/command` (one word per line) or `/etc/arrakis/serve/env`, then run `arrakis-serve restart`. When the engine stops, the machine stays up so you can fix it.
- The endpoint is the machine's address over plain HTTP. A gate on port 8000 checks its API key (`Authorization: Bearer <key>`) before a request reaches the engine, which listens only inside the machine; `/health` answers without the key and says only whether the engine is up. For private traffic, open an SSH tunnel (`ssh root@<host> -p <port> -N -L 8000:localhost:8000`) and use `http://localhost:8000/v1`.
- Your Hugging Face token goes to the machine as `HF_TOKEN`, and Arrakis doesn't keep it. The machine's host can read what runs on it, so use a fine-grained, read-only token.

More: [all docs](/docs.md) · [OpenAPI](/openapi.json) · [llms.txt](/llms.txt)