# Serve an LLM with vLLM

> Serve an open model behind an OpenAI-compatible API on your own GPU.

This example deploys the vLLM template on one GPU, serves `Qwen/Qwen2.5-7B-Instruct` at a public HTTPS address with an API key, and calls it with the OpenAI chat completions API. The model files are cached on a network volume, so later starts skip the download.

## What you need

- An API key for your server, for example from `openssl rand -hex 24`. Clients send it to call the model.

## Step 1: Create a network volume for the model cache

<tabs sync="interface">
<tabs-item label="Console">

Open the [Storage page](https://console.cloud.featherless.ai/storage), click **Create volume**, name it `models`, set **Capacity** to `100`, and click **Create volume**.

</tabs-item>

<tabs-item label="CLI">

```bash
featherless-cloud volume create models --gb 100 --wait
```

</tabs-item>
</tabs>

## Step 2: Deploy the server

<tabs sync="interface">
<tabs-item label="Console">
<steps level="4">

#### Choose the template

Open the [Instances page](https://console.cloud.featherless.ai/instances) and click **Deploy instance**. Name the instance `vllm`. Under **Template**, search for **vLLM** and select **vLLM (ROCm)**.

#### Choose compute

Under **Compute**, choose a **GPU model** and **1 GPU**.

#### Mount the volume

Under **Network volumes**, click **Add network volume**, choose `models`, and keep the mount path `/workspace`.

#### Expose the API port

Under **Public ports**, enter `8000`.

#### Start vLLM

Under **Start command**, enter:

```text
vllm serve Qwen/Qwen2.5-7B-Instruct --host 0.0.0.0 --port 8000
```

Under **Environment variables**, add:

<table>
<thead>
  <tr>
    <th>
      Name
    </th>
    
    <th>
      Value
    </th>
  </tr>
</thead>

<tbody>
  <tr>
    <td>
      <code>
        HF_HOME
      </code>
    </td>
    
    <td>
      <code>
        /workspace/hf
      </code>
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        VLLM_API_KEY
      </code>
    </td>
    
    <td>
      Your API key
    </td>
  </tr>
</tbody>
</table>

#### Launch

Click **Review launch**, then **Launch instance**.

</steps>
</tabs-item>

<tabs-item label="CLI">

```bash
featherless-cloud instance create \
  --name vllm \
  --template featherless/rocm-vllm \
  --gpu-count 1 \
  --volume models:/workspace \
  --port 8000 \
  --env HF_HOME=/workspace/hf \
  --env VLLM_API_KEY="$VLLM_API_KEY" \
  --command "vllm serve Qwen/Qwen2.5-7B-Instruct --host 0.0.0.0 --port 8000"
```

</tabs-item>
</tabs>

`HF_HOME` puts the downloaded model on the volume. `VLLM_API_KEY` makes vLLM reject requests without the key, because anyone can reach the public address.

## Step 3: Wait for the model to load

The first start downloads the image and the model, which takes several minutes. Open the instance's **Logs** tab, or follow them from the CLI, until vLLM reports that it's ready:

```bash
featherless-cloud instance logs vllm --follow
```

```text
(APIServer pid=44) INFO:     Application startup complete.
```

## Step 4: Call the model

Copy the address for port `8000` from the instance's **Connect** tab, or from `featherless-cloud instance get vllm`. Then send a chat completion:

```bash
curl https://<address>/v1/chat/completions \
  -H "Authorization: Bearer $VLLM_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen2.5-7B-Instruct",
    "messages": [{"role": "user", "content": "Say hello in five words."}],
    "max_tokens": 32
  }'
```

The response follows the OpenAI format:

```json
{
  "object": "chat.completion",
  "model": "Qwen/Qwen2.5-7B-Instruct",
  "choices": [
    {
      "index": 0,
      "message": { "role": "assistant", "content": "Hello there, friend!" },
      "finish_reason": "stop"
    }
  ]
}
```

Any OpenAI-compatible client works: set its base URL to `https://<address>/v1` and its API key to your `VLLM_API_KEY`.

## Clean up

Stop the instance when you don't need the server; the model stays cached on the volume for the next start. Delete both when you're done:

```bash
featherless-cloud instance delete vllm
featherless-cloud volume delete models
```

## Next steps

<card-group>
<card icon="i-lucide-globe" title="Expose ports" to="/instances/ports">

Limits and security for public addresses.

</card>

<card icon="i-lucide-box" title="Save it as a template" to="/templates/manage">

Deploy the same server again in one step.

</card>
</card-group>
