Self-Hosted Mistral 7B Deployment Guide: From Order to OpenAI-Compatible API in One Hour

Mistral 7B Instruct v0.3 is the most-deployed open-weight LLM in our customer base — 7B params, 32K context, native function calling, Apache 2.0. This is the deployment runbook from order to live endpoint.

Hardware pick

• Cost-anchored, low concurrency: RTX 3090 24 GB FP16
• Production, modern: RTX 5090 32 GB FP8
• Latency-critical single-stream: RTX 5080 16 GB FP8
• High concurrency (50+ users): RTX 5090

Install

sudo apt update && sudo apt install -y python3.10-venv
python3.10 -m venv ~/vllm-env && source ~/vllm-env/bin/activate
pip install --upgrade pip wheel
pip install vllm==0.6.3

huggingface-cli login   # (optional — Mistral weights are open)
huggingface-cli download mistralai/Mistral-7B-Instruct-v0.3 \
  --local-dir /data/mistral-7b

Launch vLLM

vllm serve /data/mistral-7b \
  --host 0.0.0.0 --port 8000 \
  --quantization fp8 \
  --max-model-len 32768 \
  --max-num-seqs 64 \
  --gpu-memory-utilization 0.92 \
  --enable-prefix-caching \
  --served-model-name mistral-7b \
  --api-key sk-internal-token

Verify:

curl http://localhost:8000/v1/models \
  -H "Authorization: Bearer sk-internal-token"

Function calling

Mistral 7B Instruct v0.3 supports OpenAI-compatible tools natively:

from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="sk-internal-token")
tools = [{"type": "function", "function": {
    "name": "get_weather",
    "parameters": {"type":"object","properties":{"city":{"type":"string"}}}
}}]
resp = client.chat.completions.create(
    model="mistral-7b",
    messages=[{"role": "user", "content": "Weather in London?"}],
    tools=tools, tool_choice="auto",
)

Production hardening

• systemd unit with Restart=on-failure
• LiteLLM in front for per-key auth + rate limiting
• Caddy reverse proxy with TLS
• Prometheus + Grafana for metrics
• Pin vLLM and model versions in a build manifest