RTX 5060 Ti 16GB Plus RTX 3090 Hybrid Server

A single-GPU server is simple, but a two-card heterogeneous chassis often delivers more capability per pound than any homogeneous pair. Pairing a RTX 5060 Ti 16GB with an RTX 3090 24GB on our dedicated GPU hosting gives you Blackwell FP8 speed on one card and 24 GB Ampere capacity on the other – each running the workload it wins on.

Contents

• Why heterogeneous
• Topology
• Workload allocation
• Router and OpenAI endpoints
• Cost vs a single bigger card
• Caveats

Why Heterogeneous

Heterogeneous tensor parallel is not viable – vLLM and TGI expect matched SMs, VRAM, and bandwidth when sharding a single model across cards. But model-per-card is excellent. You run two independent OpenAI-compatible endpoints on the same box, each tuned to its card.

Topology

                    +-------------------+
     client  --->    |  nginx / traefik  |
                    +---------+---------+
                              |
              +---------------+---------------+
              |                               |
     +--------v---------+            +--------v--------+
     | vllm on 5060 Ti  |            | vllm on 3090    |
     | CUDA_VISIBLE=0   |            | CUDA_VISIBLE=1  |
     | Llama 8B FP8     |            | Qwen 32B AWQ    |
     | :8000            |            | :8001           |
     +------------------+            +-----------------+

Workload Allocation

Router and OpenAI Endpoints

Front the two vLLM processes with nginx:

upstream small_model { server 127.0.0.1:8000; }
upstream large_model { server 127.0.0.1:8001; }

map $model_name $backend {
  "llama-3.1-8b"     small_model;
  "mistral-7b"       small_model;
  "qwen-2.5-32b"     large_model;
  "mixtral-8x7b"     large_model;
  default            small_model;
}

server {
  listen 443 ssl;
  location /v1/ {
    proxy_pass http://$backend;
  }
}

Your clients hit a single endpoint; the router sends the request to the correct card based on the model field in the payload.

Cost vs a Single Bigger Card

The hybrid gives you more total VRAM and two-model concurrency for less than one 5090.

Caveats

• Cannot tensor-parallel a single model across the two cards – use model-per-card only
• Driver matters: install the Blackwell-compatible driver (570+); works with the 3090 too
• Power and cooling: 180 + 350 = 530 W board draw; check PSU and thermal headroom
• Monitoring: set per-GPU Prometheus labels so you can see which card is saturated

See also: 5060 Ti vs 3090 benchmark, vLLM setup, FP8 deployment, Docker CUDA setup, first-day checklist.