RTX 5060 Ti 16GB for SOLAR 10.7B

SOLAR-10.7B-Instruct-v1.0 from Upstage is a depth-upscaled Llama 2-architecture model, punching above its parameter count on general benchmarks. On the RTX 5060 Ti 16GB at our UK dedicated GPU hosting it’s a clean FP8 fit with room for useful context.

Contents

• Model overview
• VRAM and fit
• Throughput
• Deployment
• SOLAR vs peers
• When to pick SOLAR

Model Overview

SOLAR-10.7B is built via “depth up-scaling”: duplicating and re-tuning layers from a 7B Llama 2 base to produce a 10.7B-parameter model without training from scratch. 48 layers, 32 KV heads (no GQA – KV cache is heavier than Llama 3’s), 128 head dim. Apache 2.0 licensed. Strong at English reasoning and instruction following.

VRAM and Fit

Because SOLAR lacks GQA, its KV cache is heavier per token than Llama 3 or Qwen 2.5 – factor that into your context budget.

Throughput

• FP8 + FP8 KV decode at batch 1: ~90 t/s
• AWQ INT4 decode at batch 1: ~108 t/s
• Aggregate at batch 16 (FP8): ~410 t/s
• Prefill (FP8): ~4,800 tokens/sec

Deployment

python -m vllm.entrypoints.openai.api_server \
  --model upstage/SOLAR-10.7B-Instruct-v1.0 \
  --quantization fp8 \
  --kv-cache-dtype fp8 \
  --max-model-len 4096 \
  --enable-prefix-caching \
  --gpu-memory-utilization 0.92

Native context is 4k – for longer context use the AWQ variant plus YaRN or switch to a model with built-in long context.

SOLAR vs Peers

When to Pick SOLAR

• You want Apache 2.0 specifically and already evaluated SOLAR’s behaviour for your domain
• Short-context workloads (4k is enough) where quality matters more than context length
• You’ve built a fine-tune on SOLAR and the pipeline is stable

For new projects in 2026, Mistral Nemo 12B or Gemma 2 9B generally win on quality per VRAM. SOLAR remains a sensible pick only when you have a specific reason.

See also: Mistral Nemo 12B, Gemma 2 9B benchmark, Yi 9B, FP8 deployment.