RTX 4090 24GB for Qwen 2.5-VL 7B: OCR, Charts, Video and Throughput

Qwen 2.5-VL 7B is the current sweet-spot multimodal model: strong OCR (especially on Asian and structured layouts), native video frame ingestion, grounded JSON output with bounding boxes, and a permissive Apache 2.0 license. On a RTX 4090 24GB server from Gigagpu UK hosting, FP8 weights occupy ~7.5 GB, image encode at 896 px takes ~180 ms, and decode runs at ~150 t/s – enough to keep batched OCR pipelines above 1,000 docs/hour on a single card. This guide walks through the dynamic-resolution ViT design, the full latency envelope from avatars to A4 scans to 30-second video clips, OCR quality on UK documents, and the production gotchas that bite people deploying it for the first time.

Contents

• Why Qwen 2.5-VL on a 4090
• Architecture and VRAM map
• Encoder latency by resolution
• Decode throughput and concurrency
• Video ingestion
• OCR quality and benchmarks
• Deployment configuration
• Production gotchas
• Verdict and alternatives

Why Qwen 2.5-VL on a 4090

What Alibaba built and why it matters

Qwen 2.5-VL 7B is a 7B-parameter dense text decoder (28 transformer layers, GQA grouping) paired with a dynamic-resolution Vision Transformer. The ViT does not pad images to a fixed grid – it produces a variable number of patch tokens proportional to the input pixel count, which means a 448 x 448 thumbnail consumes 256 image tokens while an A4 scan at 1568 x 1176 consumes 1,872. Image tokens are concatenated directly into the text stream (the opposite design choice to Llama 3.2 Vision’s cross-attention adapter), so they share the autoregressive KV cache. Qwen 2.5-VL also natively ingests video as a sequence of sampled frames using temporal RoPE, and emits grounded JSON with bbox coordinates without any post-training surgery. Apache 2.0 license clears the path for commercial use.

Why the 4090 is a comfortable home

The Ada AD102 die in the RTX 4090 brings 16,384 CUDA cores, 24 GB of GDDR6X at 1,008 GB/s, native 4th-gen FP8 tensor cores, and a 450 W TDP. Qwen 2.5-VL 7B at FP8 occupies only ~9.5 GB total including the ViT and a 16k KV cache, leaving roughly 14 GB of free VRAM for high concurrency, video clip processing, or co-tenanting with a Whisper sidecar. The bandwidth headroom is what makes batched OCR farms practical: the model can sustain 432 t/s aggregate decode at batch 4 while still encoding a fresh image every 180 ms.

Architecture and VRAM map

Format-by-format footprint

What the table tells you operationally

Unlike Llama 3.2 Vision which barely fits in 24 GB FP16, Qwen 2.5-VL FP8 leaves more than half the card free. That headroom is what makes 8-image batches viable, what lets you co-locate a Faster-Whisper instance, and what keeps the worker stable under traffic bursts. AWQ-on-text + FP8-on-vision shaves another 2 GB but gives back ~10% on decode speed; the FP8-everywhere path is the production default.

Encoder latency by resolution

Wall-clock encode times in FP8

Batched encode efficiency

Decode throughput and concurrency

Decode by batch size, FP8 weights

Cross-card decode comparison

The 4090 is the strict price-per-throughput winner for self-hosted Qwen 2.5-VL OCR. See 4090 vs 5090 and 4090 vs H100 for cost-per-image breakdowns.

Video ingestion

Qwen 2.5-VL processes video as a sequence of sampled frames using temporal RoPE that encodes both spatial position within frame and temporal position across frames. Sampling 1 fps from a 30-second clip yields 30 frames; on the 4090 in FP8 that batched encode takes ~2.4 seconds and produces a 7,680-token prefix that fits comfortably within an 8k decode budget. Action-recognition prompts on the 30-second clip return in under five seconds end-to-end. For longer clips, increase the context budget proportionally and expect linear scaling on encode time.

OCR quality and benchmarks

Public benchmark scores

Field extraction on 500 sampled UK invoices

Deployment configuration

vLLM launch (FP8, 16k context, 8-image batching, video enabled)

python -m vllm.entrypoints.openai.api_server \
  --model Qwen/Qwen2.5-VL-7B-Instruct \
  --quantization fp8 --kv-cache-dtype fp8_e4m3 \
  --max-model-len 16384 --max-num-seqs 8 \
  --limit-mm-per-prompt image=4,video=1 \
  --enable-chunked-prefill --enable-prefix-caching \
  --gpu-memory-utilization 0.92

Test rig and methodology

All numbers above were captured on a single-tenant Gigagpu node: RTX 4090 24GB Founders Edition (450 W stock), Ryzen 9 7950X with 64 GB DDR5-5600, Samsung 990 Pro 2TB Gen 4 NVMe; Ubuntu 24.04 LTS, NVIDIA driver 560.x, CUDA 12.6, vLLM 0.6.4, PyTorch 2.5, FlashAttention 2.6. Decode throughput is sustained mean over 60-second windows; image encode is mean of 50 samples after warm-up. UK document corpus was 500 sampled invoices anonymised and labelled by hand. See our vLLM setup guide for installation steps and FP8 deployment notes for tuning details.

Production gotchas

1. Image tokens compete for KV space. Unlike Llama 3.2 Vision’s cross-attention design, Qwen 2.5-VL prepends image tokens directly to the text stream. A 4-image A4 batch consumes ~7,488 image tokens of your 16k budget; plan accordingly or downscale to 896 px.
2. Dynamic-resolution ViT means variable encode time. Batching mixed-resolution images causes padding and wastes compute. Group by resolution before encoding for sustained throughput.
3. The video frame sampler is your responsibility. vLLM accepts video as a list of pre-decoded frames. You need ffmpeg or equivalent to sample at your chosen fps before sending to the model.
4. Bounding-box output requires the right prompt template. Use the official Qwen 2.5-VL chat template; ad-hoc prompts will get text answers without grounded boxes. The official template emits <|box_start|> and <|box_end|> tokens that your client must parse.
5. FP16 ViT in an FP8 LLM stack causes a kernel mode switch. Keep the entire stack at FP8 to avoid the 8-12 ms penalty per image of the precision mode switch.
6. Asian script OCR benefits from higher resolution. Default 896 px is fine for Latin script; Chinese, Japanese, and Korean documents benefit from 1120 px or higher, at proportional encode cost.
7. Multi-tenant prefix caching is brittle on multimodal. Image tokens have very low cross-request cache hit rates; do not over-allocate prefix cache memory.

Verdict and alternatives

Qwen 2.5-VL 7B on a 4090 is the right pick when (a) OCR throughput is your primary KPI, especially on Asian or table-heavy documents, (b) you need native video frame ingestion without a separate pipeline, and (c) you need grounded bounding-box output for downstream layout-aware processing. For pure natural-image reasoning with Western documents and longer text contexts, Llama 3.2 Vision 11B is competitive. For text-only workloads at higher throughput, drop to Llama 3 8B. For maximum vision quality if VRAM is no object, the 32B Qwen-VL variant on an H100 outperforms but at much higher cost-per-image.

See also: Llama 3.2 Vision 11B, multimodal use cases, vLLM setup, FP8 deployment, prefill/decode, concurrent users, 4090 vs 5090, tokens per watt.