What You’ll Build
In about 90 minutes, you will have a video captioning pipeline that extracts audio from video files, generates word-level accurate transcriptions with speaker labels, formats captions into proper subtitle segments with intelligent line breaking, and exports SRT, VTT, or embedded subtitle formats in the original language plus any number of translations. A one-hour video captions in under five minutes on a single dedicated GPU server.
Accessibility regulations increasingly require captioned video content. Professional captioning costs $1-3 per minute of video. Auto-generated captions from social platforms are often inaccurate and lack speaker identification. A self-hosted captioning system on open-source models produces broadcast-quality captions at machine speed while keeping proprietary video content off external processing servers.
Architecture Overview
The pipeline chains three models: Whisper for speech-to-text transcription with word-level timestamps, a speaker diarisation model for multi-speaker identification, and an LLM through vLLM for caption formatting, punctuation correction, and optional translation into target languages. LangChain coordinates the processing stages and handles batch workflows for video libraries.
The caption formatter is the quality differentiator. Raw transcription output needs intelligent segmentation: captions should break at natural phrase boundaries, not mid-sentence. The LLM analyses the transcript and creates caption segments that respect reading speed limits (typically 20 characters per second), synchronise with speaker changes, and break at grammatically natural points. For translation, the LLM translates caption segments while respecting display timing constraints.
GPU Requirements
| Video Volume | Recommended GPU | VRAM | Processing Speed |
|---|---|---|---|
| Up to 10 hours/day | RTX 5090 | 24 GB | ~12x real-time |
| 10 – 50 hours/day | RTX 6000 Pro | 40 GB | ~20x real-time |
| 50+ hours/day | RTX 6000 Pro 96 GB | 80 GB | ~30x real-time |
Whisper dominates processing time for the captioning pipeline. The large-v3 model delivers the best accuracy at approximately 8x real-time on an RTX 6000 Pro. The formatting and translation LLM adds minimal overhead since caption text is relatively short per video. See our self-hosted LLM guide for optimising the Whisper and LLM model combination on a single GPU.
Step-by-Step Build
Deploy Whisper and vLLM on your GPU server. Install FFmpeg for audio extraction and a diarisation library like pyannote-audio for speaker identification. Build the captioning pipeline that processes video files end-to-end.
# Caption formatting prompt
FORMAT_PROMPT = """Format this transcription into subtitle captions.
Language: {language}
Max characters per line: 42
Max lines per caption: 2
Max reading speed: 20 chars/second
Speaker labels: {speaker_map}
Transcription with timestamps:
{timestamped_transcript}
Return JSON array of captions:
[{start: "HH:MM:SS,mmm", end: "HH:MM:SS,mmm",
speaker: "Speaker Name",
text: "Caption text with natural line breaks"}]
Break captions at natural phrase boundaries.
Keep speaker attribution when speakers change."""Export modules generate SRT, VTT, and ASS subtitle formats from the structured caption data. For burnt-in captions, FFmpeg overlays the subtitle track onto the video. Batch mode processes entire video libraries overnight with parallel processing for independent files. See our voice processing guide for audio pipeline optimisation and vLLM production setup for inference configuration.
Performance and Accuracy
On an RTX 6000 Pro running Whisper large-v3 and Llama 3 8B, a one-hour English video captions in 4.5 minutes with a word error rate below 4% for clear speech. Adding translation to three target languages adds 90 seconds. Speaker diarisation accuracy reaches 88% for conversations with clear turn-taking. Caption formatting produces segments with 95% compliance against broadcast subtitle standards for line length and reading speed.
The system handles challenging audio scenarios including background music, overlapping speech, accented speakers, and low-quality recordings by adjusting confidence thresholds and flagging uncertain segments for review. Multi-language video content auto-detects the spoken language and captions accordingly using multilingual audio processing capabilities.
Deploy Your Captioning Pipeline
Automated video captioning meets accessibility requirements at scale without per-minute processing fees. Caption your entire video library in hours instead of weeks, in any language combination. Launch on GigaGPU dedicated GPU hosting and make your video content accessible today. Browse more automation patterns in our use case library.
