Podcast Transcription: Whisper + Diarization

You will build a pipeline that takes podcast audio files and produces speaker-labelled transcripts with timestamps. A podcast production company processing 40 episodes per week (averaging 55 minutes each) gets publication-ready transcripts in under 8 minutes per episode instead of 3 hours of manual transcription. The entire pipeline runs on a single dedicated GPU server.

Pipeline Architecture

Audio Preprocessing

import subprocess
import torch

def preprocess_audio(input_path: str, output_path: str):
    """Normalise audio to 16kHz mono WAV."""
    subprocess.run([
        "ffmpeg", "-i", input_path,
        "-ar", "16000", "-ac", "1",
        "-af", "loudnorm=I=-16:TP=-1.5:LRA=11",
        output_path, "-y"
    ], check=True)

# Voice Activity Detection to skip silence
from silero_vad import load_silero_vad, get_speech_timestamps

vad_model = load_silero_vad()

def get_speech_segments(audio_tensor: torch.Tensor) -> list:
    segments = get_speech_timestamps(audio_tensor, vad_model,
                                      threshold=0.5,
                                      min_speech_duration_ms=500)
    return segments

Loudness normalisation ensures consistent transcription accuracy across episodes with varying recording quality. VAD identifies speech regions, allowing Whisper to skip extended music intros and ad breaks.

Whisper Transcription

from faster_whisper import WhisperModel

model = WhisperModel("large-v3", device="cuda", compute_type="float16")

def transcribe(audio_path: str) -> list:
    segments, info = model.transcribe(
        audio_path,
        beam_size=5,
        language="en",
        vad_filter=True,
        vad_parameters=dict(min_silence_duration_ms=500),
        word_timestamps=True
    )
    results = []
    for seg in segments:
        results.append({
            "start": seg.start, "end": seg.end,
            "text": seg.text.strip(),
            "words": [{"word": w.word, "start": w.start, "end": w.end}
                      for w in seg.words] if seg.words else []
        })
    return results

Faster-Whisper with CTranslate2 delivers 4x speed over standard Whisper. A 55-minute episode transcribes in approximately 3 minutes on an RTX 5090.

Speaker Diarization

from pyannote.audio import Pipeline

diarize = Pipeline.from_pretrained(
    "pyannote/speaker-diarization-3.1",
    use_auth_token="your_hf_token"
).to(torch.device("cuda"))

def identify_speakers(audio_path: str, num_speakers: int = None) -> list:
    params = {"min_speakers": 2, "max_speakers": 6}
    if num_speakers:
        params = {"num_speakers": num_speakers}
    diarization = diarize(audio_path, **params)
    speakers = []
    for turn, _, speaker in diarization.itertracks(yield_label=True):
        speakers.append({
            "start": turn.start, "end": turn.end,
            "speaker": speaker
        })
    return speakers

pyannote.audio 3.1 identifies individual speakers without prior voice samples. For regular podcast hosts, provide reference audio clips to improve speaker identification consistency across episodes.

Merging Transcription with Speakers

def merge_transcript_speakers(transcript: list, speakers: list) -> list:
    """Assign speaker labels to transcript segments."""
    merged = []
    for seg in transcript:
        seg_mid = (seg["start"] + seg["end"]) / 2
        speaker = "Unknown"
        for sp in speakers:
            if sp["start"] <= seg_mid <= sp["end"]:
                speaker = sp["speaker"]
                break
        merged.append({
            "speaker": speaker,
            "start": format_timestamp(seg["start"]),
            "end": format_timestamp(seg["end"]),
            "text": seg["text"]
        })
    return collapse_consecutive_speakers(merged)

def format_timestamp(seconds: float) -> str:
    m, s = divmod(int(seconds), 60)
    h, m = divmod(m, 60)
    return f"{h:02d}:{m:02d}:{s:02d}"

Output Formats and Production

Export transcripts as SRT (for video platforms), VTT (for web players), plain text (for show notes), and JSON (for search indexing). Add chapter markers by detecting topic changes with an LLM pass over the transcript. An RTX 5090 (24 GB) runs both Whisper large-v3 and pyannote.audio simultaneously, processing a 55-minute episode end-to-end in under 8 minutes. For higher throughput, an RTX 6000 Pro (48 GB) handles batch processing of multiple episodes in parallel. Deploy on private infrastructure for client confidentiality. See vLLM hosting for adding LLM summarisation, chatbot hosting for searchable podcast archives, GDPR compliance, more tutorials, and use cases.