What You’ll Build
In about two hours, you will have an AI meeting assistant that captures audio from virtual meetings, transcribes with speaker identification in real time, generates structured meeting notes within 60 seconds of the call ending, and stores everything in a searchable archive. Team members can query past meetings conversationally. The full system runs on a dedicated GPU server keeping all meeting content private.
The average professional spends 15 hours per week in meetings. Without reliable notes, decisions get lost and action items fall through. Commercial meeting assistants send your audio to third-party servers, which is unacceptable for board meetings, HR discussions, or client calls under NDA. A self-hosted assistant on open-source models gives you the same capabilities with zero data exposure.
Architecture Overview
The assistant has four components: an audio capture service that records meeting audio via virtual audio device or bot integration, Whisper for real-time transcription with speaker diarisation, an LLM through vLLM for summary and action item extraction, and a RAG-powered archive that enables conversational search across all past meetings. LangChain orchestrates the post-meeting processing pipeline.
During the meeting, Whisper processes audio in 30-second chunks for near-real-time transcription displayed in a live dashboard. After the call ends, the full transcript passes through the LLM for structured note generation. The voice processing infrastructure handles audio routing and format conversion. Completed notes push to Slack, email, or your project management tool via webhooks.
GPU Requirements
| Concurrent Meetings | Recommended GPU | VRAM | Transcription Latency |
|---|---|---|---|
| 1-3 simultaneous | RTX 5090 | 24 GB | ~3 sec delay |
| 3-8 simultaneous | RTX 6000 Pro | 40 GB | ~2 sec delay |
| 8-20 simultaneous | RTX 6000 Pro 96 GB | 80 GB | ~1.5 sec delay |
Real-time transcription requires Whisper to process faster than real-time, which it achieves at 8-10x on GPU. Multiple concurrent meetings share the GPU through batched inference. Summary generation happens post-meeting and adds 30-60 seconds per hour of content. See our self-hosted LLM guide for concurrent workload planning.
Step-by-Step Build
Set up your GPU server with Whisper and vLLM. Configure audio capture either through a meeting bot (using libraries like Discord.py for internal calls or browser automation for Zoom/Teams) or by routing audio through a virtual audio device. Build the real-time transcription pipeline with speaker diarisation using pyannote-audio.
# Post-meeting summary generation
MEETING_NOTES_PROMPT = """Generate structured meeting notes.
Meeting: {meeting_title}
Date: {date}
Participants: {speakers}
Duration: {duration}
Transcript:
{full_transcript}
Generate:
1. Executive Summary (3-5 sentences)
2. Key Discussion Points (bullet list with speaker attribution)
3. Decisions Made (numbered list)
4. Action Items (table: item, owner, deadline if mentioned)
5. Open Questions / Follow-ups
6. Next Meeting Topics (if discussed)"""The archive layer indexes all meeting transcripts and notes into the vector database for RAG-powered search. Users query the system through a chat interface or Slack bot asking questions like “What did we decide about the Q3 budget?” and get answers with meeting date and speaker attribution. See the voice agent server guide for audio pipeline implementation patterns.
Performance and Integration
On an RTX 6000 Pro, real-time transcription runs with a 2-second display delay per 30-second audio chunk. Post-meeting summary generation for a one-hour meeting completes in 45 seconds. Speaker diarisation accuracy reaches 89% for meetings with clear turn-taking and drops to 78% for cross-talk heavy discussions. Transcription word error rate stays below 5% for native English speakers in quiet environments.
Integration with calendar systems enables automatic recording of scheduled meetings. The assistant can join Zoom, Teams, or Google Meet via a bot account, capture the audio stream, and process it without any participant installing software. Post-meeting notes distribute automatically through existing communication channels using LangChain automation.
Deploy Your Meeting Assistant
A self-hosted meeting assistant eliminates the privacy concerns of commercial tools while delivering the same productivity gains. Every meeting becomes a searchable, queryable knowledge asset. Deploy on GigaGPU dedicated GPU hosting and never lose a meeting decision again. Explore more use case build patterns in our library.
