Build a Podcast Summary Agent: Audio Processing, Transcription, and Key Takeaway Extraction

Why Build a Podcast Summary Agent

The average podcast episode is 45 to 90 minutes long. Listening at 1.5x speed still takes 30 to 60 minutes per episode. With hundreds of podcasts publishing weekly, staying informed through audio alone is unsustainable. A podcast summary agent converts audio to text, detects topic boundaries, extracts the key insights, and produces a structured summary you can scan in two minutes.

This tutorial builds the complete pipeline: audio metadata fetching, transcription simulation, chapter detection, takeaway extraction, and a conversational agent interface.

Project Setup

mkdir podcast-agent && cd podcast-agent
python -m venv venv && source venv/bin/activate
pip install openai-agents pydantic
mkdir -p src
touch src/__init__.py src/audio_fetcher.py src/transcriber.py
touch src/chapter_detector.py src/summarizer.py src/agent.py

Step 1: Podcast Metadata and Audio Fetcher

We simulate podcast fetching. In production, use feedparser for RSS feeds and requests for audio downloads.

flowchart LR
    INPUT(["User intent"])
    PARSE["Parse plus<br/>classify"]
    PLAN["Plan and tool<br/>selection"]
    AGENT["Agent loop<br/>LLM plus tools"]
    GUARD{"Guardrails<br/>and policy"}
    EXEC["Execute and<br/>verify result"]
    OBS[("Trace and metrics")]
    OUT(["Outcome plus<br/>next action"])
    INPUT --> PARSE --> PLAN --> AGENT --> GUARD
    GUARD -->|Pass| EXEC --> OUT
    GUARD -->|Fail| AGENT
    AGENT --> OBS
    style AGENT fill:#4f46e5,stroke:#4338ca,color:#fff
    style GUARD fill:#f59e0b,stroke:#d97706,color:#1f2937
    style OBS fill:#ede9fe,stroke:#7c3aed,color:#1e1b4b
    style OUT fill:#059669,stroke:#047857,color:#fff

# src/audio_fetcher.py
from pydantic import BaseModel

class PodcastEpisode(BaseModel):
    id: str
    title: str
    show: str
    duration_min: int
    published: str
    audio_url: str
    description: str

MOCK_EPISODES: dict[str, PodcastEpisode] = {
    "ep001": PodcastEpisode(
        id="ep001",
        title="The Future of AI Agents in Enterprise",
        show="Tech Frontiers",
        duration_min=52,
        published="2026-03-15",
        audio_url="https://example.com/audio/ep001.mp3",
        description="Deep dive into how AI agents are transforming enterprise workflows.",
    ),
    "ep002": PodcastEpisode(
        id="ep002",
        title="Building Resilient Distributed Systems",
        show="Software Engineering Radio",
        duration_min=67,
        published="2026-03-14",
        audio_url="https://example.com/audio/ep002.mp3",
        description="Expert discussion on fault tolerance, consensus, and observability.",
    ),
    "ep003": PodcastEpisode(
        id="ep003",
        title="Startup Fundraising in the AI Era",
        show="Venture Stories",
        duration_min=43,
        published="2026-03-13",
        audio_url="https://example.com/audio/ep003.mp3",
        description="VCs discuss what they look for in AI startup pitches.",
    ),
}

def fetch_episode(episode_id: str) -> PodcastEpisode | None:
    return MOCK_EPISODES.get(episode_id)

def list_episodes() -> list[dict]:
    return [
        {"id": ep.id, "title": ep.title, "show": ep.show,
         "duration": f"{ep.duration_min}min"}
        for ep in MOCK_EPISODES.values()
    ]

Step 2: Transcription Engine

We simulate transcription output. In production, use OpenAI Whisper, AssemblyAI, or Deepgram.

# src/transcriber.py

MOCK_TRANSCRIPTS: dict[str, list[dict]] = {
    "ep001": [
        {"timestamp": "00:00", "speaker": "Host",
         "text": "Welcome to Tech Frontiers. Today we are exploring how AI agents are reshaping enterprise software. Our guest is Dr. Sarah Chen, who leads AI strategy at a Fortune 500 company."},
        {"timestamp": "02:30", "speaker": "Guest",
         "text": "The biggest shift we are seeing is from chatbots to autonomous agents. Chatbots answer questions. Agents complete multi-step workflows independently. They can research, draft documents, send emails, and update databases without human intervention at each step."},
        {"timestamp": "08:15", "speaker": "Host",
         "text": "What about reliability? Enterprises cannot afford agents that hallucinate or take wrong actions."},
        {"timestamp": "09:00", "speaker": "Guest",
         "text": "That is the key challenge. We use guardrails at three levels. Input validation checks that the agent received the right instructions. Output validation verifies the result matches expected schemas. And human-in-the-loop approval gates for high-stakes actions like financial transactions."},
        {"timestamp": "18:30", "speaker": "Host",
         "text": "Let us talk about ROI. What numbers are you seeing?"},
        {"timestamp": "19:00", "speaker": "Guest",
         "text": "Our customer service agents handle 60 percent of tickets end-to-end. That reduced response time from 4 hours to 8 minutes and saved 2.3 million dollars annually. The key metric is resolution rate, not just deflection rate."},
        {"timestamp": "32:00", "speaker": "Host",
         "text": "Where do you see this going in the next two years?"},
        {"timestamp": "32:30", "speaker": "Guest",
         "text": "Multi-agent systems will become standard. You will have specialized agents for legal review, financial analysis, and customer interaction, all coordinated by an orchestrator agent. The enterprise AI stack will look very different from what we have today."},
        {"timestamp": "48:00", "speaker": "Host",
         "text": "Fascinating insights. Thank you, Dr. Chen. Listeners, the key takeaway is that AI agents are moving from experiments to core infrastructure. Start small, measure resolution rates, and build guardrails from day one."},
    ],
    "ep002": [
        {"timestamp": "00:00", "speaker": "Host",
         "text": "Today on Software Engineering Radio, we discuss building distributed systems that survive failures gracefully."},
        {"timestamp": "05:00", "speaker": "Guest",
         "text": "The fundamental principle is design for failure. Every network call will eventually fail. Every disk will eventually corrupt data. Your system must handle these cases without losing customer data."},
        {"timestamp": "20:00", "speaker": "Guest",
         "text": "Circuit breakers prevent cascade failures. When a downstream service starts timing out, the circuit breaker opens and returns a fallback response immediately instead of holding connections."},
        {"timestamp": "40:00", "speaker": "Guest",
         "text": "Observability is non-negotiable. You need structured logging, distributed tracing, and meaningful metrics. Without these, debugging production issues becomes guesswork."},
        {"timestamp": "60:00", "speaker": "Host",
         "text": "Great discussion. The core message is clear: build systems assuming everything will break, and invest in observability from the start."},
    ],
}

def transcribe_episode(episode_id: str) -> list[dict] | None:
    return MOCK_TRANSCRIPTS.get(episode_id)

def get_full_text(episode_id: str) -> str:
    transcript = MOCK_TRANSCRIPTS.get(episode_id, [])
    return "\n".join(
        f"[{seg['timestamp']}] {seg['speaker']}: {seg['text']}"
        for seg in transcript
    )

Step 3: Chapter Detection

The chapter detector identifies topic shifts based on timestamp gaps and content analysis.

# src/chapter_detector.py

def detect_chapters(transcript: list[dict]) -> list[dict]:
    if not transcript:
        return []

    chapters = []
    current_chapter = {
        "start": transcript[0]["timestamp"],
        "title": "Introduction",
        "segments": [transcript[0]],
    }

    for i in range(1, len(transcript)):
        prev_min = _ts_to_minutes(transcript[i - 1]["timestamp"])
        curr_min = _ts_to_minutes(transcript[i]["timestamp"])

        if curr_min - prev_min > 8:
            chapters.append(current_chapter)
            current_chapter = {
                "start": transcript[i]["timestamp"],
                "title": _infer_title(transcript[i]["text"]),
                "segments": [transcript[i]],
            }
        else:
            current_chapter["segments"].append(transcript[i])

    chapters.append(current_chapter)
    return chapters

def _ts_to_minutes(ts: str) -> float:
    parts = ts.split(":")
    return int(parts[0]) * 60 + int(parts[1])

def _infer_title(text: str) -> str:
    words = text.split()[:8]
    return " ".join(words) + "..."

def format_chapters(chapters: list[dict]) -> str:
    lines = []
    for i, ch in enumerate(chapters, 1):
        lines.append(
            f"Chapter {i}: {ch['title']} (starts at {ch['start']})"
        )
        lines.append(
            f"  Segments: {len(ch['segments'])} speaker turns"
        )
    return "\n".join(lines)

Step 4: Summary Generator

# src/summarizer.py

def extract_takeaways(transcript: list[dict]) -> list[str]:
    takeaways = []
    keywords = [
        "key", "important", "takeaway", "biggest", "fundamental",
        "million", "percent", "principle", "core message",
    ]
    for seg in transcript:
        text_lower = seg["text"].lower()
        if any(kw in text_lower for kw in keywords):
            takeaways.append(seg["text"][:200])
    return takeaways if takeaways else ["No key takeaways detected."]

def generate_summary(
    episode_title: str,
    transcript: list[dict],
    chapters: list[dict],
) -> str:
    takeaways = extract_takeaways(transcript)
    lines = [f"=== Summary: {episode_title} ===\n"]
    lines.append(f"Chapters: {len(chapters)}")
    lines.append(f"Speaker turns: {len(transcript)}\n")
    lines.append("Key Takeaways:")
    for i, t in enumerate(takeaways, 1):
        lines.append(f"  {i}. {t}")
    lines.append("\nChapter Overview:")
    for ch in chapters:
        lines.append(f"  [{ch['start']}] {ch['title']}")
    return "\n".join(lines)

Step 5: Assemble the Agent

# src/agent.py
import asyncio
import json
from agents import Agent, Runner, function_tool
from src.audio_fetcher import fetch_episode, list_episodes
from src.transcriber import transcribe_episode, get_full_text
from src.chapter_detector import detect_chapters, format_chapters
from src.summarizer import generate_summary

@function_tool
def get_available_episodes() -> str:
    """List available podcast episodes."""
    episodes = list_episodes()
    return json.dumps(episodes, indent=2)

@function_tool
def summarize_episode(episode_id: str) -> str:
    """Transcribe and summarize a podcast episode."""
    episode = fetch_episode(episode_id)
    if not episode:
        return "Episode not found."
    transcript = transcribe_episode(episode_id)
    if not transcript:
        return "Transcription not available."
    chapters = detect_chapters(transcript)
    return generate_summary(episode.title, transcript, chapters)

@function_tool
def get_transcript(episode_id: str) -> str:
    """Get the full transcript of an episode."""
    text = get_full_text(episode_id)
    return text if text else "Transcript not available."

@function_tool
def get_chapters(episode_id: str) -> str:
    """Get chapter breakdown for an episode."""
    transcript = transcribe_episode(episode_id)
    if not transcript:
        return "Episode not found."
    chapters = detect_chapters(transcript)
    return format_chapters(chapters)

podcast_agent = Agent(
    name="Podcast Summarizer",
    instructions="""You are a podcast summary agent.
Help users quickly understand podcast content without
listening to full episodes. Provide summaries, key
takeaways, chapter breakdowns, and full transcripts.
Highlight actionable insights and notable quotes.""",
    tools=[
        get_available_episodes, summarize_episode,
        get_transcript, get_chapters,
    ],
)

async def main():
    result = await Runner.run(
        podcast_agent,
        "What episodes are available? Summarize the one "
        "about AI agents and give me the key takeaways.",
    )
    print(result.final_output)

if __name__ == "__main__":
    asyncio.run(main())

The agent lists episodes, identifies the AI agents episode, transcribes it, detects chapters, and produces a structured summary with the most important insights extracted.

FAQ

How do I connect this to real audio transcription?

Install OpenAI's Whisper library (pip install openai-whisper) or use the OpenAI Audio API. Replace transcribe_episode with a function that downloads the MP3 file and sends it to Whisper for transcription. Whisper returns timestamped segments, which map directly to the transcript format used by the chapter detector and summarizer.

Can the agent handle episodes in different languages?

Yes. Whisper supports over 90 languages and can auto-detect the source language. Add a detected_language field to the transcription output and optionally translate foreign-language transcripts to English before summarization. The chapter detection works on any language since it relies on timestamp gaps rather than language-specific keywords.

How would I process a podcast feed automatically?

Use feedparser to monitor RSS feeds and detect new episodes. When a new episode appears, the agent automatically downloads, transcribes, summarizes, and stores the result. Set this up as a scheduled task that runs every few hours, building a searchable archive of podcast summaries over time.

---

#Podcast #Transcription #AIAgent #Python #AudioProcessing #AgenticAI #LearnAI #AIEngineering