Conversation Funnel Analysis: Tracking User Journeys Through AI Agent Interactions

What Is Conversation Funnel Analysis

In web analytics, a funnel tracks users through stages like landing page, product page, cart, and checkout. Conversation funnel analysis applies the same concept to AI agent interactions. Users enter a conversation, progress through stages like greeting, problem identification, solution delivery, and confirmation, and either reach a successful resolution or drop off at some point.

Understanding where users drop off reveals exactly which parts of your agent need improvement. A 90% greeting-to-identification rate but a 40% identification-to-resolution rate tells you the agent struggles with solving problems, not understanding them.

Defining Funnel Stages

Every agent conversation can be decomposed into stages. The specific stages depend on your use case, but a general framework works for most support and sales agents.

flowchart LR
    INPUT(["User intent"])
    PARSE["Parse plus<br/>classify"]
    PLAN["Plan and tool<br/>selection"]
    AGENT["Agent loop<br/>LLM plus tools"]
    GUARD{"Guardrails<br/>and policy"}
    EXEC["Execute and<br/>verify result"]
    OBS[("Trace and metrics")]
    OUT(["Outcome plus<br/>next action"])
    INPUT --> PARSE --> PLAN --> AGENT --> GUARD
    GUARD -->|Pass| EXEC --> OUT
    GUARD -->|Fail| AGENT
    AGENT --> OBS
    style AGENT fill:#4f46e5,stroke:#4338ca,color:#fff
    style GUARD fill:#f59e0b,stroke:#d97706,color:#1f2937
    style OBS fill:#ede9fe,stroke:#7c3aed,color:#1e1b4b
    style OUT fill:#059669,stroke:#047857,color:#fff

from enum import Enum
from dataclasses import dataclass, field
from datetime import datetime

class FunnelStage(Enum):
    INITIATED = "initiated"
    GREETED = "greeted"
    PROBLEM_IDENTIFIED = "problem_identified"
    SOLUTION_PROPOSED = "solution_proposed"
    SOLUTION_ACCEPTED = "solution_accepted"
    RESOLVED = "resolved"
    ABANDONED = "abandoned"

STAGE_ORDER = [
    FunnelStage.INITIATED,
    FunnelStage.GREETED,
    FunnelStage.PROBLEM_IDENTIFIED,
    FunnelStage.SOLUTION_PROPOSED,
    FunnelStage.SOLUTION_ACCEPTED,
    FunnelStage.RESOLVED,
]

@dataclass
class ConversationProgress:
    conversation_id: str
    user_id: str
    stages_reached: list[FunnelStage] = field(default_factory=list)
    timestamps: dict[str, str] = field(default_factory=dict)
    final_stage: FunnelStage = FunnelStage.INITIATED

    def advance(self, stage: FunnelStage) -> None:
        if stage not in self.stages_reached:
            self.stages_reached.append(stage)
            self.timestamps[stage.value] = datetime.utcnow().isoformat()
            self.final_stage = stage

Stage Classification

The hardest part of funnel analysis is determining which stage a conversation has reached. You can use rule-based classification, LLM-based classification, or a hybrid approach.

from openai import OpenAI
import json

client = OpenAI()

CLASSIFIER_PROMPT = """Analyze this conversation between a user and an AI agent.
Determine which stages the conversation has reached.

Stages:
- initiated: conversation started
- greeted: agent acknowledged user
- problem_identified: user's issue is clearly understood
- solution_proposed: agent offered a specific solution
- solution_accepted: user agreed to the solution
- resolved: issue is fully resolved

Return a JSON object: {"stages_reached": ["stage1", "stage2", ...]}
"""

def classify_conversation(messages: list[dict]) -> list[str]:
    formatted = "\n".join(
        f"{m['role']}: {m['content']}" for m in messages
    )
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": CLASSIFIER_PROMPT},
            {"role": "user", "content": formatted},
        ],
        response_format={"type": "json_object"},
    )
    result = json.loads(response.choices[0].message.content)
    return result.get("stages_reached", [])

Computing Funnel Metrics

With classified conversations, you can calculate the conversion rate between every pair of consecutive stages.

from collections import Counter

def compute_funnel(conversations: list[ConversationProgress]) -> list[dict]:
    stage_counts = Counter()
    for conv in conversations:
        for stage in conv.stages_reached:
            stage_counts[stage] += 1

    funnel = []
    for i, stage in enumerate(STAGE_ORDER):
        count = stage_counts.get(stage, 0)
        prev_count = stage_counts.get(STAGE_ORDER[i - 1], 0) if i > 0 else len(conversations)
        conversion_rate = (count / prev_count * 100) if prev_count > 0 else 0
        funnel.append({
            "stage": stage.value,
            "count": count,
            "conversion_rate": round(conversion_rate, 1),
            "drop_off": prev_count - count if i > 0 else 0,
        })
    return funnel

def print_funnel(funnel: list[dict]) -> None:
    print(f"{'Stage':<25} {'Count':>8} {'Conv %':>8} {'Drop-off':>10}")
    print("-" * 55)
    for step in funnel:
        print(
            f"{step['stage']:<25} {step['count']:>8} "
            f"{step['conversion_rate']:>7.1f}% {step['drop_off']:>10}"
        )

Drop-Off Analysis

Identifying where users drop off is only half the battle. You also need to understand why. Analyzing the last messages before abandonment reveals common patterns.

def analyze_dropoffs(
    conversations: list[ConversationProgress],
    messages_store: dict[str, list[dict]],
    target_stage: FunnelStage,
) -> list[dict]:
    dropoffs = []
    prev_idx = STAGE_ORDER.index(target_stage) - 1
    prev_stage = STAGE_ORDER[prev_idx] if prev_idx >= 0 else None

    for conv in conversations:
        reached = set(conv.stages_reached)
        if prev_stage in reached and target_stage not in reached:
            msgs = messages_store.get(conv.conversation_id, [])
            last_user_msg = ""
            for m in reversed(msgs):
                if m["role"] == "user":
                    last_user_msg = m["content"]
                    break
            dropoffs.append({
                "conversation_id": conv.conversation_id,
                "final_stage": conv.final_stage.value,
                "last_user_message": last_user_msg,
            })
    return dropoffs

FAQ

How many conversations do I need before funnel analysis is meaningful?

Aim for at least 500 conversations per funnel to get statistically significant conversion rates. Below that threshold, individual conversations have too much influence on the percentages. For A/B testing prompt changes, you typically need 1,000 or more per variant to detect a 5% difference in conversion.

Should I classify stages in real-time or batch?

Both approaches have merit. Real-time classification lets you trigger interventions, like escalating to a human when the agent fails to identify the problem after three exchanges. Batch classification is cheaper and lets you use more sophisticated models. Most teams start with nightly batch classification and add real-time for high-value triggers.

How do I handle conversations that skip stages?

It is normal for some conversations to skip stages. A returning user might jump straight to a solution request without a greeting phase. Track the stages actually reached rather than enforcing a strict linear progression. Your funnel should show percentages based on users who reached each stage, regardless of whether they passed through every prior stage.

---

#FunnelAnalysis #UserJourney #Conversion #Analytics #AIAgents #AgenticAI #LearnAI #AIEngineering