Building a Phone Screening Agent: AI-Powered Call Screening and Routing

Why AI Call Screening Changes Everything

Traditional call screening is binary — either you answer or you do not. AI screening adds intelligence: the agent answers every call, identifies the caller, understands their purpose, and makes a routing decision in seconds. Legitimate callers get connected quickly. Spam calls get blocked. High-priority calls jump the queue.

This pattern is valuable for businesses that receive high call volumes (medical practices, law firms, real estate agencies) and for executives who need a smart gatekeeper without hiring a human receptionist.

Architecture of a Screening Agent

The screening agent operates in three phases: identification, intent assessment, and routing decision. Here is the core structure:

flowchart LR
    CALLER(["Client"])
    subgraph TEL["Telephony"]
        SIP["Twilio SIP and PSTN"]
    end
    subgraph BRAIN["Salon AI Agent"]
        STT["Streaming STT<br/>Deepgram or Whisper"]
        NLU{"Intent and<br/>Entity Extraction"}
        TOOLS["Tool Calls"]
        TTS["Streaming TTS<br/>ElevenLabs or Rime"]
    end
    subgraph DATA["Live Data Plane"]
        CRM[("CRM and Notes")]
        CAL[("Calendar and<br/>Schedule")]
        KB[("Knowledge Base<br/>and Policies")]
    end
    subgraph OUT["Outcomes"]
        O1(["Appointment booked"])
        O2(["Reschedule completed"])
        O3(["Stylist handoff"])
    end
    CALLER --> SIP --> STT --> NLU
    NLU -->|Lookup| TOOLS
    TOOLS <--> CRM
    TOOLS <--> CAL
    TOOLS <--> KB
    NLU --> TTS --> SIP --> CALLER
    NLU -->|Resolved| O1
    NLU -->|Schedule| O2
    NLU -->|Escalate| O3
    style CALLER fill:#f1f5f9,stroke:#64748b,color:#0f172a
    style NLU fill:#4f46e5,stroke:#4338ca,color:#fff
    style O1 fill:#059669,stroke:#047857,color:#fff
    style O2 fill:#0ea5e9,stroke:#0369a1,color:#fff
    style O3 fill:#f59e0b,stroke:#d97706,color:#1f2937

from dataclasses import dataclass, field
from enum import Enum
from typing import Optional
from datetime import datetime

class CallerPriority(Enum):
    VIP = "vip"
    KNOWN = "known"
    NEW = "new"
    SUSPICIOUS = "suspicious"
    SPAM = "spam"

class RoutingAction(Enum):
    CONNECT_IMMEDIATELY = "connect_immediately"
    CONNECT_WITH_ANNOUNCE = "connect_with_announce"
    TAKE_MESSAGE = "take_message"
    SEND_TO_VOICEMAIL = "send_to_voicemail"
    BLOCK = "block"

@dataclass
class ScreeningResult:
    caller_number: str
    caller_name: Optional[str]
    priority: CallerPriority
    intent_summary: str
    routing_action: RoutingAction
    confidence: float
    metadata: dict = field(default_factory=dict)

class PhoneScreeningAgent:
    """AI-powered call screening and routing agent."""

    def __init__(self, db_pool, ai_client, spam_checker):
        self.db_pool = db_pool
        self.ai_client = ai_client
        self.spam_checker = spam_checker

    async def screen_call(self, caller_number: str, call_sid: str):
        """Full screening pipeline for an incoming call."""
        # Phase 1: Identification
        caller_info = await self.identify_caller(caller_number)

        # Phase 2: Spam check (fast path for known spam)
        if await self.spam_checker.is_spam(caller_number):
            return ScreeningResult(
                caller_number=caller_number,
                caller_name=None,
                priority=CallerPriority.SPAM,
                intent_summary="Spam call detected",
                routing_action=RoutingAction.BLOCK,
                confidence=0.95,
            )

        # Phase 3: For non-spam, engage in brief conversation
        intent = await self.assess_intent(call_sid)

        # Phase 4: Make routing decision
        return await self.decide_routing(caller_info, intent)

Caller Identification

Before the AI even speaks, look up the caller against your known contacts, CRM, and spam databases:

import asyncpg

class CallerIdentifier:
    """Identifies callers from phone number lookup."""

    def __init__(self, db_pool: asyncpg.Pool):
        self.db_pool = db_pool

    async def identify(self, phone_number: str) -> dict:
        """Look up caller in CRM and contact databases."""
        # Check internal contacts first
        contact = await self.db_pool.fetchrow(
            """
            SELECT name, company, relationship, priority_level,
                   last_call_date, total_calls, notes
            FROM contacts
            WHERE phone_number = $1 AND is_active = true
            """,
            phone_number,
        )

        if contact:
            return {
                "known": True,
                "name": contact["name"],
                "company": contact["company"],
                "priority": contact["priority_level"],
                "relationship": contact["relationship"],
                "call_history": {
                    "last_call": contact["last_call_date"],
                    "total_calls": contact["total_calls"],
                },
                "notes": contact["notes"],
            }

        # Check recent call history for repeat unknown callers
        recent_calls = await self.db_pool.fetchval(
            """
            SELECT COUNT(*) FROM call_log
            WHERE caller_number = $1
            AND created_at > NOW() - INTERVAL '30 days'
            """,
            phone_number,
        )

        return {
            "known": False,
            "name": None,
            "recent_call_count": recent_calls,
        }

Spam Detection Pipeline

Layer multiple spam signals for accurate detection:

class SpamDetector:
    """Multi-signal spam detection for incoming calls."""

    def __init__(self, db_pool, external_api_key=None):
        self.db_pool = db_pool
        self.external_api_key = external_api_key

    async def is_spam(self, phone_number: str) -> bool:
        """Check multiple spam signals."""
        score = 0.0

        # Check internal blocklist
        blocked = await self.db_pool.fetchval(
            "SELECT EXISTS(SELECT 1 FROM blocklist WHERE phone = $1)",
            phone_number,
        )
        if blocked:
            return True

        # Check call frequency (high frequency = suspicious)
        calls_today = await self.db_pool.fetchval(
            """
            SELECT COUNT(*) FROM call_log
            WHERE caller_number = $1
            AND created_at > NOW() - INTERVAL '1 hour'
            """,
            phone_number,
        )
        if calls_today > 5:
            score += 0.4

        # Check against known spam patterns
        if self.is_spoofed_pattern(phone_number):
            score += 0.3

        # External spam database lookup
        if self.external_api_key:
            external_score = await self.check_external_db(phone_number)
            score += external_score * 0.3

        return score >= 0.6

    def is_spoofed_pattern(self, number: str) -> bool:
        """Detect common spoofing patterns."""
        # Numbers with all same digits, sequential patterns
        digits = number.replace("+", "").replace("-", "")
        if len(set(digits[-4:])) == 1:  # Last 4 digits identical
            return True
        return False

Intent Assessment via Conversation

For callers who pass the spam check, the AI engages in a brief screening conversation:

from openai import AsyncOpenAI

class IntentAssessor:
    """Assesses caller intent through brief conversation."""

    def __init__(self):
        self.client = AsyncOpenAI()

    async def assess(self, transcript: str, caller_info: dict) -> dict:
        context = ""
        if caller_info.get("known"):
            context = (
                f"Known caller: {caller_info['name']} from "
                f"{caller_info.get('company', 'N/A')}. "
                f"Relationship: {caller_info.get('relationship', 'unknown')}."
            )

        response = await self.client.chat.completions.create(
            model="gpt-4o-mini",
            messages=[
                {
                    "role": "system",
                    "content": (
                        "You are a call screening assistant. Assess the "
                        "caller's intent and urgency from their statement. "
                        f"Context: {context}\n"
                        "Return JSON with: intent, urgency (low/medium/high/"
                        "emergency), summary, and recommended_action "
                        "(connect/message/voicemail/block)."
                    ),
                },
                {"role": "user", "content": transcript},
            ],
            response_format={"type": "json_object"},
            temperature=0.1,
        )

        import json
        return json.loads(response.choices[0].message.content)

Priority Routing Logic

Combine identification and intent to make the final routing decision:

async def decide_routing(
    self, caller_info: dict, intent: dict
) -> ScreeningResult:
    """Determine routing based on caller identity and intent."""
    # VIP callers always connect immediately
    if caller_info.get("priority") == "vip":
        return ScreeningResult(
            caller_number=caller_info.get("phone", ""),
            caller_name=caller_info.get("name"),
            priority=CallerPriority.VIP,
            intent_summary=intent.get("summary", ""),
            routing_action=RoutingAction.CONNECT_IMMEDIATELY,
            confidence=1.0,
        )

    urgency = intent.get("urgency", "low")

    # Emergency calls always connect
    if urgency == "emergency":
        return ScreeningResult(
            caller_number=caller_info.get("phone", ""),
            caller_name=caller_info.get("name"),
            priority=CallerPriority.NEW,
            intent_summary=intent["summary"],
            routing_action=RoutingAction.CONNECT_IMMEDIATELY,
            confidence=0.9,
        )

    # Known callers with medium+ urgency connect with announcement
    if caller_info.get("known") and urgency in ("medium", "high"):
        return ScreeningResult(
            caller_number=caller_info.get("phone", ""),
            caller_name=caller_info["name"],
            priority=CallerPriority.KNOWN,
            intent_summary=intent["summary"],
            routing_action=RoutingAction.CONNECT_WITH_ANNOUNCE,
            confidence=0.85,
        )

    # Low urgency or unknown callers take a message
    return ScreeningResult(
        caller_number=caller_info.get("phone", ""),
        caller_name=caller_info.get("name"),
        priority=CallerPriority.NEW,
        intent_summary=intent["summary"],
        routing_action=RoutingAction.TAKE_MESSAGE,
        confidence=0.7,
    )

FAQ

How do I avoid blocking legitimate calls?

Use a multi-signal approach and set conservative thresholds. Never block based on a single signal unless it is an explicit blocklist entry. Implement a "soft block" tier that sends callers to voicemail instead of disconnecting — they can still leave a message. Review blocked calls weekly and adjust thresholds. Allow users to whitelist numbers that were incorrectly flagged.

What is the typical screening conversation duration?

An effective screening interaction should complete in 10-15 seconds. The AI greets the caller, asks how it can help, and captures their initial statement. One exchange is usually enough to determine intent and urgency. Longer screening creates friction — if you cannot resolve the screening in two exchanges, route to a human.

How do I handle the "agent whisper" when connecting a screened call?

When routing a screened call to a human, use a conference bridge pattern. Connect the human first and play a whisper message (e.g., "Incoming call from John Smith regarding a billing dispute, high urgency") that only the human hears. Then bridge in the caller. Twilio supports this with the <Dial> verb's callerId attribute and <Conference> with coach mode.

---

#CallScreening #SpamDetection #CallRouting #VoiceAI #PriorityRouting #Telephony #AgenticAI #LearnAI #AIEngineering