--- title: "On-Call for AI Agent Systems: Alert Routing, Escalation, and Response Procedures" description: "Design effective on-call systems for AI agents with PagerDuty setup, rotation design, escalation policies, alert routing, and post-incident review processes tailored to the unique demands of autonomous agent systems." canonical: https://callsphere.ai/blog/on-call-ai-agent-systems-alert-routing-escalation-response category: "Learn Agentic AI" tags: ["On-Call", "AI Agents", "Alerting", "PagerDuty", "Incident Response"] author: "CallSphere Team" published: 2026-03-17T00:00:00.000Z updated: 2026-05-29T01:58:43.240Z --- # On-Call for AI Agent Systems: Alert Routing, Escalation, and Response Procedures > Design effective on-call systems for AI agents with PagerDuty setup, rotation design, escalation policies, alert routing, and post-incident review processes tailored to the unique demands of autonomous agent systems. ## On-Call Challenges Unique to AI Agents Traditional on-call rotations handle server outages, database failures, and deployment rollbacks. AI agent systems add a new class of issues: behavioral problems. The agent is technically running, latency is normal, no errors in the logs — but it is giving users wrong answers, calling tools with fabricated parameters, or responding in an inappropriate tone. These behavioral alerts require on-call engineers who understand not just infrastructure, but also prompt engineering, model behavior, and the agent's domain context. ## Designing Alert Routing for Agents Route alerts to the right team based on the failure type, not just severity. ```mermaid flowchart LR INC(["Production incident"]) DETECT["Detect alerts plus user reports"] MIT["Mitigate rollback or feature flag"] RES["Resolve"] DOC["Timeline doc events plus actions"] RCA{"5 whys plus causal graph"} AI["Action items owner plus due date"] SHARE(["Blameless review"]) LEARN[("Runbook plus eval added")] INC --> DETECT --> MIT --> RES --> DOC --> RCA --> AI --> SHARE --> LEARN style RCA fill:#4f46e5,stroke:#4338ca,color:#fff style LEARN fill:#059669,stroke:#047857,color:#fff ``` ```python from dataclasses import dataclass from enum import Enum from typing import List class AlertCategory(Enum): INFRASTRUCTURE = "infrastructure" # pods, networking, database LLM_PROVIDER = "llm_provider" # API errors, rate limits, latency AGENT_BEHAVIOR = "agent_behavior" # wrong answers, safety issues BUSINESS_LOGIC = "business_logic" # tool failures, workflow errors @dataclass class AlertRoute: category: AlertCategory severity: str pagerduty_service: str escalation_policy: str notification_channels: List[str] ALERT_ROUTES = [ AlertRoute( category=AlertCategory.INFRASTRUCTURE, severity="critical", pagerduty_service="ai-platform-infra", escalation_policy="infra-escalation", notification_channels=["#agent-ops", "#infra-alerts"], ), AlertRoute( category=AlertCategory.AGENT_BEHAVIOR, severity="critical", pagerduty_service="ai-agent-safety", escalation_policy="safety-escalation", notification_channels=["#agent-safety", "#agent-ops"], ), AlertRoute( category=AlertCategory.LLM_PROVIDER, severity="warning", pagerduty_service="ai-platform-infra", escalation_policy="provider-escalation", notification_channels=["#agent-ops"], ), AlertRoute( category=AlertCategory.BUSINESS_LOGIC, severity="warning", pagerduty_service="ai-agent-product", escalation_policy="product-escalation", notification_channels=["#agent-product"], ), ] class AlertRouter: def __init__(self, routes: List[AlertRoute], pagerduty_client): self.routes = {(r.category, r.severity): r for r in routes} self.pd = pagerduty_client async def route_alert(self, category: AlertCategory, severity: str, title: str, details: dict): route = self.routes.get((category, severity)) if not route: # Default: page infra team for unknown alerts route = self.routes[(AlertCategory.INFRASTRUCTURE, "critical")] await self.pd.create_incident( service=route.pagerduty_service, escalation_policy=route.escalation_policy, title=title, severity=severity, details=details, ) for channel in route.notification_channels: await self.notify_channel(channel, title, severity) ``` The key insight is separating infrastructure alerts from behavioral alerts. An infra engineer can restart pods, but investigating why the agent recommended a dangerous medication dosage requires someone who understands the agent's guardrails and prompt architecture. ## Rotation Design ```yaml # on-call-rotation.yaml rotations: - name: "agent-infra-primary" type: weekly handoff_day: monday handoff_time: "09:00" timezone: "America/New_York" members: - "engineer-a" - "engineer-b" - "engineer-c" - "engineer-d" restrictions: max_consecutive_weeks: 2 min_gap_between_shifts: 2 # weeks - name: "agent-behavior-primary" type: weekly handoff_day: monday handoff_time: "09:00" timezone: "America/New_York" members: - "ai-engineer-a" - "ai-engineer-b" - "ai-engineer-c" restrictions: max_consecutive_weeks: 1 min_gap_between_shifts: 3 escalation_policies: infra-escalation: - level: 1 target: "agent-infra-primary" timeout_minutes: 10 - level: 2 target: "infra-team-lead" timeout_minutes: 15 - level: 3 target: "vp-engineering" timeout_minutes: 30 safety-escalation: - level: 1 target: "agent-behavior-primary" timeout_minutes: 5 - level: 2 target: "ai-safety-lead" timeout_minutes: 10 - level: 3 target: "cto" timeout_minutes: 15 ``` Notice the safety escalation has shorter timeouts at every level. A safety issue that is not acknowledged within 5 minutes automatically escalates to the AI safety lead. ## Alert Quality Management Alert fatigue is the number one cause of missed critical incidents. Manage alert quality aggressively. ```python from datetime import datetime, timedelta from collections import defaultdict class AlertQualityTracker: def __init__(self): self.alerts = [] def record_alert(self, alert_name: str, was_actionable: bool, time_to_acknowledge: float, time_to_resolve: float): self.alerts.append({ "name": alert_name, "timestamp": datetime.utcnow(), "actionable": was_actionable, "tta_minutes": time_to_acknowledge, "ttr_minutes": time_to_resolve, }) def weekly_report(self) -> dict: week_ago = datetime.utcnow() - timedelta(days=7) recent = [a for a in self.alerts if a["timestamp"] > week_ago] if not recent: return {"total_alerts": 0} by_name = defaultdict(list) for a in recent: by_name[a["name"]].append(a) actionable_rate = sum(1 for a in recent if a["actionable"]) / len(recent) noisy_alerts = [ name for name, alerts in by_name.items() if len(alerts) > 10 and sum(1 for a in alerts if a["actionable"]) / len(alerts) = 0.85 else "REVIEW needed" ), } ``` If fewer than 70% of your alerts are actionable, engineers will start ignoring pages. Review and tune or remove noisy alerts weekly. ## Post-Incident Review Integration Every page should feed back into the system improvement cycle. ```python class OnCallHandoffReport: def generate(self, shift_start: datetime, shift_end: datetime, incidents: list, alerts: list) -> dict: return { "shift_period": f"{shift_start.isoformat()} to {shift_end.isoformat()}", "total_pages": len(alerts), "incidents_opened": len([i for i in incidents if i["opened_during_shift"]]), "incidents_resolved": len([i for i in incidents if i["resolved_during_shift"]]), "sleep_interruptions": len([ a for a in alerts if a["timestamp"].hour >= 22 or a["timestamp"].hour <= 6 ]), "action_items": [ i.get("follow_up") for i in incidents if i.get("follow_up") ], "alerts_to_tune": [ a["name"] for a in alerts if not a.get("actionable", True) ], } ``` ## FAQ ### Should AI engineers or infrastructure engineers be on-call for agent systems? Both, with separate rotations. Infrastructure engineers handle pod failures, database issues, and networking problems. AI engineers handle behavioral issues — hallucinations, safety violations, and prompt regressions. Route alerts to the right rotation based on the alert category, not a single combined rotation. ### How do I reduce alert fatigue for AI agent systems? Track your actionable alert rate and target above 85%. Remove alerts that fire frequently but never require action. Consolidate related alerts into a single notification with context. Use alert grouping to batch multiple instances of the same issue. Review the noisiest alerts weekly and either tune thresholds, add suppression rules, or delete them. ### What should an on-call handoff include for AI agent systems? Include: active incidents and their status, alerts that fired and whether they were actionable, any ongoing behavioral issues being monitored, recent deployments that might cause problems, and LLM provider status. The handoff should take less than 15 minutes. Write it as a structured document, not a verbal conversation. --- #OnCall #AIAgents #Alerting #PagerDuty #IncidentResponse #AgenticAI #LearnAI #AIEngineering --- Source: https://callsphere.ai/blog/on-call-ai-agent-systems-alert-routing-escalation-response