Production AI Incident Response: Debugging Rogue Agents

When AI Agents Go Wrong in Production

Unlike a traditional API that returns a bad response, a misbehaving AI agent can take multiple actions before anyone notices something is wrong. It can send emails, modify databases, call external services, and generate content that reaches end users, all within seconds and all based on a single misinterpreted instruction.

Production AI incidents fall into categories that require different response strategies. Understanding these categories before an incident occurs is the difference between a 5-minute fix and a 5-hour fire drill.

Incident Classification for AI Agents

Category 1: Output Quality Degradation

The agent is functional but producing lower-quality outputs. Common causes include prompt drift (system prompts modified without testing), model version changes, or degraded retrieval quality.

flowchart LR
    APP(["Agent or API"])
    SDK["OTel SDK<br/>GenAI conventions"]
    COL["OTel Collector"]
    subgraph BACKENDS["Backends"]
        TR[("Traces<br/>Tempo or Honeycomb")]
        MET[("Metrics<br/>Prometheus")]
        LOG[("Logs<br/>Loki or ELK")]
    end
    DASH["Grafana plus alerts"]
    PAGE(["Pager"])
    APP --> SDK --> COL
    COL --> TR
    COL --> MET
    COL --> LOG
    TR --> DASH
    MET --> DASH
    LOG --> DASH
    DASH --> PAGE
    style SDK fill:#4f46e5,stroke:#4338ca,color:#fff
    style DASH fill:#f59e0b,stroke:#d97706,color:#1f2937
    style PAGE fill:#dc2626,stroke:#b91c1c,color:#fff

Symptoms:

• Increased user complaint rate
• Lower automated quality scores
• Higher escalation rates to human support
• Response times remain normal

Typical root cause: A dependency changed (model version, retrieval index, system prompt) and quality testing did not catch the regression.

Category 2: Behavioral Deviation

The agent is doing things it should not be doing, calling tools it should not call, or ignoring constraints.

Symptoms:

• Agent calling tools outside its allowed set
• Ignoring safety guardrails or content policies
• Taking actions without required confirmation steps
• Processing requests it should decline

Typical root cause: Prompt injection (malicious or accidental), system prompt gap, or tool definition that is too permissive.

Category 3: Infinite Loops and Resource Exhaustion

The agent gets stuck in a loop, repeatedly calling the same tool or generating endless responses.

Symptoms:

• Abnormally high API costs over a short period
• Individual requests consuming 10-100x normal token usage
• Timeouts and cascading failures downstream
• Rapid rate limit exhaustion

Typical root cause: Missing loop guards, ambiguous tool results that the agent keeps retrying, or circular tool dependencies.

Category 4: Data Integrity Violations

The agent writes incorrect data to databases, sends wrong information to users, or corrupts state.

Symptoms:

• Database inconsistencies detected by integrity checks
• User reports of incorrect information
• Downstream systems receiving malformed data

Typical root cause: Hallucinated data passed to write tools, race conditions in concurrent agent executions, or insufficient validation in tool implementations.

The Kill Switch Pattern

Every production AI agent must have an immediate shutdown mechanism that does not require a code deployment.

import redis
from functools import wraps

redis_client = redis.Redis(host="localhost", port=6379, db=0)

KILL_SWITCH_KEY = "agent:kill_switch:{agent_id}"
RATE_LIMIT_KEY = "agent:rate_limit:{agent_id}"

def check_kill_switch(agent_id: str):
    """Check if the agent has been manually killed."""
    if redis_client.get(KILL_SWITCH_KEY.format(agent_id=agent_id)):
        raise AgentKilledException(
            f"Agent {agent_id} has been manually stopped. "
            f"Check incident channel for details."
        )

def kill_agent(agent_id: str, reason: str, killed_by: str):
    """Immediately stop an agent from processing new requests."""
    redis_client.set(
        KILL_SWITCH_KEY.format(agent_id=agent_id),
        json.dumps({
            "reason": reason,
            "killed_by": killed_by,
            "timestamp": datetime.utcnow().isoformat()
        })
    )
    # Alert the team
    send_alert(
        severity="critical",
        message=f"Agent {agent_id} killed by {killed_by}: {reason}"
    )

def with_kill_switch(agent_id: str):
    """Decorator to check kill switch before each agent step."""
    def decorator(func):
        @wraps(func)
        async def wrapper(*args, **kwargs):
            check_kill_switch(agent_id)
            return await func(*args, **kwargs)
        return wrapper
    return decorator

Applying the Kill Switch in the Agent Loop

@with_kill_switch(agent_id="customer-service-v2")
async def agent_step(messages: list, tools: list) -> dict:
    """Single step of the agent loop with kill switch protection."""
    response = await async_client.messages.create(
        model="claude-sonnet-4-20250514",
        max_tokens=4096,
        tools=tools,
        messages=messages
    )

    # Also check after each tool execution
    for block in response.content:
        if block.type == "tool_use":
            check_kill_switch("customer-service-v2")
            result = await execute_tool(block.name, block.input)

    return response

Logging for Debuggability

Standard application logging is insufficient for AI agents. You need structured logs that capture the full reasoning chain.

import structlog
from uuid import uuid4

logger = structlog.get_logger()

class AgentTracer:
    """Structured tracing for AI agent execution."""

    def __init__(self, agent_id: str, session_id: str):
        self.agent_id = agent_id
        self.session_id = session_id
        self.trace_id = str(uuid4())
        self.step_count = 0

    def log_step(self, step_type: str, **kwargs):
        self.step_count += 1
        logger.info(
            "agent_step",
            agent_id=self.agent_id,
            session_id=self.session_id,
            trace_id=self.trace_id,
            step_number=self.step_count,
            step_type=step_type,
            **kwargs
        )

    def log_api_call(self, model: str, input_tokens: int,
                     output_tokens: int, stop_reason: str):
        self.log_step(
            "api_call",
            model=model,
            input_tokens=input_tokens,
            output_tokens=output_tokens,
            stop_reason=stop_reason
        )

    def log_tool_call(self, tool_name: str, tool_input: dict,
                      tool_output: str, duration_ms: float):
        self.log_step(
            "tool_call",
            tool_name=tool_name,
            tool_input=self._redact_sensitive(tool_input),
            tool_output_length=len(tool_output),
            duration_ms=duration_ms
        )

    def log_decision(self, decision: str, reasoning: str):
        self.log_step(
            "decision",
            decision=decision,
            reasoning=reasoning
        )

    def _redact_sensitive(self, data: dict) -> dict:
        """Redact PII and sensitive fields from logs."""
        sensitive_keys = {"password", "ssn", "credit_card", "api_key", "token"}
        return {
            k: "[REDACTED]" if k.lower() in sensitive_keys else v
            for k, v in data.items()
        }

Loop Guards: Preventing Runaway Agents

Every agent loop needs hard limits that prevent runaway execution.

class AgentLoopGuard:
    """Prevent runaway agent execution."""

    def __init__(
        self,
        max_steps: int = 25,
        max_tokens: int = 200_000,
        max_duration_seconds: int = 300,
        max_tool_calls: int = 50,
        max_consecutive_same_tool: int = 3
    ):
        self.max_steps = max_steps
        self.max_tokens = max_tokens
        self.max_duration_seconds = max_duration_seconds
        self.max_tool_calls = max_tool_calls
        self.max_consecutive_same_tool = max_consecutive_same_tool

        self.step_count = 0
        self.total_tokens = 0
        self.tool_call_count = 0
        self.start_time = time.time()
        self.recent_tools: list[str] = []

    def check(self, tokens_used: int = 0, tool_name: str | None = None):
        self.step_count += 1
        self.total_tokens += tokens_used

        if tool_name:
            self.tool_call_count += 1
            self.recent_tools.append(tool_name)

        elapsed = time.time() - self.start_time

        if self.step_count > self.max_steps:
            raise LoopGuardError(f"Exceeded max steps: {self.max_steps}")

        if self.total_tokens > self.max_tokens:
            raise LoopGuardError(f"Exceeded max tokens: {self.max_tokens}")

        if elapsed > self.max_duration_seconds:
            raise LoopGuardError(f"Exceeded max duration: {self.max_duration_seconds}s")

        if self.tool_call_count > self.max_tool_calls:
            raise LoopGuardError(f"Exceeded max tool calls: {self.max_tool_calls}")

        # Detect repeated tool calls (possible loop)
        if len(self.recent_tools) >= self.max_consecutive_same_tool:
            last_n = self.recent_tools[-self.max_consecutive_same_tool:]
            if len(set(last_n)) == 1:
                raise LoopGuardError(
                    f"Detected loop: {last_n[0]} called "
                    f"{self.max_consecutive_same_tool} times consecutively"
                )

Post-Incident Review Process

After resolving an AI agent incident, conduct a structured review that covers AI-specific factors.

Standard post-mortem questions plus AI-specific additions:

1. What changed? Model version, system prompt, tool definitions, retrieval index, training data?
2. What was the agent's reasoning? Review the full trace from structured logs.
3. Was this a known failure mode? Check against your agent's evaluation suite.
4. Would the evaluation suite have caught this? If not, add a test case.
5. Are the guardrails sufficient? Did the kill switch, loop guards, and validation layers work?
6. What is the blast radius? How many users were affected? What data was impacted?

Turning Incidents into Evaluation Cases

Every incident should generate at least one automated test case for your agent evaluation suite.

def incident_to_eval_case(incident: dict) -> dict:
    """Convert a production incident into a regression test."""
    return {
        "test_id": f"incident-{incident['id']}",
        "input": incident["triggering_input"],
        "expected_behavior": incident["correct_behavior"],
        "forbidden_actions": incident["actions_taken_incorrectly"],
        "category": incident["category"],
        "severity": incident["severity"],
        "date_added": datetime.utcnow().isoformat(),
        "source": f"Incident #{incident['id']}"
    }

Summary

Production AI incidents are fundamentally different from traditional software incidents because agents can take multiple autonomous actions before detection. The defense-in-depth strategy includes kill switches for immediate shutdown, loop guards to prevent runaway execution, structured tracing for full-chain debuggability, and a post-incident process that converts every failure into an automated regression test. Building these systems before your first incident is dramatically cheaper than building them during one.