Debugging Claude Computer Use: Loops, Bad Tool Calls

The first time you watch a Claude computer-use agent fail, it rarely crashes. It does something far more unsettling: it takes a screenshot, moves the mouse three pixels, takes another screenshot, moves the mouse three pixels again, and keeps doing that until it burns your entire token budget. Nothing errored. The model was confident the whole time. That gap — between a confident agent and a correct one — is where most computer-use debugging actually lives.

Computer use lets Claude operate a real desktop the way a person does: it receives a screenshot, reasons about what it sees, and emits actions like click, type, key, and scroll through the computer-use tool. That loop is powerful and also fragile, because every step depends on the model correctly perceiving a pixel image and correctly grounding an action in it. When something goes wrong, the failure is almost never a stack trace. It is a behavioral pattern you have to learn to recognize.

Key takeaways

• Loops are the dominant failure mode; they come from no progress signal, stale screenshots, or an ambiguous goal — break them with step caps and state hashing.
• Wrong tool calls usually trace back to a vague tool description or missing affordances, not model stupidity.
• Hallucinated arguments (invented coordinates, fake filenames) appear when the screenshot is low-resolution or the element is off-screen.
• Log every screenshot, every action, and the model's stated reasoning so you can replay a run deterministically.
• A small set of guardrails — max steps, action diff checks, and a verification sub-step — eliminates most production incidents.

Why computer-use agents loop

A loop is the canonical computer-use bug. The model believes it is making progress, but the environment is not changing in the way it expects. The most common cause is the absence of an explicit progress signal. The agent sees a screenshot, decides to click a button, and then sees a screenshot that looks almost identical because the click missed or the page was still loading. With no memory of "I already tried this exact action and it did nothing," Claude rationally tries again.

The second common cause is a goal that is satisfiable in the model's head but not observable on screen. If you ask the agent to "make sure the form is submitted" but the success state is a toast notification that disappears after two seconds, the agent will keep re-submitting because it never captures evidence the task is done. The fix is to define done-ness in terms of something durably visible: a URL change, a confirmation page, a row that appears in a table.

The third cause is timing. Desktops are asynchronous. A screenshot taken 200ms after a click captures a half-rendered state, and the model reasons over a frame that no longer reflects reality. Adding a short, deliberate settle delay before each screenshot removes a surprising fraction of loops.

A debugging decision flow

flowchart TD
  A["Agent stuck or slow"] --> B{"Screen changing between steps?"}
  B -->|No| C["Loop: add state hash & step cap"]
  B -->|Yes| D{"Right tool being called?"}
  D -->|No| E["Tighten tool description & examples"]
  D -->|Yes| F{"Args grounded in screenshot?"}
  F -->|No| G["Hallucinated coords: raise resolution, scroll into view"]
  F -->|Yes| H["Add explicit verify step before declaring done"]

This flow is the order I actually debug in. Start by asking whether the screen is changing at all between steps, because that single question splits loops from grounding errors. Only once you know the environment is responding do you move on to whether the right tool and the right arguments are being chosen.

Diagnosing wrong tool calls

When Claude reaches for the wrong tool — calling a generic bash action when it should have clicked, or typing into the wrong field — the instinct is to blame the model. Far more often the tool definitions are underspecified. Computer use ships with a small set of primitive actions, but most real agents add custom tools alongside them. If two tools have overlapping descriptions, the model has to guess, and guessing is where errors enter.

The most effective remedy is description discipline. Each tool's description should say exactly when to use it and, critically, when not to. Add a one-line negative constraint: "Use this only after the document is open; do not use it to open documents." Pair the description with a worked example in the system prompt showing the tool used correctly in context. Claude grounds new behavior in examples extremely well.

Logging is your other lever. Capture the full assistant turn — the reasoning text, the tool name, and the arguments — for every step. When a wrong tool fires, you can read the reasoning and usually see the exact misunderstanding in plain English, which tells you which sentence in your tool description to rewrite.

Hallucinated arguments and coordinate grounding

Hallucinated arguments are the most computer-use-specific failure. Claude emits a click at coordinates that point at empty space, or types a filename it never saw. This is fundamentally a perception problem. The model is grounding an action in a screenshot, and if that screenshot is downscaled, blurry, or missing the target because it is below the fold, the model fills the gap with a plausible guess.

The single highest-leverage fix is screenshot resolution and scaling. Send images at a resolution where text and buttons are legible, and make sure your coordinate space matches the image you actually sent — a mismatch between the displayed resolution and the coordinate system the model assumes produces clicks that are consistently offset. Before clicking an element that might be off-screen, instruct the agent to scroll it into view and re-screenshot, so the target is genuinely visible when coordinates are chosen.

def settle_and_capture(env, settle_ms=400):
    time.sleep(settle_ms / 1000)
    img = env.screenshot()
    h = hashlib.sha256(img.tobytes()).hexdigest()
    return img, h

prev = None
for step in range(MAX_STEPS):
    img, h = settle_and_capture(env)
    if h == prev:
        # screen identical to last turn: likely a loop
        inject_hint("The screen did not change. Your last action had no effect; try a different approach.")
    prev = h
    action = claude_step(img)
    env.apply(action)

That harness does three things at once: it settles before capturing, it hashes the screen to detect no-change loops, and it injects a corrective hint when it sees one. Telling the model "your last action had no effect" is remarkably good at jolting it out of a loop, because it supplies the progress signal the environment failed to.

Common pitfalls

• No step cap. Always set a hard maximum on actions per task. An uncapped agent in a loop is an uncapped bill.
• Trusting self-reported success. The model saying "I have completed the task" is a claim, not evidence. Verify against an observable end state before exiting.
• Downscaled screenshots. If buttons are illegible to you in the logged image, they are illegible to Claude. Coordinate hallucinations follow.
• Capturing before render. Screenshotting immediately after an action gives the model a stale frame. Add a settle delay.
• Throwing away reasoning. If you only log actions, you lose the one artifact that explains why a wrong tool fired.

Debug a stuck agent in 6 steps

1. Pull the full run trace: screenshots, actions, and the model's reasoning text for every step.
2. Diff consecutive screenshots; if they are identical, you have a loop, not a grounding bug.
3. For loops, add a screen-state hash and inject a "no change" hint, then cap steps.
4. For wrong tools, read the reasoning, find the ambiguous tool description, and add a negative constraint plus one example.
5. For bad coordinates, raise screenshot resolution and confirm your coordinate space matches the sent image.
6. Add an explicit verification step that checks an observable end state before the agent declares success.

Failure modes at a glance

Frequently asked questions

Why does my Claude computer-use agent keep taking screenshots without acting?

This is the classic loop. The model is waiting for the screen to reach a state that never arrives, often because a click missed or the page is still loading. Add a settle delay before each capture, hash the screen to detect no-change turns, and inject a hint telling the model its last action had no effect.

How do I stop Claude from clicking the wrong coordinates?

Coordinate errors are a perception problem. Send higher-resolution screenshots, ensure the coordinate space you expect matches the image you actually sent, and have the agent scroll a target into view and re-screenshot before clicking anything that might be below the fold.

Is a wrong tool call the model's fault or mine?

Usually yours. Vague or overlapping tool descriptions force the model to guess. Add an explicit "use this only when" and "do not use this for" line to each description, and include one worked example. Then read the model's logged reasoning to confirm the misunderstanding is gone.

What is the single most useful thing to log?

The model's reasoning text alongside each action and screenshot. Actions tell you what happened; reasoning tells you why, and that is what lets you fix the prompt or tool definition instead of guessing.

From flaky agents to dependable ones

CallSphere brings the same debugging discipline — progress signals, verification before declaring done, and full run traces — to voice and chat agents that answer every call and message, use tools mid-conversation, and book work around the clock. See how dependable agentic automation looks in production at callsphere.ai.

Source & attribution: This is an independent, original explainer inspired by Anthropic's coverage on the Claude blog. Claude, Claude Code, Claude Cowork, Claude Opus, and the Model Context Protocol are products and trademarks of Anthropic. CallSphere is not affiliated with or endorsed by Anthropic.