Office Hours — How do I prevent an AI agent from hallucinating tool calls and wasting tokens on actions that don't exist?

How do I prevent an AI agent from hallucinating tool calls and wasting tokens on actions that don’t exist?

The Real Problem

An AI agent decides to call validate_payment_with_stripe_webhook(). Your system doesn’t have that tool. The model confidently invents parameters. Your agent logs an error, retries, burns tokens on the failure case, and now you’re paying for a ghost function call that never should have happened.

This isn’t a hallucination in the traditional sense—the model isn’t making up facts about the world. It’s worse. It’s confident inventing tool calls that don’t match your actual API surface. And because agents operate in loops, one bad tool invocation can cascade into five retries before you notice.

The problem compounds in production: frontier models like Claude Opus 5.5 or GPT-6 Astra are so good at plausible tool use that they’ll invent convincing function signatures for tools that don’t exist. They’re not being reckless—they’re generalizing from patterns in their training data and assuming your agent environment is similarly flexible.

What Actually Works

The most effective mitigation isn’t a prompt tweak. It’s structural: make tool hallucinations impossible or immediately detectable before they waste tokens.

1. Strict tool schema validation at runtime. Before the agent’s tool call reaches your backend, parse it against a JSON schema that exactly matches your available tools. Claude and GPT both support structured outputs now, so you can enforce schema compliance before the model even tries to call something.

from pydantic import BaseModel, ValidationError
import anthropic

class PaymentToolCall(BaseModel):
    tool_name: str
    amount_cents: int
    customer_id: str

client = anthropic.Anthropic()

# Define what tools actually exist
AVAILABLE_TOOLS = {
    "process_payment": PaymentToolCall,
    "refund_payment": PaymentToolCall,
}

response = client.messages.create(
    model="claude-opus-5-5",
    max_tokens=1024,
    tools=[
        {
            "name": "process_payment",
            "description": "Process a payment via Stripe",
            "input_schema": {
                "type": "object",
                "properties": {
                    "amount_cents": {"type": "integer"},
                    "customer_id": {"type": "string"},
                },
                "required": ["amount_cents", "customer_id"],
            },
        },
        {
            "name": "refund_payment",
            "description": "Issue a refund",
            "input_schema": {
                "type": "object",
                "properties": {
                    "amount_cents": {"type": "integer"},
                    "customer_id": {"type": "string"},
                },
                "required": ["amount_cents", "customer_id"],
            },
        },
    ],
    messages=[{"role": "user", "content": "Refund $50 for customer 12345"}],
)

# When the model tries to call a tool, validate it first
for block in response.content:
    if block.type == "tool_use":
        if block.name not in AVAILABLE_TOOLS:
            # Tool doesn't exist. Reject it immediately.
            print(f"Tool {block.name} not in AVAILABLE_TOOLS. Halting.")
            continue
        
        try:
            validated = AVAILABLE_TOOLS[block.name](**block.input)
            # Now it's safe to execute
            print(f"Valid call: {validated}")
        except ValidationError as e:
            print(f"Invalid tool input: {e}")
            # Return error to agent instead of executing

2. Constrain the tool list itself. This is the underrated move: don’t give the agent a kitchen sink of 47 tools. Give it 3-5 that directly solve the current task. Every tool in the list increases the probability the model invents variants or misuses them.

A banking agent doesn’t need access to inventory management, email, and customer support simultaneously. If it does, you’re overloading it. Split it into smaller agents with tighter scope.

3. Use tool calling feedback loops. When the model attempts an invalid tool call, don’t just fail silently. Send the error back in the conversation and give it one chance to correct course.

# After validation fails
if block.name not in AVAILABLE_TOOLS:
    error_msg = f"Tool '{block.name}' does not exist. Available tools: {', '.join(AVAILABLE_TOOLS.keys())}"
    messages.append({"role": "assistant", "content": response.content})
    messages.append({"role": "user", "content": f"Error: {error_msg}. Please use an available tool."})
    
    # Re-query with the same model to self-correct
    retry = client.messages.create(
        model="claude-opus-5-5",
        max_tokens=1024,
        tools=[...],
        messages=messages,
    )
    # Process retry response

This is cheaper than blindly retrying the entire task and often works—the model catches its own error when explicitly told.

4. Log and alert on tool hallucinations. Track how often the agent attempts tools that don’t exist or pass invalid parameters. If that rate climbs above 5-10%, something is wrong with either your tool definitions (they’re ambiguous) or the model (it’s degraded or you’re using the wrong one for the task).

What Doesn’t Work

Don’t rely on prompt warnings like “only use these tools” or “if a tool doesn’t exist, say so.” The model hears this, agrees with you, and then invokes get_user_balance_safely() anyway because it seems reasonable in context.

Don’t assume structured outputs alone will save you. They help enforce schema compliance but won’t catch semantic errors—the model can still call refund_payment when it should call process_payment, and both have the same schema.

The Token Cost Reality

An invalid tool call burns tokens three ways: the initial invocation, the error response from your system, and the retry. On Claude Opus 5.5 cached inputs at $0.20/M, each bad tool cycle costs roughly $0.001-0.003. In a 10,000-task pipeline, that’s $10-30 of waste per 1% hallucination rate. Tighter tool validation cuts that to near zero.

Bottom line: Prevent tool hallucinations by validating against a strict schema before execution, constraining the tool list to the immediate task, and giving the model one corrective feedback loop if it fails. Log everything and watch your hallucination rate as an operational metric—if it drifts up, your tool definitions are too ambiguous or your model choice is wrong.