Office Hours — What's your strategy for evaluating whether an AI agent is actually using its tools and skills effectively?
A daily developer question about AI/LLMs, answered with a direct, opinionated take.
What’s your strategy for evaluating whether an AI agent is actually using its tools and skills effectively?
The honest answer is that most teams don’t have a systematic strategy yet, and the ones that do usually built it by accident after shipping something that broke in production. But there are concrete patterns emerging from places where agents are actually running at scale.
The Gap Between “Tool Available” and “Tool Actually Used”
Your agent can have access to ten tools and never call the right one. A session-start hook raised Claude Code’s tool-firing rate from 15% to 62%—meaning initialization patterns matter far more than having good tool descriptions. But that’s just detecting whether a tool gets called, not whether the agent is using it effectively.
The distinction matters: an agent might call your database query tool and get the wrong schema, call your search tool with a hallucinated query, or invoke your code execution tool with syntax that won’t run. Tool firing is a prerequisite for effectiveness. Actual effectiveness means the tool call solves the problem.
What Actually Works: Objective Signal + Silent Drift Detection
Production data from 100+ real agent deployments shows that quality and token cost drift account for 38% of agent failures, while actual errors cause only ~1%. This flips the conventional monitoring playbook. You need to watch for silent degradation, not exceptions.
Start here:
-
Quality metrics per task category. If your agent handles database migrations, track schema correctness and rollback frequency. If it writes code, track test pass rate on the generated changes. If it navigates UIs, track successful form submission rate. These are objective signals specific to what you’re asking the agent to do.
-
Token cost per successful task completion. When this drifts upward without a corresponding capability improvement, something is wrong—the model changed, your tools became unreliable, or the agent started taking inefficient paths. Set an alert at 1.5x historical average.
-
Latency to first tool call. If your agent suddenly starts thinking for 10 seconds before calling a tool it used to invoke immediately, that’s a signal the model or prompt changed. Variance here usually means the agent is uncertain about which tool to use.
-
Tool call success rate by tool type. Some tools will fail more under production load than others. If your rate-limiting tool starts returning 429s, the agent will fail downstream even if it’s calling the tool correctly. Instrument success at the tool level, not the agent level.
The Practical Test: Does It Skip Tools When It Should Use Them?
Build a small validation set of tasks you know require specific tools. Give your agent 10 tasks that genuinely need the database tool, 10 that need search, 10 that need code execution. Measure:
- Did it call the right tool?
- Did the tool call have correct parameters?
- Did it recover if the tool returned an error?
This is cheap to automate. Here’s a sketch:
tasks = [
{"description": "Find all users created after 2024-01", "required_tool": "database"},
{"description": "Look up the latest news on Anthropic", "required_tool": "search"},
{"description": "Write a function that reverses a string", "required_tool": "code_execution"},
]
for task in tasks:
response = agent.run(task["description"])
called_tools = [t.name for t in response.tool_calls]
if task["required_tool"] not in called_tools:
print(f"FAIL: {task['description']} skipped {task['required_tool']}")
elif not validate_tool_params(response.tool_calls[0]):
print(f"FAIL: {task['description']} had bad parameters")
else:
print(f"PASS: {task['description']}")
Run this weekly. When your pass rate drops below your baseline, investigate before it hits production.
The Hard Part: Detecting When Tools Are Called But Ineffective
An agent can call your search tool with a well-formed query and still get useless results. It can invoke your code execution tool and write valid Python that does the wrong thing. It can query the database and misinterpret the schema.
This is where you need downstream validation. If your agent is writing code, does the generated code pass tests? If it’s making database changes, do those changes satisfy the original requirement? If it’s extracting data, does the extracted data match a ground truth set you’ve labeled?
Anthropic’s research on using AI to audit agent misbehavior found models achieve high detection rates on synthetic cases but generate so many false positives that the signal becomes useless for real deployments—AI auditors miss 96% of actual failures while flooding reviewers with false alarms. Don’t rely on another LLM to validate your agent’s tool use. Build deterministic checks where you can.
For coding agents specifically: test coverage and test pass rate are your friends. If the agent writes a function and the tests pass, the tool worked. If tests fail, the tool call either wasn’t made or was made incorrectly.
Cost-Per-Task as a Proxy for Efficiency
Databricks benchmarked its own codebase and found open-source GLM-5.2 matched Claude Opus 4.8 on agent coding while cutting costs from $1.94 to $1.28 per task. Cost per successful task completion is a real efficiency metric. If you’re paying $5 per task and a human freelancer costs $20, the agent is working. If you’re paying $15 per task, something is inefficient—either the model is overkill, or the agent is flailing.
Track this explicitly. Separate the cost of the model inference from the cost of the tools it calls. If your search tool costs $0.10 per call and your agent is calling it 20 times per task, that’s worth investigating.
The Session-Start Hook Finding
That 15% → 62% jump in tool firing was from adding explicit initialization: tell the agent what tools it has and when to use each one before it starts reasoning. This is not a prompt engineering hack. It’s architectural.
# Before: agent sees tools as options
agent = Agent(tools=[database, search, code_exec])
response = agent.run("Find recent data and write a script")
# After: agent gets explicit tool introduction
initialization_prompt = """
You have three tools available:
1. database: Query schemas and fetch data. Use when you need structured information.
2. search: Find web results and recent information. Use when you need current events.
3. code_exec: Run Python code. Use when you need to process data or create scripts.
For this task, you may need multiple tools. Pick the right one for each step.
"""
agent = Agent(tools=[database, search, code_exec], system_prompt=initialization_prompt)
response = agent.run("Find recent data and write a script")
If you’re seeing low tool-firing rates, test this before assuming the model is broken.
Bottom line: Stop monitoring whether your agent had an error. Start monitoring whether it’s calling the right tools, at the right token cost, with the right parameters. Build a small validation set of tasks with known tool requirements and run it weekly. Track quality metrics downstream (test pass rate for code, schema correctness for database work, extraction accuracy for data tasks). When token cost per task drifts upward or tool-firing rate drops without explanation, that’s your early warning.
Question via Hacker News