Paper of the Week — Which Metadata Matters for LLM Tool Selection in MCP? Evidence from Controlled Factor-Level Interventions
Which MCP tool metadata actually drives model selection — name, description, or parameter schema — measured across 367K real tools.
Which Metadata Matters for LLM Tool Selection in MCP? Evidence from Controlled Factor-Level Interventions
Xiaofan Li, Fengchun Qiao, Xing Gao. Published 2026-10-09. arXiv:2610.14334
One sentence summary
A controlled study of 367,174 MCP tools finds that tool names dominate selection decisions, descriptions add meaningful signal, and parameter schema structure matters far less than practitioners assume.
Why this paper
As MCP becomes the de facto standard for connecting LLMs to external services, tool selection quality is quietly becoming a reliability bottleneck. Most teams tune descriptions and hope for the best — this paper gives you the first large-scale empirical grounding for that intuition.
What they did
The authors analyzed metadata from 367,174 real MCP tools and ran controlled factor-level interventions — surgically varying names, descriptions, and parameter schemas while holding everything else fixed. They measured how each metadata component influences which tool an LLM selects, isolating contributions that anecdote-based guidance couldn’t.
Key findings
- Tool name is the single most influential metadata field; models use it as a strong prior before reading anything else
- Tool descriptions provide meaningful disambiguation when names are ambiguous or similar, but are largely ignored when names already resolve the choice
- Parameter schema structure (field names, nesting, required vs. optional) has the weakest independent effect on selection — contrary to the common belief that a well-typed schema guides the model
- Poorly named tools cause selection errors that accurate descriptions cannot reliably recover from
- The gap between best and worst metadata configurations translates directly to measurable task failure rates in multi-tool environments
Why it matters for practitioners
If you’re registering tools in an MCP server, your naming strategy is doing more work than your docstrings. Teams that invest heavily in description quality while leaving tool names generic or overlapping are optimizing the wrong lever. This is especially consequential in large catalogs where the model never sees the full tool list — poor names cause early filtering failures that cascade into downstream task errors.
What you can use today
- Audit your tool names first: check for names that are ambiguous, overly generic (e.g.,
get_data,run_query), or that overlap with sibling tools — these are your highest-leverage fixes - Use descriptions for disambiguation, not explanation: a description that restates what the name already says adds nothing; reserve description tokens for edge cases and boundary conditions the name can’t convey
- Don’t rely on schema design to correct bad naming: restructuring your parameter schema will not rescue a tool with an unclear name — fix the name, then tune the schema if needed