mcpgrade audit: a third of 36 popular MCP servers score D or F
mcpgrade graded 36 MCP servers on agent usability, not spec compliance — a third scored D/F due to undocumented parameters and naming collisions.
A new static-analysis tool called mcpgrade measures something the MCP spec doesn't: whether an AI agent can actually use a server's tools correctly, not just whether it's protocol-compliant. Scanning 36 popular servers, 15 scored an A while 11 — including official servers from MongoDB, Notion, Airtable, Todoist, and firecrawl — landed in D/F territory. The dominant root cause across nearly all low scorers was the same: parameters with zero description, typically because schemas are auto-generated from zod or OpenAPI without ever adding a description field.
The author validated the static findings against a live model via an --eval mode. On well-documented servers, tool-selection accuracy was 100%; on firecrawl it dropped to 84%, with errors clustering exactly on the naming collisions the static rules flagged (extract vs scrape, etc.). More concerning was refusal behavior: on small, clean catalogs the model correctly declined out-of-scope tasks 100% of the time, but on firecrawl's 26 loosely-named tools that dropped to 50% — meaning the model invented a plausible-but-wrong tool call half the time.
The takeaway for engineers: spec compliance says nothing about agent usability. Description quality, naming conventions, and error messages matter far more than protocol correctness, and most of the ecosystem is currently failing on exactly those axes.