How to monitor MCP servers for tool description drift
Short answer: hash the tool contract you approved and diff it against the live one - on a schedule at minimum, in the call path if an agent acts on the results. Nothing in the protocol does this for you. MCP defines an optional tools/list_changed notification, but servers are not required to send it and clients are not required to surface it. Tool descriptions are unsigned strings fetched at runtime; nothing ties what your agent sees today to what you reviewed at install.
The base rates say this is not paranoia. Within 89 days, 19.1% of servers on the official registry changed their published descriptor, and on the live contract surface 62.4% of safety-relevant changes shipped with no version bump - the measured numbers.
Option 1: snapshot and diff it yourself
At approval time, dump tools/list, canonicalize each tool (name, description, inputSchema, annotations), and store a SHA-256 per tool in git. Re-fetch on a cron, diff, alert on any hash move. This is cheap, dependency-free, and catches every declared change. The costs arrive later: you own alerting and dedup, a raw diff has no opinion about which changes are dangerous, and a rename is indistinguishable from a removal plus an addition. Median churn in our registry panel ran about 1.5% of descriptors per day, so weekly snapshots leave a real blind window; daily is the floor.
Option 2: client-side pinning with mcp-scan
Invariant Labs' mcp-scan pins the tool descriptions of the servers configured on your machine, re-checks them for changes, and scans descriptions for injection patterns. For a single workstation with a handful of servers it is a good answer. Its scope is also its limit: it sees only servers you already installed, only when it runs, and it has nothing to say about a server you are still evaluating.
Option 3: an in-path gate plus registry-scale monitoring
mcpindex works the problem from both ends. The gate pins each tool contract on first sight and diffs it on every later call, in the call path: a safety-relevant change - a removed tool, a new required parameter, an annotation flip toward destructive, a narrowed constraint - HOLDs the call before it executes, while proven-benign additions proceed. Separately, the crawler re-observes public registry servers continuously and publishes every contract change to the public ledger, so a server's drift history is checkable before you ever install it. That before-install view is the piece neither a DIY diff nor a client-side scanner can give you.
All three options share a limit: they diff the declared contract. A server that keeps its contract byte-identical and changes behavior underneath is invisible to each of them. That boundary is spelled out honestly in the silent-drift guide.
Which one, when
A single dev machine with a few trusted servers: mcp-scan or the DIY diff is enough. Agents acting on tool results in production: monitoring that is not in the call path alerts you after the bad call already ran, so put the diff in-path. Evaluating a server you have not installed yet: check its drift history on the ledger and its /server page first.
Related
- Does MCP notify clients when a server's tools change?
- The spec defines an optional tools/list_changed notification, but servers are not required to send it and clients are not required to act on it or surface it to you. In practice a reliable signal comes one way: re-fetch tools/list yourself and compare it against the contract you approved.
- What is tool description drift?
- Any change to a tool's declared contract - its name, description text, input or output schema, or annotations - after you first approved it. It matters because agents act on the declared contract, and MCP servers can change it remotely with no version bump your client notices.
- What is the difference between mcp-scan and mcpindex?
- mcp-scan pins and re-checks the tool descriptions of servers installed on your machine, and scans them for injection patterns. mcpindex runs an in-path gate that diffs the live contract on every call and can hold a breaking change before the call executes, and it also monitors public registry servers continuously, publishing changes to a public ledger so you can check a server's drift history before installing it.
- How often should I re-check MCP tool contracts?
- If an agent acts on tool results, diff in the call path so there is no blind window. Otherwise daily: in mcpindex's 89-day registry panel the median day saw about 1.5% of server descriptors change, so weekly snapshots can run well behind reality.