Why does an mcp.json scan never say a server is safe?
Short answer: a config file can prove that a server is able to change, hold a credential, fetch code at launch, or reach the internet. It cannot prove that any of those is absent, and it cannot prove that a tool contract will hold still. Evidence that runs in only one direction can raise a finding; it can never issue a clearance. That is why the scan reports findings and never a green checkmark.
This page is the design rationale for that choice, including the two heuristics we shipped first and removed because they were wrong.
Transport is the wrong axis
The obvious first cut is to label remote servers changeable and local servers fixed. A hosted endpoint can be swapped server-side; a process on your own machine cannot.
We shipped that, then scanned our own config.
The most volatile server in it listens on 127.0.0.1. It is a local service that a process supervisor restarts on its own, running Python out of a git working tree that gets edited most days. Every restart picks up whatever is on disk. The hosted endpoints in the same file had not shipped a contract change in months. The label was exactly backwards for the most changeable thing in the file.
Transport tells you who can reach a server. It does not tell you whether the contract holds still. Those are different questions, and the first is not evidence for the second.
What survives is reach on its own terms: a server is spawned as a local process, listens on loopback, or is reachable off-machine. That is a fact about exposure, not a stability ranking, and the scan now states it that way.
A version pin is a claim about authority, not stability
The second heuristic: flag npx foo as risky, treat foo@1.2.3 as safe.
Two problems killed it.
It does not see most of what it is meant to catch. Across a frozen 40-day census of the public MCP registry we recorded 2,503 safety-relevant contract changes, and in 62.4% of them the declared version never moved (DOI 10.5281/zenodo.21449150). Across the wider running population of all observed contract changes, not just the safety-relevant subset, the silent share is higher still. A pin is blind to the majority of the events you would want it to see. The mechanism is covered in MCP rug pulls and silent contract drift.
And it is free to game. A maintainer adds a version string to their install snippet, every user's scan reclassifies that server as safe, and nothing about the actual risk has changed. A signal a vendor can flip by editing a README is not a security signal.
There is a real claim inside pinning, and it is worth stating precisely, because the sloppy version of it is what we removed. foo@1.2.3 does not mean the bytes hold still. It means a human has to move them. That is a statement about who holds the authority to change your dependency, and it is a genuine property. It is not a safety verdict, and a scanner cannot score it without the score being read as one.
What a config can actually prove
Every finding below is one-sided. It is real when present. Its absence proves nothing.
| Finding | What its presence proves | What its absence proves |
|---|---|---|
Credential present, and whether the value is a literal or a ${ENV_REF} | This server holds a secret, and whether that secret is sitting in a file that syncs to cloud storage and lands in dotfiles repos | Nothing. The credential may arrive by a path this file does not describe |
Fetches code at launch (npx pkg, uvx pkg, @latest, an untagged image) | The code that runs is re-resolved from a public registry every time the server starts | Nothing. A pinned command can still resolve to a mutable target |
| Reach: process, loopback, or internet | Where this server is able to talk | Nothing about whether it should |
| Filesystem breadth granted by a path argument | The server was handed a path spanning far more than a project | Nothing. Scope can be granted at runtime instead |
Plaintext http:// to a non-loopback host | The credential and every tool argument cross the network in the clear | Nothing |
The right-hand column is the whole argument.
The bias runs one direction on purpose. A wrong "broad filesystem access" is an accusation, so the scanner claims broad scope only for unmistakable cases and reports unknown otherwise. Under-claiming is the safe direction when the output is a statement about someone else's software.
Why one-sided evidence means no checkmarks
A green checkmark is an absence claim. "No credential exposure", "no drift risk" and "safe to install" all assert that something is not there, and a text file is not a source of evidence for that.
So the scan reports what it found and stops. Zero findings means this config disclosed nothing of the kinds listed above. It does not mean you are fine. That is a worse demo, and it is the only version that survives a reader who checks.
The question a config cannot answer
Contract stability is a runtime property. Four things we learned building that side, all of which hold whether or not you use ours.
Do not wait to be told. MCP defines notifications/tools/list_changed, and it is a SHOULD. The servers most likely to drift are the least likely to implement it. Treat the notification as a hint, re-pull tools/list yourself, and diff against what you pinned.
The baseline has to survive a restart. Our first pin lived in memory for the life of the session. Within a session it worked. Then a client restart re-pinned whatever the server was advertising at that moment as the new trusted baseline, so a change that landed while the host was down got absorbed as normal. That is exactly the 127.0.0.1 case above, where a restart is the delivery mechanism. Pins now persist per server on disk at ~/.mcpindex/pins/<server_id>.json. Nothing throws when you get this wrong, which is why it survived as long as it did.
"Something changed" is not "you should care." A hash over the whole tool surface tells you a diff exists and nothing about its weight. Ship that raw and the alert gets muted inside a week, which is worse than never shipping it. We classify instead: 14 change kinds detected, 12 surfaced, 10 carrying a safety bit. The two dropped are description-only and tool-added, and they are most of the volume. An annotation flipped toward destructive and a reworded docstring do not belong in the same bucket.
A pin is trust on first use. Whatever a server was advertising the day you first connected becomes your baseline. A server that was already lying then reads as perfectly stable forever. Drift measures distance from your baseline, not distance from honest, and "0 drifts observed" is easy to misread as safe. It is a different claim.
Where each of these layers runs, and what each one misses, is mapped in MCP scanners vs gateways.
What one real config looked like
Twelve servers. Six re-resolve their code from a public registry at every launch. Two hold a credential, both as ${ENV} references rather than literal values. Two can reach off-machine.
The six was the surprise, and it is the number most people have never looked at for their own setup.
What this page does not claim
Nothing here says a server is safe, and the scan will not say it either. A clean result means the file disclosed no findings of the kinds above. It is silent about behavior, about intent, and about what the contract will look like tomorrow. For the tool-by-tool view of what your agent can already do, see What an mcp.json audit tells you.
The scan runs entirely in your browser. Nothing is uploaded and no account is required.
- Does a remote MCP server change more often than a local one?
- Not reliably. Transport tells you who can reach a server, not whether its tool contract holds still. A loopback service running out of a live git checkout under a process supervisor can change on every restart, while a hosted endpoint may go months without a contract change. Transport is a fact about exposure, not a stability ranking.
- Does pinning a version stop MCP contract drift?
- No. Across a frozen 40-day census of the public MCP registry, 62.4% of 2,503 safety-relevant contract changes kept the declared version unchanged, so a pin does not see them. A pin is a claim about who holds authority to change your dependency, not a claim that the contract is stable.
- What does a scan with zero findings mean?
- It means the config file disclosed nothing of the kinds a scanner can prove from text. It is not a clearance. Change-capability can be proven from a config; its absence cannot, so the scan reports findings and never a green checkmark.
- How do you detect an MCP tool contract change at runtime?
- Re-pull tools/list yourself rather than waiting for notifications/tools/list_changed, which the spec marks as a SHOULD. Diff the live surface against a baseline pin stored on disk so it survives a client restart, and classify the diff by change kind rather than alerting on any difference.
- Can drift detection tell me an MCP server is honest?
- No. A pin is trust on first use, so a server that was already misrepresenting itself when you first connected reads as perfectly stable. Drift measures distance from your baseline, not distance from honest.