Methodology · v1 advisory

How a verdict is produced.

mcpindex evaluates MCP tools and publishes a finding per tool. Today every published screen verdict is semantic-only: an LLM judge reads the tool description for hidden instructions, bound to the exact tool definition that was seen (tool_definition_hash). The deterministic conformance probe is built but has not yet run against the public corpus - so a conforming ALLOW (which the probe would have to earn) is not produced at v1; the screen emits REVIEW or UNVERIFIED. The finding is what an agent reads before it calls. Confidence is reported but not yet calibrated (calibrated=false) - the honest limits below.

The eval
  • Conformance probe
    roadmap

    Built but not yet run on the public corpus. When it runs, it drives the tool against its declared schema and checks whether observed behavior matches what the description claims (a pass/fail dimension verdict with a captured trace), gated to the D3 labeled-corpus milestone. At v1 it is built-not-run on the screen: no public verdict carries a conformance result yet, and a conformance result, when it lands, will be monitored, not enforced - it surfaces in the verdict; it does not block the call upstream.

  • Intent judge
    LLM

    Reads the tool description, schema, and example outputs adversarially. Flags hidden instructions, exfiltration patterns, prompt-injection payloads, and overclaims (e.g. 'validates' a field it never checks). Output is a pass/fail dimension verdict with rationale and severity.

  • History
    OTS

    OTS hash-chained history with cadence bound = confirmation latency (~10 min for pending; ~1 hour at N=6 confirmations for Bitcoin-finalized); sub-window precision asserted, not proven. The verdict stream for a tool is hash-chained and timestamped via OpenTimestamps; the chain is auditable end-to-end once a block confirms.

The drift gate method

The screen above is the prior an agent reads before it wires a tool. The drift gate is the live, in-path check during use. It answers a narrower, provable question: did this tool’s contract change since you pinned it? The verdict is a contract-diff, not a safety verdict - but the gate sits in the call path, so it can HOLD the call before your agent acts on the change, not merely report it after. The gate runs deterministically and entirely on your host. Its verdicts are produced by the live gate code, not a hand-written table; the reproducible scenario battery is documented in the whitepaper.

  • TOFU pin
    baseline

    On first sight of a tool (the client's tools/list), the gate pins the tool's contract trust-on-first-use: a hash over name + description + input schema, plus the captured schema. The pin can persist across restarts, so a contract that changes while your agent is offline is still caught on the next call. The first-seen contract is the baseline; the gate cannot catch drift that happened before it was installed.

  • Contract-diff
    deterministic

    On a call, the gate re-derives the live contract and compares it to the pin. A mismatch is classified into a fixed taxonomy (ChangeKind): added-required-param, required-set-expanded, constraint-narrowed, type-changed, enum-values-removed, removed-param, annotation-flip-to-destructive, output-schema-added, output-schema-changed, tool-added/removed. It also scans for injection/exfil markers in the input AND output schema and the description. No LLM, no scoring you cannot trace; a structural surprise it cannot classify fails closed (deep-schema-undiffable), never open.

  • Postures
    policy

    Monitor never blocks: every drift returns proceed-with-note. Guard (default) holds the unambiguously-breaking and dangerous changes while letting a proven-benign drift through; two kinds where behaviour is the gate (annotation-flip-to-destructive, output-schema-changed) resolve to INCONCLUSIVE rather than a flat block. Strict holds everything it cannot prove benign - NOT every drift: the benign auto-accept (added optional param, new tool, first-time output schema; description byte-identical, no risk escalation, no marker) runs BEFORE the posture layer, so a proven-benign change is re-pinned and proceeds under strict too. Anything else holds before the call.

The tier ladderA four-rung ladder. Tier 0, the deterministic contract diff over the ChangeKind taxonomy plus a marker scan, is live, runs first, is local and egresses nothing, and fails closed. Tier 1 cloud corpus lookup, tier 2 LLM consult and tier 3 behavioural verifier are built as in-path seams but each is held off by default and requires explicit opt-in. The default build is tier 0 only. The behavioural tier clears or refutes a change; it never proves a tool safe.TIER 3behavioural verifierexercises the changed toolclears or refutes a changeHELD OFF · OPT-INexecutes the toolTIER 2LLM consultreads the ambiguous casejudges what the diff cannotHELD OFF · OPT-INan LLM callTIER 1cloud corpus lookupone judgement, sharedjudged once, known everywhereHELD OFF · OPT-INa contract hash leavesTIER 0deterministic contract-diffChangeKind taxonomy + injection / exfil marker scandid this contract change versus what you pinned?no LLM, no scoring you cannot traceLIVE · RUNS FIRSTlocal · no egress · fail-closedTHE DEFAULT BUILD IS TIER-0 ONLY. THE BEHAVIOURAL TIER CLEARS OR REFUTES; IT NEVER PROVES A TOOL SAFE.
Fig. 04 · Only the deterministic rung is on. Tiers 1 to 3 are built and held off by default.
Read Fig. 04 as text
  TIER 3  behavioural verifier   clears or refutes a change   [ ] HELD OFF - opt-in
          exercises the changed tool                             executes the tool
  TIER 2  LLM consult            judges the ambiguous         [ ] HELD OFF - opt-in
                                                                 an LLM call
  TIER 1  cloud corpus lookup    judged once, known           [ ] HELD OFF - opt-in
                                 everywhere                      a contract hash leaves
  ---------------------------------------------------------------------------------
  TIER 0  deterministic          did this contract change     [X] LIVE - runs first
          contract-diff          versus what you pinned?          local - no egress
          ChangeKind taxonomy                                     fail-closed
          + marker scan

  the default build is tier-0 only. it egresses nothing.
  the behavioural tier clears or refutes. it never proves a tool safe.

Permalink, SVG & reuse · CC BY 4.0

Posture and ChangeKindA matrix of the twelve surfaced ChangeKinds against the three postures, generated from the gate source. Monitor never blocks: every kind returns proceed-with-note. Guard, the default, holds the eight kinds that carry the safety bit, resolves annotation-flip-to-destructive and output-schema-changed to inconclusive because behaviour is the gate rather than a block, and lets the two provably benign kinds proceed. Strict matches guard except that it also holds the inconclusive pair; it does not hold every drift, because the benign auto-accept runs before the posture layer and re-pins a proven-benign change. An injection or exfiltration marker is a separate scan rather than a ChangeKind, so it is not a row, though guard blocks on it.GENERATED FROM THE GATE SOURCE — VERIFIED BY DRIVING IT AT ALL THREE POSTURESCHANGEKINDWHY IT LANDS HEREMONITORGUARD (DEFAULT)STRICTadded-required-paramcarries the safety bitnotifyHOLDHOLDconstraint-narrowedcarries the safety bitnotifyHOLDHOLDdeep-schema-undiffablecarries the safety bitnotifyHOLDHOLDenum-values-removedcarries the safety bitnotifyHOLDHOLDremoved-paramcarries the safety bitnotifyHOLDHOLDrequired-set-expandedcarries the safety bitnotifyHOLDHOLDtool-removedcarries the safety bitnotifyHOLDHOLDtype-changedcarries the safety bitnotifyHOLDHOLDannotation-flip-to-destructivebehaviour is the gate, not a blocknotifyINCONCLUSIVEINCONCLUSIVEoutput-schema-changedbehaviour is the gate, not a blocknotifyINCONCLUSIVEINCONCLUSIVEadded-optional-paramproven benign - re-pinned, then proceedsproceedproceedproceedoutput-schema-addedproven benign - re-pinned, then proceedsproceedproceedproceedOf 12 surfaced kinds: 8 HOLD under guard, 2 resolve to INCONCLUSIVE (behaviour is the gate, not a block), 2 proceed as proven-benign.Strict does not hold every drift.The benign auto-accept runs before the posture layer, so aproven-benign change is re-pinned and proceeds under strict too. Monitor never blocks.An injection / exfil marker is a separate scan, not a ChangeKind, so it is not a row —but guard blocks on it, as it does on a risk escalation and any fail-closed error.
Fig. 07 · The gate reads a fixed table, not a judgement call - and strict does not hold every drift.
Read Fig. 07 as text
  ChangeKind                       MONITOR    GUARD*      STRICT
  ------------------------------------------------------------------
  added-required-param             notify     | HOLD      | HOLD
  constraint-narrowed              notify     | HOLD      | HOLD
  deep-schema-undiffable           notify     | HOLD      | HOLD
  enum-values-removed              notify     | HOLD      | HOLD
  removed-param                    notify     | HOLD      | HOLD
  required-set-expanded            notify     | HOLD      | HOLD
  tool-removed                     notify     | HOLD      | HOLD
  type-changed                     notify     | HOLD      | HOLD
  annotation-flip-to-destructive   notify     ? INCONCL.  ? INCONCL.
  output-schema-changed            notify     ? INCONCL.  ? INCONCL.
  added-optional-param             -> proceed -> proceed  -> proceed
  output-schema-added              -> proceed -> proceed  -> proceed

  * guard is the default posture.

  of 12 surfaced kinds: 8 HOLD under guard, 2 resolve to
  INCONCLUSIVE (behaviour is the gate, not a block), 2 proceed as proven-benign.

  STRICT DOES NOT HOLD EVERY DRIFT. the benign auto-accept runs BEFORE the posture
  layer, so a proven-benign change (an added optional param, a first-time output
  schema) is re-pinned and proceeds under strict too. strict holds everything it
  cannot prove benign.

  MONITOR never blocks: every kind returns PROCEED-with-note.

  an injection / exfil MARKER found in a schema or description is a separate scan,
  not a ChangeKind, so it is not a row here - but guard blocks on it, as it does on
  a risk escalation, a description change, and any fail-closed error.

  verified 2026-07-27 by driving the gate (corpus_eval/tooling/cse/gate.py) at all
  three postures.

Permalink, SVG & reuse · CC BY 4.0

Honest limits (the gate)
  • · Contract-diff, not a safety verdict. A HOLD means “this tool’s contract changed vs what you pinned” - not that the new contract is unsafe. You review the before/after and re-pin if the change is expected.
  • · Advisory in judgment, in-path in effect. The gate does not assert a tool is safe; it asserts what changed. Because it sits in the call path, that judgment can actually HOLD the call - a passive scanner can only alert after the fact.
  • · Deterministic tier-0 live; tiers 1-3 built but held off by default. The contract-diff is deterministic, runs first, and is the live, deterministic leg. Above it the ladder is built as in-path seams - a cloud tier-1 corpus lookup (a contract judged once clears or condemns it everywhere), a tier-2 LLM consult on the ambiguous, and a tier-3 behavioral verifier that exercises a changed tool to clear or refute the change - but each is held off by default and requires explicit opt-in; the default build egresses nothing and stays fail-closed. When enabled, the behavioral tier clears or refutes a contract change; it is not a proof of safety, and confidence is reported but not yet calibrated (calibrated=false at v1).
  • · Fail-closed. A tool with no pin, an unreadable contract, or a diff the gate cannot complete holds rather than proceeds. The gate never silently allows what it could not verify.
The drift network method

The in-path gate catches a change the first time you see it. The drift network catches it before you do. mcpindex crawls the public MCP registry every day, re-derives each tool’s contract, and records every silent change as a fingerprint-only entry. When you pin a tool, the gate can ask the network one question: has the crawler already caught this contract drifting? If it has, you are warned on the first call - a contract-diff advisory that rides alongside the verdict and never moves PROCEED or HOLD. Every drift the crawler catches is public in the drift ledger.

  • · Crawler-corroborated, not crowd-sourced. The public corroboration count floors at the crawler (one first-party source); forgeable install reports are excluded from the public number. The warning is real today because the crawler sees the drift, not because other installs reported it.
  • · Opt-in, privacy-by-construction. Off by default. When enabled, the only thing that leaves is a salted (HMAC) fingerprint plus closed-vocabulary fields (change type, safety flag, hour-rounded time) - never a schema, argument, description, URL, or server/tool name. Fail-open: it never blocks or changes a call.
  • · Advisory, never the decision. The fleet advisory informs; the gate’s deterministic contract-diff still decides. The network can raise your attention; it cannot move a PROCEED or a HOLD.
  • · Tool removals counted (as of 2026-07-19). The ledger also records when a tool present in one snapshot is absent from the next, scoped to servers reachable in both (a server going offline is never counted as removals). Earlier totals exclude removals. Most removals arrive as full toolset replacements rather than single-tool deletions, and the ledger labels them so. Removal entries are historical observations - a same-named tool may have since returned.
The drift network loopmcpindex crawls the public MCP registry every day, re-derives every tool contract, and records each silent change as a fingerprint-only ledger entry anchored to Bitcoin via OpenTimestamps and published on the public ledger. When your gate pins a tool it can query that network and warn you on the very first call. The corroboration count floors at the crawler as a single first-party source; forgeable install reports are excluded from the public number. The advisory rides alongside the verdict and never moves a PROCEED or a HOLD.01daily crawl of the public MCP registryevery reachable remote server02re-derive every tool contractthe same derivation your gate runs03change detectedrecorded as a fingerprint-only entry04your gate queries the network on pinGET /api/v1/drift/any?fp=…OTS anchor → the public drift ledger/ledger · fingerprint-only, hash-chained, timestampedwarned on the FIRST calladvisory only · it never moves a PROCEED or a HOLDCORROBORATION FLOORS AT THE CRAWLER (SOURCES=1). FORGEABLE INSTALL REPORTS ARE EXCLUDED FROM THE PUBLIC COUNT.
Fig. 10 · The crawler catches the drift before you do, so you are warned on call one.
Read Fig. 10 as text
        daily crawl of the public MCP registry
                     |
                     v
        re-derive every tool contract
                     |
                     v
        change detected --> fingerprint-only ledger entry --> OTS anchor --> /ledger
                     |                                                        public
                     v
        fleet query  GET /api/v1/drift/any?fp=...
                     |
                     v
        your gate warns you on the FIRST call
        advisory only - it never moves a PROCEED or a HOLD

  corroboration floors at the crawler (sources=1). forgeable install reports are
  excluded from the public count. entries are fingerprint-only: no schema, no
  argument, no description, no URL, no server or tool name.

Permalink, SVG & reuse · CC BY 4.0

Four-state verdict

The directive an agent reads is one of three decisions, on top of a status that says how the eval went. Together they are four states an agent must distinguish.

  • ALLOW
    decision (roadmap)

    The eval ran end-to-end and the tool cleared its checks at the recorded clearance level; the agent may invoke within that clearance until expires_at. Not produced at v1: a clearing ALLOW requires the behavioral conformance probe, gated to the D3 labeled-corpus milestone. Today the screen emits REVIEW or UNVERIFIED only.

  • DENY
    decision (roadmap)

    The eval ran end-to-end and a finding crossed the deny threshold (high-severity intent flag, conformance regression, or a poisoned description); the agent should not invoke. Reserved in the contract; at v1 a high-severity finding surfaces as REVIEW for human adjudication rather than an automatic public DENY.

  • REVIEW
    decision

    The eval ran but produced ambiguous or partial findings (e.g. medium-severity flag, partial conformance, provider disagreement). Surfaces the dimension findings; agent should defer to a human or fall back to its own checks.

  • UNVERIFIED
    status

    No verdict on file for this tool yet (the wire term the trust API returns when a tool has not been screened). The agent should NOT infer trust; treat as not-yet-cleared. Coverage rolls out as the corpus expands (adversarial cases first).

Two verdict surfacesTwo surfaces emit verdicts and they are separate axes. The advisory screen, read before you wire a tool, is semantic-only and out of the call path: REVIEW and UNVERIFIED are produced today, while ALLOW and DENY are reserved in the contract and not produced at v1 because a clearing ALLOW requires the behavioural conformance probe that is gated to the D3 labelled-corpus milestone. The in-path gate emits HOLD when the live contract differs from your pin and PROCEED when it matches. The screen is a prior; the gate is the decision.ADVISORY SCREEN · BEFORE YOU WIREDID THE EVAL RUN?YESNOclearedcrossed the deny thresholdambiguous or partialno eval on fileALLOWRESERVEDDENYRESERVEDREVIEWPRODUCED AT V1UNVERIFIEDPRODUCED AT V1IN-PATH GATE · DURING THE CALLHOLDthe live contract differs from what you pinned→ PROCEEDthe live contract matches your pinDETERMINISTIC · FAIL-CLOSED · IN THE CALL PATHALLOW and DENY are reserved in the contract, not produced at v1: a clearing ALLOWrequires the behavioural conformance probe, gated to the D3 labelled corpus (15/150).Neither surface overrides the other. The screen is a prior; the gate is the decision.
Fig. 09 · The screen verdict and the gate verdict are different axes. Neither overrides the other.
Read Fig. 09 as text
  ADVISORY SCREEN (before you wire)      |  IN-PATH GATE (during the call)
  -------------------------------------  |  --------------------------------
              eval ran?                  |
              yes         no             |   |- HOLD     the live contract differs
   cleared    ALLOW       .              |               from what you pinned
              reserved                   |
   crossed    DENY        .              |   -> PROCEED  the contract matches
              reserved                   |
   ambiguous  REVIEW      .              |  deterministic. fail-closed.
              live                       |
       none   .           UNVERIFIED     |
                          live           |
  -------------------------------------  |  --------------------------------
  semantic, advisory, out of path        |  in-path, can actually stop the call

  ALLOW and DENY are reserved in the contract, not produced at v1: a clearing
  ALLOW requires the behavioural conformance probe, gated to the D3 labelled
  corpus (15/150 labels). neither surface overrides the other - the
  screen is a prior, the gate is the decision.

Permalink, SVG & reuse · CC BY 4.0

Honest limits (v1)
  • · Definition, not runtime. The eval is bound to the tool definition (description + schema) at evaluation time. Runtime behavior on a specific call is not covered.
  • · Conformance built, not yet run; monitored, not enforced. The deterministic conformance probe has not run on the public corpus, so no published screen verdict carries a conformance result today. When it runs, a conformance result is reported in the verdict and the public surface, not enforced on the wire.
  • · OTS cadence bound = confirmation latency. The OTS anchor proves the verdict existed by some Bitcoin block; it does not prove minute-level ordering inside the confirmation window.
  • · calibrated = false at v1. Confidence scores are reported but not calibrated against a held-out adversarial corpus yet.
  • · Advisory, not blocking. mcpindex publishes the verdict. The agent or IDE decides whether to act.
  • · D3 graduation gate. >=150 conforming labels with FP upper-95 <=2%. Current: 15/150.

The honest-limits list is a contract. If any of these stops being true, the methodology page changes first, the verdict surface changes second, and the network only sees the upgrade after both.

Quality score (directory axis)

The trust verdict answers “does this tool behave as it claims.” The directory still answers a simpler question: which servers look mature from public registry signal. The 0-100 MCP Quality Score is a public-data composite (freshness, completeness, installability, documentation, semver stability) and remains the secondary axis on every page. Source: lib/quality.ts.

Cite this
"mcpindex: the in-path trust gate for agent tool calls." mcpindex.ai/methodology, 2026.
https://mcpindex.ai/methodology

Or just link to a server's detail page. The verdict surface and the score breakdown both render there.