mcpindex.aiv0
← Index

io.github.sharozdawa/robotstxt-ai

io.github.sharozdawa/robotstxt-ai·v1.0.1·Web Scraping
Quality Score
84
/100

Analyze and generate robots.txt with AI crawler detection for 20+ bots like GPTBot and ClaudeBot.

Repository →
§01  Install
Claude Desktop (claude_desktop_config.json)
{
  "mcpServers": {
    "robotstxt-ai": {
      "command": "npx",
      "args": [
        "-y",
        "robotstxt-ai-mcp"
      ]
    }
  }
}
Cursor (.cursor/mcp.json)
{
  "mcpServers": {
    "robotstxt-ai": {
      "command": "npx",
      "args": [
        "-y",
        "robotstxt-ai-mcp"
      ]
    }
  }
}
Cline (cline_mcp_settings.json)
npx -y robotstxt-ai-mcp
§03  MCP Quality Score  ·  methodology
freshness
24
completeness
10
installability
25
documentation
15
stability
10
§04  Alternatives in Web Scraping
ai.exa/exa
ai.exa/exa

Fast, intelligent web search and web crawling. New mcp tool: Exa-code is a context tool for coding

ai.smithery/BadRooBot-test_m
ai.smithery/BadRooBot-test_m

Send quick greetings, scrape website content, and generate text or images on demand. Perform web s…

ai.smithery/ScrapeGraphAI-scrapegraph-mcp
ai.smithery/ScrapeGraphAI-scrapegraph-mcp

Enable language models to perform advanced AI-powered web scraping with enterprise-grade reliabili…