mcpindex.aiv0
← Index

TheCrawler

io.github.manchittlab/thecrawler·v0.1.1·Web Scraping
Quality Score
85
/100

Universal web scraper with LLM-ready markdown, RAG chunking, PDF/DOCX support.

Repository →
§01  Install
Claude Desktop (claude_desktop_config.json)
{
  "mcpServers": {
    "thecrawler": {
      "command": "npx",
      "args": [
        "-y",
        "thecrawler"
      ]
    }
  }
}
Cursor (.cursor/mcp.json)
{
  "mcpServers": {
    "thecrawler": {
      "command": "npx",
      "args": [
        "-y",
        "thecrawler"
      ]
    }
  }
}
Cline (cline_mcp_settings.json)
npx -y thecrawler
§03  MCP Quality Score  ·  methodology
freshness
25
completeness
15
installability
25
documentation
15
stability
5
§04  Alternatives in Web Scraping
ai.exa/exa
ai.exa/exa

Fast, intelligent web search and web crawling. New mcp tool: Exa-code is a context tool for coding

ai.smithery/BadRooBot-test_m
ai.smithery/BadRooBot-test_m

Send quick greetings, scrape website content, and generate text or images on demand. Perform web s…

ai.smithery/ScrapeGraphAI-scrapegraph-mcp
ai.smithery/ScrapeGraphAI-scrapegraph-mcp

Enable language models to perform advanced AI-powered web scraping with enterprise-grade reliabili…