Best Web Scraping MCP servers.

Analyse robots.txt content and report whether a path may be crawled.

One MCP server for 180+ live web-data APIs returning clean JSON from sites that block scrapers.

Personal context for AI, written by you and never scraped: your profile and files, any MCP client.

All HasData scraping tools in one MCP server: Google, TikTok, Instagram, maps, e-commerce and more.

Customer-voice research: scrape reviews and comments from 30+ sites, find ads, hooks, creators.

Read any page past its anti-bot wall, crawl sites, search without an API key, solve captchas locally

Asks 8 AI crawlers for your page, checks who really answered, and writes the fixes. MIT.

Your AI agent drives a real, logged-in Chrome — test authenticated apps, 2FA flows, scrape logins

All public upAPI operations as MCP tools: web scraping, search, screenshots, PDF, OCR and more.

Give it a store URL, get structured rows. A model writes the parser once; replays are free.

Turn any website or API into structured JSON with LLM-authored declarative scraping configs.

Crawl documentation sites into local corpora agents can search, read, and diff fully offline.

Reddit posts, comments, search, users & subreddits as JSON via Apify. No Reddit API key. $2/1k

Reddit posts, comments, search, users & subreddits as JSON via Apify. No Reddit API key. $1.50/1k

AI visibility checker: GPTBot vs OAI-SearchBot, robots.txt audit, raw-HTML schema. 165 crawlers.

X/Twitter automation MCP: scrape, post, schedule, analyze, engage. No API key required.

Ask before you fetch: will this domain serve your crawler, refuse it, or charge it?

Web scraping & search for AI agents: extract, SearXNG search, rerank, robots compliance. No API key.

Scrape, crawl, and map websites to Markdown or JSON via local CLI.

Scrape, crawl, map and extract the web. Pay per call in USDC, no account or API key.