Check whether AI crawlers (GPTBot, ClaudeBot, PerplexityBot) can access a URL. Read-only.
crawlerpeek · npm 0.3.1 · stdio
claude mcp add --transport stdio 'crawlerpeek' -- npx -y 'crawlerpeek@0.3.1'Run in a terminal; adds the server to Claude Code.
crawlerpeekExtract data from any website with thousands of scrapers, crawlers, and automations on Apify Store ⚡
Firecrawl, the context API for AI agents: search, scrape, interact with the web, plus paper search.
Connect AI agents to Exa for web search, content fetching, and multi-step research.
Brave Search MCP Server: web results, images, videos, rich results, AI summaries, and more.
Real-time web search, reasoning, and research through Perplexity's API
Real-time web search, reasoning, and research through Perplexity's API
