AI Crawler Access Check
Can GPTBot, ClaudeBot and PerplexityBot reach a site? Real fetch per crawler, firewall/CDN blocks.
Should I use this
Quality & Safety
Based on automated analysis of tool definitions and protocol compliance.
Context Cost
This is the approximate number of tokens consumed each time the server's tools are loaded into a model's context. Higher counts reduce the attention available for other tasks.
Install
One-Click Install
Add this to your `claude_desktop_config.json` file:
{
"mcpServers": {
"ai-crawler-check": {
"url": "https://check.citant.ai/mcp?src=registry"
}
}
}Remote endpoints
https://check.citant.ai/mcp?src=registrystreamable-httpWhat it can do
Tool inventory
Tools (1)
🟢check_ai_crawler_access(url)
Checks whether AI crawlers (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended and others) can actually reach a website. Goes beyond robots.txt: fetches the site as each crawler and detects firewall/CDN blocks (Cloudflare, Akamai, Imperva), challenge pages, noindex and llms.txt. Returns a 0-100 AI Access Score, a per-crawler verdict and the top issues with fixes. Takes 15-60 seconds.
Input Schema
{
"type": "object",
"properties": {
"url": {
"type": "string",
"description": "Website domain or URL, for example example.com"
}
},
"required": [
"url"
],
"additionalProperties": false
}Community
Evidence