Robots.txt & AI Crawler Checker
robots.txt tester (RFC 9309), AI crawler access (GPTBot, ClaudeBot...) and sitemap check.
Should I use this
Quality & Safety
Based on automated analysis of tool definitions and protocol compliance.
Context Cost
This is the approximate number of tokens consumed each time the server's tools are loaded into a model's context. Higher counts reduce the attention available for other tasks.
Install
One-Click Install
Add this to your `claude_desktop_config.json` file:
{
"mcpServers": {
"robots-check": {
"url": "https://robots-check.quietworklabs.workers.dev/mcp"
}
}
}Remote endpoints
https://robots-check.quietworklabs.workers.dev/mcpstreamable-httpWhat it can do
Tool inventory
Tools (3)
🟢robots_check(url, user_agent)
Crawl-permission preflight for agents and scrapers: fetches the site's robots.txt and evaluates the URL for a user-agent token per RFC 9309 (group by product token else *, longest match, Allow wins ties, * and $ wildcards, 4xx=allow all, 5xx=disallow all). Returns allowed, the deciding rule and line, crawl-delay and sitemaps. Deterministic, no AI. Price: $0.001 USDC per call via x402 (Base); the first 5 calls per day per client are free.
Input Schema
{
"type": "object",
"properties": {
"url": {
"type": "string",
"description": "Page URL to test",
"examples": [
"https://www.nytimes.com/section/technology"
]
},
"user_agent": {
"type": "string",
"description": "Crawler product token, e.g. GPTBot or MyAgent. Default *",
"examples": [
"GPTBot"
]
}
},
"required": [
"url"
]
}🟢ai_crawler_access(url)
AI crawler access report for a URL: robots.txt verdict for 20 tokens (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, Google-Extended, Applebot-Extended, PerplexityBot, Meta-ExternalAgent, CCBot, Bytespider, Amazonbot and more) grouped by training/search/user purpose, plus page meta robots, X-Robots-Tag (noindex/noai) and llms.txt presence. Deterministic, no AI. Price: $0.002 USDC per call via x402 (Base); the first 5 calls per day per client are free.
Input Schema
{
"type": "object",
"properties": {
"url": {
"type": "string",
"description": "Site or page URL",
"examples": [
"https://www.theguardian.com/"
]
}
},
"required": [
"url"
]
}🟢sitemap_check(url)
Finds a site's sitemap (from robots.txt, else /sitemap.xml) or checks the sitemap URL given, and validates it: urlset vs sitemapindex, entry count against the 50,000 limit, lastmod format and range, relative/off-host/duplicate/http URLs, gzip support. Returns sample URLs or child sitemaps. Deterministic, no AI. Price: $0.002 USDC per call via x402 (Base); the first 5 calls per day per client are free.
Input Schema
{
"type": "object",
"properties": {
"url": {
"type": "string",
"description": "Site URL (discovers the sitemap) or a sitemap URL",
"examples": [
"https://www.python.org/"
]
}
},
"required": [
"url"
]
}Community
Evidence