Robots.txt & AI Crawler Checker
robots.txt tester (RFC 9309), AI crawler access (GPTBot, ClaudeBot...) and sitemap check.
Sollte ich dies verwenden
Qualität und Sicherheit
Basierend auf einer automatisierten Analyse der Tool-Definitionen und der Einhaltung des Protokolls.
Kontextkosten
Dies ist die ungefähre Anzahl der Tokens, die jedes Mal verbraucht werden, wenn die Tools des Servers in den Kontext eines Modells geladen werden. Höhere Werte verringern die Aufmerksamkeit, die für andere Aufgaben verfügbar ist.
Installieren
Installation mit einem Klick
Fügen Sie dies Ihrer Datei `claude_desktop_config.json` hinzu:
{
"mcpServers": {
"robots-check": {
"url": "https://robots-check.quietworklabs.workers.dev/mcp"
}
}
}Remote-Endpunkte
https://robots-check.quietworklabs.workers.dev/mcpstreamable-httpWas es kann
Tool-Inventar
Tools (3)
🟢robots_check(url, user_agent)
Crawl-permission preflight for agents and scrapers: fetches the site's robots.txt and evaluates the URL for a user-agent token per RFC 9309 (group by product token else *, longest match, Allow wins ties, * and $ wildcards, 4xx=allow all, 5xx=disallow all). Returns allowed, the deciding rule and line, crawl-delay and sitemaps. Deterministic, no AI. Price: $0.001 USDC per call via x402 (Base); the first 5 calls per day per client are free.
Eingabe-Schema
{
"type": "object",
"properties": {
"url": {
"type": "string",
"description": "Page URL to test",
"examples": [
"https://www.nytimes.com/section/technology"
]
},
"user_agent": {
"type": "string",
"description": "Crawler product token, e.g. GPTBot or MyAgent. Default *",
"examples": [
"GPTBot"
]
}
},
"required": [
"url"
]
}🟢ai_crawler_access(url)
AI crawler access report for a URL: robots.txt verdict for 20 tokens (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, Google-Extended, Applebot-Extended, PerplexityBot, Meta-ExternalAgent, CCBot, Bytespider, Amazonbot and more) grouped by training/search/user purpose, plus page meta robots, X-Robots-Tag (noindex/noai) and llms.txt presence. Deterministic, no AI. Price: $0.002 USDC per call via x402 (Base); the first 5 calls per day per client are free.
Eingabe-Schema
{
"type": "object",
"properties": {
"url": {
"type": "string",
"description": "Site or page URL",
"examples": [
"https://www.theguardian.com/"
]
}
},
"required": [
"url"
]
}🟢sitemap_check(url)
Finds a site's sitemap (from robots.txt, else /sitemap.xml) or checks the sitemap URL given, and validates it: urlset vs sitemapindex, entry count against the 50,000 limit, lastmod format and range, relative/off-host/duplicate/http URLs, gzip support. Returns sample URLs or child sitemaps. Deterministic, no AI. Price: $0.002 USDC per call via x402 (Base); the first 5 calls per day per client are free.
Eingabe-Schema
{
"type": "object",
"properties": {
"url": {
"type": "string",
"description": "Site URL (discovers the sitemap) or a sitemap URL",
"examples": [
"https://www.python.org/"
]
}
},
"required": [
"url"
]
}Community
Nachweis