Robots.txt & AI Crawler Checker
robots.txt tester (RFC 9309), AI crawler access (GPTBot, ClaudeBot...) and sitemap check.
¿Debería usar esto?
Calidad y seguridad
Basado en el análisis automatizado de las definiciones de herramientas y el cumplimiento del protocolo.
Costo de contexto
Este es el número aproximado de tokens que se consumen cada vez que las herramientas del servidor se cargan en el contexto de un modelo. Los recuentos más altos reducen la atención disponible para otras tareas.
Instalar
Instalación con un clic
Agrega esto a tu archivo `claude_desktop_config.json`:
{
"mcpServers": {
"robots-check": {
"url": "https://robots-check.quietworklabs.workers.dev/mcp"
}
}
}Puntos de conexión remotos
https://robots-check.quietworklabs.workers.dev/mcpstreamable-httpQué puede hacer
Inventario de herramientas
Herramientas (3)
🟢robots_check(url, user_agent)
Crawl-permission preflight for agents and scrapers: fetches the site's robots.txt and evaluates the URL for a user-agent token per RFC 9309 (group by product token else *, longest match, Allow wins ties, * and $ wildcards, 4xx=allow all, 5xx=disallow all). Returns allowed, the deciding rule and line, crawl-delay and sitemaps. Deterministic, no AI. Price: $0.001 USDC per call via x402 (Base); the first 5 calls per day per client are free.
Esquema de entrada
{
"type": "object",
"properties": {
"url": {
"type": "string",
"description": "Page URL to test",
"examples": [
"https://www.nytimes.com/section/technology"
]
},
"user_agent": {
"type": "string",
"description": "Crawler product token, e.g. GPTBot or MyAgent. Default *",
"examples": [
"GPTBot"
]
}
},
"required": [
"url"
]
}🟢ai_crawler_access(url)
AI crawler access report for a URL: robots.txt verdict for 20 tokens (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, Google-Extended, Applebot-Extended, PerplexityBot, Meta-ExternalAgent, CCBot, Bytespider, Amazonbot and more) grouped by training/search/user purpose, plus page meta robots, X-Robots-Tag (noindex/noai) and llms.txt presence. Deterministic, no AI. Price: $0.002 USDC per call via x402 (Base); the first 5 calls per day per client are free.
Esquema de entrada
{
"type": "object",
"properties": {
"url": {
"type": "string",
"description": "Site or page URL",
"examples": [
"https://www.theguardian.com/"
]
}
},
"required": [
"url"
]
}🟢sitemap_check(url)
Finds a site's sitemap (from robots.txt, else /sitemap.xml) or checks the sitemap URL given, and validates it: urlset vs sitemapindex, entry count against the 50,000 limit, lastmod format and range, relative/off-host/duplicate/http URLs, gzip support. Returns sample URLs or child sitemaps. Deterministic, no AI. Price: $0.002 USDC per call via x402 (Base); the first 5 calls per day per client are free.
Esquema de entrada
{
"type": "object",
"properties": {
"url": {
"type": "string",
"description": "Site URL (discovers the sitemap) or a sitemap URL",
"examples": [
"https://www.python.org/"
]
}
},
"required": [
"url"
]
}Comunidad
Evidencia