Robots.txt & AI Crawler Checker
robots.txt tester (RFC 9309), AI crawler access (GPTBot, ClaudeBot...) and sitemap check.
我該用這個嗎
品質與安全性
根據工具定義與協定合規性的自動化分析。
上下文成本
這是每次將伺服器的工具載入模型上下文時所消耗的約略 token 數量。數量越高，可用於其他工作的注意力就越少。
安裝
一鍵安裝
將以下內容加入你的 `claude_desktop_config.json` 檔案：
{
"mcpServers": {
"robots-check": {
"url": "https://robots-check.quietworklabs.workers.dev/mcp"
}
}
}遠端端點
https://robots-check.quietworklabs.workers.dev/mcpstreamable-http它能做什麼
工具清單
工具（3）
🟢robots_check(url, user_agent)
Crawl-permission preflight for agents and scrapers: fetches the site's robots.txt and evaluates the URL for a user-agent token per RFC 9309 (group by product token else *, longest match, Allow wins ties, * and $ wildcards, 4xx=allow all, 5xx=disallow all). Returns allowed, the deciding rule and line, crawl-delay and sitemaps. Deterministic, no AI. Price: $0.001 USDC per call via x402 (Base); the first 5 calls per day per client are free.
輸入結構描述
{
"type": "object",
"properties": {
"url": {
"type": "string",
"description": "Page URL to test",
"examples": [
"https://www.nytimes.com/section/technology"
]
},
"user_agent": {
"type": "string",
"description": "Crawler product token, e.g. GPTBot or MyAgent. Default *",
"examples": [
"GPTBot"
]
}
},
"required": [
"url"
]
}🟢ai_crawler_access(url)
AI crawler access report for a URL: robots.txt verdict for 20 tokens (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, Google-Extended, Applebot-Extended, PerplexityBot, Meta-ExternalAgent, CCBot, Bytespider, Amazonbot and more) grouped by training/search/user purpose, plus page meta robots, X-Robots-Tag (noindex/noai) and llms.txt presence. Deterministic, no AI. Price: $0.002 USDC per call via x402 (Base); the first 5 calls per day per client are free.
輸入結構描述
{
"type": "object",
"properties": {
"url": {
"type": "string",
"description": "Site or page URL",
"examples": [
"https://www.theguardian.com/"
]
}
},
"required": [
"url"
]
}🟢sitemap_check(url)
Finds a site's sitemap (from robots.txt, else /sitemap.xml) or checks the sitemap URL given, and validates it: urlset vs sitemapindex, entry count against the 50,000 limit, lastmod format and range, relative/off-host/duplicate/http URLs, gzip support. Returns sample URLs or child sitemaps. Deterministic, no AI. Price: $0.002 USDC per call via x402 (Base); the first 5 calls per day per client are free.
輸入結構描述
{
"type": "object",
"properties": {
"url": {
"type": "string",
"description": "Site URL (discovers the sitemap) or a sitemap URL",
"examples": [
"https://www.python.org/"
]
}
},
"required": [
"url"
]
}社群
證據