RobotsGate
Generate, validate and check robots.txt rules for AI crawlers. Free; optional Pro key.
Should I use this
Quality & Safety
Based on automated analysis of tool definitions and protocol compliance.
Context Cost
This is the approximate number of tokens consumed each time the server's tools are loaded into a model's context. Higher counts reduce the attention available for other tasks.
Install
One-Click Install
Add this to your `claude_desktop_config.json` file:
{
"mcpServers": {
"robotsgate": {
"url": "https://robotsgate.mike-tusa.workers.dev/mcp"
}
}
}Remote endpoints
https://robotsgate.mike-tusa.workers.dev/mcpstreamable-httpWhat it can do
Tool inventory
Tools (4)
🟢list_crawlers(category)
Return RobotsGate's AI crawler registry: each crawler's robots.txt token, operator, category (training, dataset, search, user_fetch or other), whether the operator says it respects robots.txt, and a link to the operator's documentation, plus unverified entries and notes. Read-only, no network access. Same data as GET /api/crawlers.
Input Schema
{
"type": "object",
"properties": {
"category": {
"type": "string",
"enum": [
"training",
"dataset",
"search",
"user_fetch",
"other"
],
"description": "Optional: only crawlers in this category."
}
},
"additionalProperties": false
}Output Schema
{
"type": "object",
"properties": {
"schema_version": {
"type": "integer"
},
"generated": {
"type": "string"
},
"categories": {
"type": "object"
},
"crawlers": {
"type": "array",
"items": {
"type": "object"
}
},
"unverified": {
"type": "array",
"items": {
"type": "object"
}
}
},
"required": [
"categories",
"crawlers"
]
}🟢generate_robots(categories, agents, default, disallow_paths, allow_paths, ...)
Build a robots.txt from per-category choices (allow, block or omit for training, dataset, search, user_fetch or other), optional per-token overrides, a default policy for other bots, paths, sitemaps and custom rules. Returns the file text, a per-crawler summary and notes. Read-only: it only returns text and changes nothing. Same engine as POST /api/generate.
Input Schema
{
"type": "object",
"properties": {
"categories": {
"type": "object",
"description": "Action per crawler category. Omitted categories get no group of their own.",
"properties": {
"training": {
"type": "string",
"enum": [
"allow",
"block",
"omit"
]
},
"dataset": {
"type": "string",
"enum": [
"allow",
"block",
"omit"
]
},
"search": {
"type": "string",
"enum": [
"allow",
"block",
"omit"
]
},
"user_fetch": {
"type": "string",
"enum": [
"allow",
"block",
"omit"
]
},
"other": {
"type": "string",
"enum": [
"allow",
"block",
"omit"
]
}
},
"additionalProperties": false
},
"agents": {
"type": "object",
"description": "Per-token overrides, e.g. {\"GPTBot\": \"block\"}. Tokens: letters, digits, \"_\", \".\" and \"-\", up to 64 characters (not \"*\"). At most 100.",
"maxProperties": 100,
"propertyNames": {
"pattern": "^[A-Za-z0-9_.-]{1,64}$"
},
"additionalProperties": {
"type": "string",
"enum": [
"allow",
"block",
"omit"
]
}
},
"default": {
"type": "string",
"enum": [
"allow",
"block"
],
"description": "Policy for \"User-agent: *\" (default allow)."
},
"disallow_paths": {
"type": "array",
"maxItems": 100,
"items": {
"type": "string",
"minLength": 1,
"maxLength": 512
},
"description": "Paths to disallow for \"*\" and for allowed AI groups. Each path starts with \"/\" or \"*\" and has no whitespace or \"#\". At most 100."
},
"allow_paths": {
"type": "array",
"maxItems": 100,
"items": {
"type": "string",
"minLength": 1,
"maxLength": 512
},
"description": "Paths to allow explicitly. Each path starts with \"/\" or \"*\" and has no whitespace or \"#\". At most 100."
},
"sitemaps": {
"type": "array",
"maxItems": 20,
"items": {
"type": "string",
"maxLength": 2048
},
"description": "Absolute http(s) sitemap URLs. At most 20."
},
"sitemap": {
"type": "string",
"maxLength": 2048,
"description": "A single sitemap URL (alternative to sitemaps)."
},
"custom_rules": {
"type": "string",
"maxLength": 20000,
"description": "Extra robots.txt lines appended after validation (max 20000 bytes)."
},
"date": {
"type": "string",
"maxLength": 10,
"pattern": "^\\d{4}-\\d{2}-\\d{2}$",
"description": "Date for the header comment (YYYY-MM-DD). Default: today (UTC)."
}
},
"additionalProperties": false
}Output Schema
{
"type": "object",
"properties": {
"robots_txt": {
"type": "string"
},
"summary": {
"type": "array",
"items": {
"type": "object"
}
},
"notes": {
"type": "array",
"items": {
"type": "string"
}
},
"personal_agents_note": {
"type": "string"
}
},
"required": [
"robots_txt",
"summary",
"notes"
]
}🟢validate_robots(robots_txt, path)
Validate robots.txt text against RFC 9309 and report errors, warnings and, for one URL path, which registry AI crawlers are allowed or blocked and by which rule. Read-only, no network access. Same engine as POST /api/validate. Up to 50,000 characters over MCP (the whole JSON-RPC request must also fit in 64 KB); for larger files use POST /api/validate or check_site.
Input Schema
{
"type": "object",
"properties": {
"robots_txt": {
"type": "string",
"maxLength": 50000,
"description": "The robots.txt file content."
},
"path": {
"type": "string",
"maxLength": 2048,
"description": "URL path to test, starting with \"/\" (default \"/\")."
}
},
"required": [
"robots_txt"
],
"additionalProperties": false
}Output Schema
{
"type": "object",
"properties": {
"valid": {
"type": "boolean"
},
"errors": {
"type": "array",
"items": {
"type": "object"
}
},
"warnings": {
"type": "array",
"items": {
"type": "object"
}
},
"info": {
"type": "array",
"items": {
"type": "object"
}
},
"diagnostics": {
"type": "object"
},
"stats": {
"type": "object"
},
"path": {
"type": "string"
},
"ai_crawlers": {
"type": "array",
"items": {
"type": "object"
}
},
"personal_agents": {
"type": "object"
}
},
"required": [
"valid",
"errors",
"warnings",
"ai_crawlers"
]
}🟢check_site(url, path)
Fetch a site's /robots.txt (http/https on the default port only; private, loopback and internal addresses are refused; up to 512,000 bytes; results cached for 5 minutes) and report, for one path, which registry AI crawlers are allowed or blocked, plus validation errors and warnings. Read-only: one GET of /robots.txt with a RobotsGate user-agent. Same engine as GET /api/check.
Input Schema
{
"type": "object",
"properties": {
"url": {
"type": "string",
"minLength": 1,
"maxLength": 2048,
"description": "Site URL or bare hostname, e.g. example.com. Only the scheme and host are used."
},
"path": {
"type": "string",
"maxLength": 2048,
"description": "URL path to test, starting with \"/\" (default \"/\")."
}
},
"required": [
"url"
],
"additionalProperties": false
}Output Schema
{
"type": "object",
"properties": {
"url": {
"type": "string"
},
"final_url": {
"type": "string"
},
"http_status": {
"type": "integer"
},
"redirects": {
"type": "integer"
},
"fetched_at": {
"type": "string"
},
"cache": {
"type": "object"
},
"robots_found": {
"type": "boolean"
},
"interpretation": {
"type": "string",
"enum": [
"parsed",
"unavailable",
"unreachable"
]
},
"path": {
"type": "string"
},
"ai_crawlers": {
"anyOf": [
{
"type": "array",
"items": {
"type": "object"
}
},
{
"type": "null"
}
],
"description": "Per-crawler results, or null when robots.txt was unreachable"
},
"personal_agents": {
"anyOf": [
{
"type": "object"
},
{
"type": "null"
}
],
"description": "Object, or null when robots.txt was unreachable"
}
},
"required": [
"url",
"final_url",
"http_status",
"robots_found",
"interpretation"
]
}Community
Evidence