RobotsGate
Generate, validate and check robots.txt rules for AI crawlers. Free; optional Pro key.
Sollte ich dies verwenden
Qualität und Sicherheit
Basierend auf einer automatisierten Analyse der Tool-Definitionen und der Einhaltung des Protokolls.
Kontextkosten
Dies ist die ungefähre Anzahl der Tokens, die jedes Mal verbraucht werden, wenn die Tools des Servers in den Kontext eines Modells geladen werden. Höhere Werte verringern die Aufmerksamkeit, die für andere Aufgaben verfügbar ist.
Installieren
Installation mit einem Klick
Fügen Sie dies Ihrer Datei `claude_desktop_config.json` hinzu:
{
"mcpServers": {
"robotsgate": {
"url": "https://robotsgate.mike-tusa.workers.dev/mcp"
}
}
}Remote-Endpunkte
https://robotsgate.mike-tusa.workers.dev/mcpstreamable-httpWas es kann
Tool-Inventar
Tools (4)
🟢list_crawlers(category)
Return RobotsGate's AI crawler registry: each crawler's robots.txt token, operator, category (training, dataset, search, user_fetch or other), whether the operator says it respects robots.txt, and a link to the operator's documentation, plus unverified entries and notes. Read-only, no network access. Same data as GET /api/crawlers.
Eingabe-Schema
{
"type": "object",
"properties": {
"category": {
"type": "string",
"enum": [
"training",
"dataset",
"search",
"user_fetch",
"other"
],
"description": "Optional: only crawlers in this category."
}
},
"additionalProperties": false
}Ausgabe-Schema
{
"type": "object",
"properties": {
"schema_version": {
"type": "integer"
},
"generated": {
"type": "string"
},
"categories": {
"type": "object"
},
"crawlers": {
"type": "array",
"items": {
"type": "object"
}
},
"unverified": {
"type": "array",
"items": {
"type": "object"
}
}
},
"required": [
"categories",
"crawlers"
]
}🟢generate_robots(categories, agents, default, disallow_paths, allow_paths, ...)
Build a robots.txt from per-category choices (allow, block or omit for training, dataset, search, user_fetch or other), optional per-token overrides, a default policy for other bots, paths, sitemaps and custom rules. Returns the file text, a per-crawler summary and notes. Read-only: it only returns text and changes nothing. Same engine as POST /api/generate.
Eingabe-Schema
{
"type": "object",
"properties": {
"categories": {
"type": "object",
"description": "Action per crawler category. Omitted categories get no group of their own.",
"properties": {
"training": {
"type": "string",
"enum": [
"allow",
"block",
"omit"
]
},
"dataset": {
"type": "string",
"enum": [
"allow",
"block",
"omit"
]
},
"search": {
"type": "string",
"enum": [
"allow",
"block",
"omit"
]
},
"user_fetch": {
"type": "string",
"enum": [
"allow",
"block",
"omit"
]
},
"other": {
"type": "string",
"enum": [
"allow",
"block",
"omit"
]
}
},
"additionalProperties": false
},
"agents": {
"type": "object",
"description": "Per-token overrides, e.g. {\"GPTBot\": \"block\"}. Tokens: letters, digits, \"_\", \".\" and \"-\", up to 64 characters (not \"*\"). At most 100.",
"maxProperties": 100,
"propertyNames": {
"pattern": "^[A-Za-z0-9_.-]{1,64}$"
},
"additionalProperties": {
"type": "string",
"enum": [
"allow",
"block",
"omit"
]
}
},
"default": {
"type": "string",
"enum": [
"allow",
"block"
],
"description": "Policy for \"User-agent: *\" (default allow)."
},
"disallow_paths": {
"type": "array",
"maxItems": 100,
"items": {
"type": "string",
"minLength": 1,
"maxLength": 512
},
"description": "Paths to disallow for \"*\" and for allowed AI groups. Each path starts with \"/\" or \"*\" and has no whitespace or \"#\". At most 100."
},
"allow_paths": {
"type": "array",
"maxItems": 100,
"items": {
"type": "string",
"minLength": 1,
"maxLength": 512
},
"description": "Paths to allow explicitly. Each path starts with \"/\" or \"*\" and has no whitespace or \"#\". At most 100."
},
"sitemaps": {
"type": "array",
"maxItems": 20,
"items": {
"type": "string",
"maxLength": 2048
},
"description": "Absolute http(s) sitemap URLs. At most 20."
},
"sitemap": {
"type": "string",
"maxLength": 2048,
"description": "A single sitemap URL (alternative to sitemaps)."
},
"custom_rules": {
"type": "string",
"maxLength": 20000,
"description": "Extra robots.txt lines appended after validation (max 20000 bytes)."
},
"date": {
"type": "string",
"maxLength": 10,
"pattern": "^\\d{4}-\\d{2}-\\d{2}$",
"description": "Date for the header comment (YYYY-MM-DD). Default: today (UTC)."
}
},
"additionalProperties": false
}Ausgabe-Schema
{
"type": "object",
"properties": {
"robots_txt": {
"type": "string"
},
"summary": {
"type": "array",
"items": {
"type": "object"
}
},
"notes": {
"type": "array",
"items": {
"type": "string"
}
},
"personal_agents_note": {
"type": "string"
}
},
"required": [
"robots_txt",
"summary",
"notes"
]
}🟢validate_robots(robots_txt, path)
Validate robots.txt text against RFC 9309 and report errors, warnings and, for one URL path, which registry AI crawlers are allowed or blocked and by which rule. Read-only, no network access. Same engine as POST /api/validate. Up to 50,000 characters over MCP (the whole JSON-RPC request must also fit in 64 KB); for larger files use POST /api/validate or check_site.
Eingabe-Schema
{
"type": "object",
"properties": {
"robots_txt": {
"type": "string",
"maxLength": 50000,
"description": "The robots.txt file content."
},
"path": {
"type": "string",
"maxLength": 2048,
"description": "URL path to test, starting with \"/\" (default \"/\")."
}
},
"required": [
"robots_txt"
],
"additionalProperties": false
}Ausgabe-Schema
{
"type": "object",
"properties": {
"valid": {
"type": "boolean"
},
"errors": {
"type": "array",
"items": {
"type": "object"
}
},
"warnings": {
"type": "array",
"items": {
"type": "object"
}
},
"info": {
"type": "array",
"items": {
"type": "object"
}
},
"diagnostics": {
"type": "object"
},
"stats": {
"type": "object"
},
"path": {
"type": "string"
},
"ai_crawlers": {
"type": "array",
"items": {
"type": "object"
}
},
"personal_agents": {
"type": "object"
}
},
"required": [
"valid",
"errors",
"warnings",
"ai_crawlers"
]
}🟢check_site(url, path)
Fetch a site's /robots.txt (http/https on the default port only; private, loopback and internal addresses are refused; up to 512,000 bytes; results cached for 5 minutes) and report, for one path, which registry AI crawlers are allowed or blocked, plus validation errors and warnings. Read-only: one GET of /robots.txt with a RobotsGate user-agent. Same engine as GET /api/check.
Eingabe-Schema
{
"type": "object",
"properties": {
"url": {
"type": "string",
"minLength": 1,
"maxLength": 2048,
"description": "Site URL or bare hostname, e.g. example.com. Only the scheme and host are used."
},
"path": {
"type": "string",
"maxLength": 2048,
"description": "URL path to test, starting with \"/\" (default \"/\")."
}
},
"required": [
"url"
],
"additionalProperties": false
}Ausgabe-Schema
{
"type": "object",
"properties": {
"url": {
"type": "string"
},
"final_url": {
"type": "string"
},
"http_status": {
"type": "integer"
},
"redirects": {
"type": "integer"
},
"fetched_at": {
"type": "string"
},
"cache": {
"type": "object"
},
"robots_found": {
"type": "boolean"
},
"interpretation": {
"type": "string",
"enum": [
"parsed",
"unavailable",
"unreachable"
]
},
"path": {
"type": "string"
},
"ai_crawlers": {
"anyOf": [
{
"type": "array",
"items": {
"type": "object"
}
},
{
"type": "null"
}
],
"description": "Per-crawler results, or null when robots.txt was unreachable"
},
"personal_agents": {
"anyOf": [
{
"type": "object"
},
{
"type": "null"
}
],
"description": "Object, or null when robots.txt was unreachable"
}
},
"required": [
"url",
"final_url",
"http_status",
"robots_found",
"interpretation"
]
}Community
Nachweis