Octocrawl
Web pages as clean Markdown and JSON for AI agents, in fewer tokens, with an Evidence Record.
Should I use this
Quality & Safety
A
Based on automated analysis of tool definitions and protocol compliance.
Context Cost
~2,836Tokens (tool definitions)
~15.7 KBTypical response size
Significant attention impact (2.22% of 128k context)
This is the approximate number of tokens consumed each time the server's tools are loaded into a model's context. Higher counts reduce the attention available for other tasks.
Install
One-Click Install
Add this to your `claude_desktop_config.json` file:
{
"mcpServers": {
"octocrawl": {
"command": "npx",
"args": [
"@octocrawl/mcp"
]
}
}
}Runnable packages
npm@octocrawl/mcp
0.7.0stdioRemote endpoints
https://mcp.octocrawl.dev/mcpstreamable-httphttps://mcp.octocrawl.dev/mcp-oauthstreamable-httpWhat it can do
Tool inventory
Tools (2)
🟢 Read-only🟡 Write🔴 Delete⚪ Unknown
🟢scrape(url, mode, allowlistedDomains, formats, includeLinks, ...)
Read one URL as Markdown with its Evidence Record (final URL, fetch time, HTTP status, robots.txt decision, hashes), over HTTP first and in the browser when the page needs it. Not for: many URLs or a whole site section. Compact by default; debug adds the full audit. warnings name what the content cannot vouch for (client_rendered_suspected, robots_overridden), and agentHints the next step.
Input Schema
{
"type": "object",
"properties": {
"url": {
"type": "string",
"description": "An http(s) URL."
},
"mode": {
"type": "string",
"enum": [
"standard",
"research"
],
"description": "The declared identity the page is read under: standard (default) or research. A hosted server holds no saved logins, so it has no authed mode."
},
"allowlistedDomains": {
"type": "array",
"items": {
"type": "string"
}
},
"formats": {
"type": "array",
"minItems": 1,
"description": "What to return (default markdown): markdown, links, html, rawHtml, images, tables, json, attributes ({ selectors }), list ({ itemSelector?, fields? }: a record per repeated item, detected without itemSelector), screenshot or screenshot@fullPage (browser only). This connection has no browser lane, so it offers no screenshot.",
"items": {
"anyOf": [
{
"type": "string",
"enum": [
"markdown",
"links",
"json",
"html",
"rawHtml",
"images",
"tables"
]
},
{
"type": "object",
"properties": {
"type": {
"const": "json"
},
"schema": {
"type": "object"
},
"prompt": {
"type": "string",
"maxLength": 4000
},
"modelFallback": {
"type": "boolean"
}
},
"required": [
"type",
"schema"
],
"additionalProperties": false
},
{
"type": "object",
"properties": {
"type": {
"const": "attributes"
},
"selectors": {
"type": "array",
"minItems": 1,
"maxItems": 50,
"items": {
"type": "object",
"properties": {
"selector": {
"type": "string",
"minLength": 1,
"maxLength": 200
},
"attribute": {
"type": "string",
"minLength": 1,
"maxLength": 100,
"pattern": "^[A-Za-z_][A-Za-z0-9_:.-]*$"
}
},
"required": [
"selector",
"attribute"
],
"additionalProperties": false
}
}
},
"required": [
"type",
"selectors"
],
"additionalProperties": false
},
{
"type": "object",
"properties": {
"type": {
"const": "list"
},
"itemSelector": {
"type": "string",
"minLength": 1,
"maxLength": 200
},
"fields": {
"type": "array",
"minItems": 1,
"maxItems": 50,
"items": {
"type": "object",
"properties": {
"name": {
"type": "string",
"minLength": 1,
"maxLength": 64
},
"selector": {
"type": "string",
"minLength": 1,
"maxLength": 200
},
"attribute": {
"type": "string",
"pattern": "^[A-Za-z_][A-Za-z0-9_:.-]*$"
}
},
"required": [
"name"
],
"additionalProperties": false
}
}
},
"required": [
"type"
],
"additionalProperties": false
}
]
}
},
"includeLinks": {
"type": "boolean",
"description": "Also return the page's outbound links. Default false."
},
"debug": {
"type": "boolean",
"description": "Return the full audit: trace, ladderTrace and every attempt."
},
"onlyMainContent": {
"type": "boolean",
"description": "false keeps the whole page (header, navigation, footer). Default true."
},
"timeout": {
"type": "integer",
"minimum": 1000,
"maximum": 100000,
"description": "Deadline in milliseconds for the whole scrape. When it fires the result is partial with the content so far, or failed/timeout. Here at most and by default 100000; a longer one is lowered to it."
},
"maxFileBytes": {
"type": "integer",
"minimum": 1,
"maximum": 5242880,
"description": "Largest file to download; a larger one is failed/body_too_large."
},
"includeTags": {
"type": "array",
"maxItems": 100,
"items": {
"type": "string",
"minLength": 1,
"maxLength": 200
},
"description": "CSS selectors of the only elements to keep, whatever onlyMainContent says (no sibling combinators, :nth-child or :has)."
},
"excludeTags": {
"type": "array",
"maxItems": 100,
"items": {
"type": "string",
"minLength": 1,
"maxLength": 200
},
"description": "CSS selectors removed with their contents."
},
"headers": {
"type": "object",
"maxProperties": 32,
"additionalProperties": {
"type": "string",
"maxLength": 4096
},
"description": "Extra headers to the page's origin, recorded; User-Agent, client hints, authorization and cookie are refused."
},
"mobile": {
"type": "boolean",
"description": "Read as declared mobile Chrome (Android UA, 412x915). Default false."
},
"fastMode": {
"type": "boolean",
"description": "HTTP only: a page needing scripts returns the HTTP verdict (a shell is failed/empty_unverified). Default false."
},
"blockAds": {
"type": "boolean",
"description": "Block ad hosts and ad or cookie-banner elements in the browser. Default true."
},
"removeBase64Images": {
"type": "boolean",
"description": "Drop data: URI images from the Markdown, keeping alt text. Default true."
},
"verify": {
"type": "object",
"description": "Checks judged into verification: passed (these checks only), failed or not_requested (unverified). Examine a failure before retrying. Types: markdownIncludes {text}, markdownMatches {pattern, flags?}, markdownCountMin {pattern, flags?, min}, minTables {min}, tableRowsMin {tableIndex, columns, min} (zero-based; needs tables format), listRecordsMin {min, path?}, recordFields {fields, min}, field {path + one of equals, in, present, min/max}. Markdown checks and minTables need markdown in formats.",
"properties": {
"checks": {
"type": "array",
"minItems": 1,
"maxItems": 32,
"items": {
"type": "object",
"properties": {
"type": {
"type": "string",
"enum": [
"markdownIncludes",
"minTables",
"tableRowsMin",
"listRecordsMin",
"recordFields",
"field"
]
},
"text": {
"type": "string",
"minLength": 1,
"maxLength": 1000
},
"pattern": {
"type": "string",
"minLength": 1,
"maxLength": 1000
},
"flags": {
"type": "string",
"pattern": "^[imsuy]*$"
},
"min": {
"type": "number"
},
"max": {
"type": "number"
},
"path": {
"type": "string",
"minLength": 1,
"maxLength": 1000
},
"fields": {
"type": "array",
"minItems": 1,
"maxItems": 50,
"items": {
"type": "string",
"minLength": 1,
"maxLength": 1000
}
},
"tableIndex": {
"type": "integer",
"minimum": 0,
"maximum": 1000000
},
"columns": {
"type": "array",
"minItems": 1,
"maxItems": 50,
"uniqueItems": true,
"items": {
"type": "integer",
"minimum": 0,
"maximum": 1000000
}
},
"equals": {
"type": [
"string",
"number",
"boolean",
"null"
]
},
"in": {
"type": "array",
"minItems": 1,
"maxItems": 32,
"items": {
"type": [
"string",
"number",
"boolean",
"null"
]
}
},
"present": {
"type": "boolean"
}
},
"required": [
"type"
],
"additionalProperties": false
}
},
"emptyOk": {
"type": "boolean",
"description": "An empty result (empty_verified, e.g. no search hits) passes. Default false."
}
},
"required": [
"checks"
],
"additionalProperties": false
},
"integration": {
"type": "string",
"minLength": 1,
"maxLength": 100,
"pattern": "^[\\x21-\\x7e]+$",
"description": "Your workflow label (printable, no spaces); recorded, not sent to sites."
}
},
"required": [
"url"
],
"additionalProperties": false
}Output Schema
{
"type": "object",
"properties": {
"status": {
"enum": [
"success",
"partial",
"empty_verified",
"blocked",
"failed",
"cancelled",
"budget_exceeded",
"duplicate"
]
},
"failureReason": {
"enum": [
"empty_unverified",
"timeout",
"dns_error",
"connection_error",
"tls_error",
"http_error",
"redirect_limit",
"redirect_loop",
"body_too_large",
"decompressed_too_large",
"unsupported_content_type",
"unsupported_content_encoding",
"parse_error",
"loop_detected",
"policy_denied",
"provider_error",
"identity_compromised",
"internal_error",
"cache_miss",
"action_failed",
null
]
},
"blockReason": {
"enum": [
"cloudflare_challenge",
"captcha",
"rate_limit",
"login_wall",
"geo_restricted",
"bot_detected_generic",
null
]
},
"verification": {
"type": "object",
"properties": {
"status": {
"enum": [
"passed",
"failed",
"not_requested"
]
}
},
"required": [
"status"
]
},
"readiness": {
"type": [
"object",
"null"
],
"properties": {
"state": {
"enum": [
"ready",
"not_loaded",
"not_extracted",
"not_served"
]
}
}
},
"agentHints": {
"type": "array",
"items": {
"type": "string"
}
},
"truncated": {
"type": "boolean"
},
"truncatedAt": {
"type": [
"integer",
"null"
]
},
"finalUrl": {
"type": [
"string",
"null"
]
},
"evidenceRecord": {
"type": [
"object",
"null"
]
},
"markdown": {
"type": [
"string",
"null"
]
}
},
"required": [
"status"
]
}🟢map(url, search, sitemap, includeSubdomains, ignoreQueryParameters, ...)
List a site's URLs without reading each page: the start page's links (read over HTTP alone) and the entries of the sitemaps the site declares, within one deadline. URLs stay in scope (the start host and its www twin, the start path's subtree, assets left out, similar URLs folded) and robots.txt applies unless ignoreRobotsTxt is set; what was left out is counted. No page title is fetched. At the deadline the answer is what was found (partial, stoppedBy timeout). Compact by default; debug adds each link's evidence and the sources read. Not for: reading pages, as one page body is read at most.
Input Schema
{
"type": "object",
"properties": {
"url": {
"type": "string",
"description": "http(s) URL of the start page"
},
"search": {
"type": "string",
"minLength": 1,
"maxLength": 200,
"description": "Keep only the URLs in which every word (at most 10) appears, case-insensitively, in the decoded URL or its title. A filter, not a ranking: the order stays the discovery order, and limit counts the matches."
},
"sitemap": {
"type": "string",
"enum": [
"include",
"skip",
"only"
],
"description": "include (default): the start page's links and the sitemaps. skip: no sitemap is read. only: no page is read; the links are the sitemap entries in their listed order (the start URL only when a sitemap lists it)."
},
"includeSubdomains": {
"type": "boolean",
"description": "Admit every host under the start URL's apex (the host with one leading www. removed; no public-suffix list). Default false. Each new host's robots.txt is read, for at most 20 hosts."
},
"ignoreQueryParameters": {
"type": "boolean",
"description": "Fold URLs that differ only in their query string into the first one seen, returned without its query; each merge is counted (refused.collapsed, with samples under debug). Default false."
},
"limit": {
"type": "integer",
"minimum": 1,
"maximum": 5000,
"description": "Links returned at most. Default 5000; a hosted server takes up to 5000. Reaching it is status completed with stoppedBy limit."
},
"timeout": {
"type": "integer",
"minimum": 1000,
"maximum": 60000,
"description": "Milliseconds for the whole map. Default 60000; a hosted server takes up to 60000."
},
"includePaths": {
"type": "array",
"items": {
"type": "string"
},
"description": "Pathname regexes a URL must match."
},
"excludePaths": {
"type": "array",
"items": {
"type": "string"
},
"description": "Pathname regexes that leave a URL out; they win over includePaths."
},
"regexOnFullURL": {
"type": "boolean",
"description": "Match includePaths and excludePaths against the canonical URL instead of its pathname. Default false."
},
"crawlEntireDomain": {
"type": "boolean",
"description": "Admit URLs anywhere on the start host, not only in the start URL's path subtree. Default false."
},
"deduplicateSimilarURLs": {
"type": "boolean",
"description": "Fold /a and /a/, / and /index.html, www and apex, http and https into one URL. Default true."
},
"mode": {
"type": "string",
"enum": [
"standard",
"research"
],
"description": "The declared identity robots.txt, the page and the sitemaps are read under. authed is not offered: a map reads public sitemaps and one public page."
},
"debug": {
"type": "boolean",
"description": "Return the full map response instead of the compact one."
},
"integration": {
"type": "string",
"minLength": 1,
"maxLength": 100,
"pattern": "^[\\x21-\\x7e]+$",
"description": "Your workflow label (printable, no spaces); recorded, not sent to sites."
}
},
"required": [
"url"
],
"additionalProperties": false
}Output Schema
{
"type": "object",
"properties": {
"id": {
"type": "string",
"description": "The map's record id."
},
"status": {
"type": "string",
"enum": [
"completed",
"partial",
"failed"
]
},
"stoppedBy": {
"enum": [
"limit",
"timeout",
null
]
},
"links": {
"type": "array",
"items": {
"type": "object",
"properties": {
"url": {
"type": "string"
},
"title": {
"type": "string"
},
"description": {
"type": "string"
},
"robots": {
"enum": [
"allowed",
"no_robots",
"disallowed",
"unreachable"
],
"description": "The link's robots.txt verdict: every link with debug=true; in the compact answer only on a link robots.txt keeps out, returned under ignoreRobotsTxt."
}
},
"required": [
"url"
]
}
},
"warning": {
"type": "string",
"description": "The warnings' messages, joined."
},
"agentHints": {
"type": "array",
"items": {
"type": "string"
}
},
"counts": {
"type": "object",
"properties": {
"returned": {
"type": "integer"
},
"refused": {
"type": "integer"
}
}
}
},
"required": [
"id",
"status",
"stoppedBy",
"links"
]
}Community
Evidence
Recent observations
verifiedversion not recorded2 tools
auth requiredversion not recorded—
verifiedversion not recorded3 tools