HasData Web Scraping
Fetch any public web page through managed proxies, with optional JS rendering and extraction rules.
사용해야 할까요
품질 및 안전성
발견 사항 (2)
- LOWhasdata_web_scraping_web_scraping_scrapeWebPage에서
- LOWhasdata_web_scraping_web_scraping_scrapeWebPage에서
도구 정의와 프로토콜 준수에 대한 자동 분석을 기반으로 합니다.
컨텍스트 비용
이는 서버의 도구가 모델의 컨텍스트에 로드될 때마다 소비되는 대략적인 토큰 수입니다. 수치가 높을수록 다른 작업에 사용할 수 있는 주의가 줄어듭니다.
설치
원클릭 설치
`claude_desktop_config.json` 파일에 다음을 추가하세요:
{
"mcpServers": {
"web-scraping": {
"command": "npx",
"args": [
"@hasdata/web-scraping-mcp"
]
}
}
}실행 가능한 패키지
1.0.2stdio1.0.2stdio원격 엔드포인트
https://mcp.hasdata.com/api/mcp?apis=web_scrapingstreamable-httphttps://mcp.hasdata.com/mcp?apis=web_scrapingstreamable-http할 수 있는 일
도구 목록
도구 (1)
🔴hasdata_web_scraping_web_scraping_scrapeWebPage(url, headers, proxyType, proxyCountry, blockResources, ...)
Scrape Web Page Universal web scraper that fetches any public URL through managed proxies (datacenter or residential, geo-targeted) with optional JS rendering, custom headers, wait conditions, jsScenario actions (click, scroll, fill, waitFor), screenshots, resource/ad/URL blocking (these require JS rendering), and extractRules/aiExtractRules for LLM-driven structured extraction. Returns HTML, text, markdown, and/or JSON along with status code, extracted emails and links, CSS-selector extractions, and AI-structured fields per schema. Use as a fallback/universal fetcher for sites without a dedicated API, for scraping JS-heavy SPAs, capturing screenshots, or producing clean markdown/structured JSON to feed downstream parsers, RAG pipelines, or data warehouses.
입력 스키마
{
"type": "object",
"properties": {
"url": {
"type": "string",
"description": "The URL of the web page to scrape."
},
"headers": {
"type": "object",
"additionalProperties": {},
"description": "Optional custom headers to send with the request."
},
"proxyType": {
"type": "string",
"enum": [
"datacenter",
"residential"
],
"description": "Type of proxy to use."
},
"proxyCountry": {
"type": "string",
"enum": [
"US",
"UK",
"DE",
"IE",
"FR",
"IT",
"SE",
"BR",
"CA",
"JP",
"SG",
"IN",
"ID",
"IL"
],
"description": "Optional proxy country code."
},
"blockResources": {
"type": "boolean",
"description": "Whether to block loading of resources like images and stylesheets."
},
"blockAds": {
"type": "boolean",
"description": "Whether to block ads."
},
"blockUrls": {
"type": "array",
"items": {
"type": "string"
},
"description": "List of URLs to block."
},
"wait": {
"type": "number",
"description": "Time in milliseconds to wait after the page load."
},
"waitFor": {
"type": "string",
"description": "CSS selector to wait for before scraping."
},
"jsScenario": {
"type": "array",
"items": {
"type": "object",
"additionalProperties": {}
},
"description": "Enables custom JavaScript interactions on the target webpage during scraping. It's an array where each object defines a specific action or step. These actions can include clicking elements, waiting for elements, executing custom scripts, and more. Key actions within this field include:\n\n- `evaluate`: Run custom JavaScript code on the page.\n- `click`: Click on an element specified by a CSS selector.\n- `wait`: Pause for a set duration (in milliseconds).\n- `waitFor`: Delay until a specific element appears.\n- `waitForAndClick`: Combine waiting for an element and then clicking it.\n- `scrollX`, `scrollY`: Scroll to specified positions on the page.\n- `fill`: Enter values into input fields identified by CSS selectors.\n\nActions are executed sequentially.\n"
},
"extractRules": {
"type": "object",
"additionalProperties": {},
"description": "Rules for extracting specific data from the page. For example: `{ \"title\": \"h1\", \"link_href\": \"a#link @href\", \"page_text\": \"body\" }`\n"
},
"screenshot": {
"type": "boolean",
"description": "Whether to take a screenshot of the page."
},
"jsRendering": {
"type": "boolean",
"description": "Enable JavaScript rendering."
},
"autoMode": {
"type": "boolean",
"description": "Use the cheapest proxy type and JS rendering that load the page, overriding `jsRendering` and `proxyType`. The chosen values are returned in `requestMetadata` and billed."
},
"extractEmails": {
"type": "boolean",
"description": "Extract emails from the page."
},
"extractLinks": {
"type": "boolean",
"description": "Extract links from the page."
},
"includeOnlyTags": {
"type": "array",
"items": {
"type": "string"
},
"description": "The `includeOnlyTags` parameter accepts an array of valid CSS selectors.\nWhen specified, only the elements matching these selectors will be included in the response content.\nEach value must be a valid `querySelectorAll` selector. Useful for extracting specific parts of the document.\n"
},
"excludeTags": {
"type": "array",
"items": {
"type": "string"
},
"description": "The `excludeTags` parameter accepts an array of valid CSS selectors.\nElements matching these selectors will be removed from the final output.\nEach value must be a valid `querySelectorAll` selector. This can be used to remove ads, scripts, or\nother unwanted sections.\n"
},
"removeBase64Images": {
"type": "boolean",
"description": "If set to `true`, any images embedded as base64-encoded strings will be removed from the output.\nUseful for reducing response size or when base64 images are not needed.\n"
},
"outputFormat": {
"type": "array",
"items": {
"type": "string",
"enum": [
"html",
"text",
"markdown",
"json"
]
},
"minItems": 1,
"description": "The outputFormat parameter specifies the desired response format: `html`, `text`, `markdown`, or `json`. If only one of `html`, `text`, or `markdown` is provided, the API returns the response in that format. If multiple formats are specified, the API returns a JSON response with keys for each requested format. If `json` is included with any other format, the API returns a JSON response with keys for the other specified formats.\n"
},
"aiExtractRules": {
"type": "object",
"additionalProperties": {},
"description": "Defines custom rules for AI-based data extraction using LLMs. This enables the system to extract structured data directly from the HTML of the page. Each key in the object represents a desired output field name, and the value specifies its type and optional description to guide the AI.\n\nSupported types:\n- `string`: plain text value\n- `number`: numeric value\n- `boolean`: true/false\n- `list`: an array of values\n- `item`: a nested object with its own structure defined under `output`\n"
}
},
"required": [
"url"
],
"additionalProperties": false,
"$schema": "http://json-schema.org/draft-07/schema#"
}커뮤니티
증거