agentcheck
Synthetic checks, nightly regression replay and model-drift alerts for AI agents
我该使用它吗
质量与安全性
A
发现（2）
- HIGHTool poisoning patterns detected
- MEDIUMTool description contains URL to non-standard domain在 agentcheck_get_status 中
基于对工具定义和协议合规性的自动分析。
上下文开销
~3,041token 数（工具定义）
~3.3 KB典型响应大小
对注意力有显著影响（占 128k 上下文窗口的 2.38%）
这是每次将服务器的工具加载到模型上下文窗口时所消耗的大致 token 数。数值越高，可用于其他任务的注意力就越少。
安装
一键安装
将以下内容添加到你的 `claude_desktop_config.json` 文件中：
{
"mcpServers": {
"agentcheck": {
"url": "https://agentwares-agentcheck.vercel.app/api/mcp"
}
}
}远程端点
https://agentwares-agentcheck.vercel.app/api/mcpstreamable-http它能做什么
工具清单
工具（8）
🟢 只读🟡 写入🔴 删除⚪ 未知
🟢agentcheck_get_status(owner, slug)
Current status of a public monitored target: overall state, uptime over 24h/7d/30d, last check time, last nightly scores, open incidents and the badge/status URLs. Use the owner (GitHub login) and target slug from the status page URL https://agentwares-agentcheck.vercel.app/<owner>/<slug>. No API key needed.
输入模式
{
"type": "object",
"properties": {
"owner": {
"type": "string",
"minLength": 1,
"description": "GitHub login of the target's owner, e.g. demo"
},
"slug": {
"type": "string",
"minLength": 1,
"description": "target slug, e.g. demo"
}
},
"required": [
"owner",
"slug"
],
"$schema": "https://json-schema.org/draft/2020-12/schema",
"additionalProperties": false
}🟢agentcheck_get_pricing
Machine-readable pricing for agentcheck: tiers with monthly USD price, target limits, check interval and features, plus per-run add-ons. Same data as /pricing.json. No API key needed.
输入模式
{
"type": "object",
"properties": {},
"$schema": "https://json-schema.org/draft/2020-12/schema",
"additionalProperties": false
}🟡agentcheck_create_target(name, slug, kind, url, packageSpec, ...)
Enroll something to monitor: an http endpoint (JSON or OpenAI-style chat), a remote MCP server (Streamable HTTP url) or an A2A agent (origin with /.well-known/agent-card.json). Pass `checks` to create checks in the same call (POST /api/v1/probe proposes three). Returns the target id, the public status page, the badge SVG URL and a README snippet. The first check runs on the next minute tick; call agentcheck_run_now to run immediately. Free tier: 1 target, hourly; Starter+: 5-minute checks. Requires an API key.
输入模式
{
"type": "object",
"properties": {
"name": {
"description": "display name; defaults to the host",
"type": "string",
"minLength": 1,
"maxLength": 80
},
"slug": {
"description": "URL slug for /<owner>/<slug>; defaults to a slug of the name",
"type": "string",
"pattern": "^[a-z0-9][a-z0-9-]{0,62}$"
},
"kind": {
"type": "string",
"enum": [
"http",
"mcp",
"a2a"
],
"description": "http (JSON or chat endpoint), mcp (Streamable HTTP url, or npx/uvx package spec), a2a (agent card)"
},
"url": {
"description": "endpoint URL (http/mcp) or the agent's origin (a2a)",
"type": "string",
"format": "uri"
},
"packageSpec": {
"description": "mcp only: `npx @org/server` / `uvx server` — runs on the mcpcheck runner, not the minute checks",
"type": "string"
},
"agentCardUrl": {
"description": "a2a: explicit agent card URL when it is not at /.well-known/agent-card.json",
"type": "string",
"format": "uri"
},
"format": {
"description": "http: openai_chat (POST /chat/completions body), json (raw POST), get (plain fetch)",
"type": "string",
"enum": [
"openai_chat",
"json",
"get"
]
},
"headers": {
"description": "extra request headers",
"type": "object",
"propertyNames": {
"type": "string"
},
"additionalProperties": {
"type": "string"
}
},
"auth": {
"description": "credential the prober sends to your endpoint: {type:'bearer',token} or {type:'header',name,value}. Stored encrypted and never returned or logged.",
"anyOf": [
{
"type": "object",
"properties": {
"type": {
"type": "string",
"const": "bearer"
},
"token": {
"type": "string"
}
},
"required": [
"type",
"token"
]
},
{
"type": "object",
"properties": {
"type": {
"type": "string",
"const": "header"
},
"name": {
"type": "string"
},
"value": {
"type": "string"
}
},
"required": [
"type",
"name",
"value"
]
}
]
},
"modelVar": {
"description": "the model your agent runs on (e.g. claude-sonnet-5); enables model-drift re-runs",
"type": "string"
},
"modelHeader": {
"description": "request header your endpoint accepts to override the model (drift re-runs try the new model)",
"type": "string"
},
"corpusUrl": {
"description": "RAG targets: public corpus URL for the nightly groundedness scorer (Pro)",
"type": "string",
"format": "uri"
},
"alerts": {
"description": "where to send an incident when a check starts failing. Any combination; omit it and incidents are visible only on the status page.",
"type": "object",
"properties": {
"emails": {
"type": "array",
"items": {
"type": "string",
"format": "email",
"pattern": "^(?!\\.)(?!.*\\.\\.)([A-Za-z0-9_'+\\-\\.]*)[A-Za-z0-9_+-]@([A-Za-z0-9][A-Za-z0-9\\-]*\\.)+[A-Za-z]{2,}$"
}
},
"slackWebhookUrl": {
"type": "string",
"format": "uri"
},
"discordWebhookUrl": {
"type": "string",
"format": "uri"
}
}
},
"isPublic": {
"default": true,
"description": "public status page + badge (default true)",
"type": "boolean"
},
"checks": {
"description": "checks to create right away (the probe proposes three)",
"maxItems": 20,
"type": "array",
"items": {
"type": "object",
"properties": {
"name": {
"type": "string",
"minLength": 1,
"maxLength": 120,
"description": "short name shown on the status page"
},
"kind": {
"type": "string",
"enum": [
"http",
"mcp_tools_list",
"mcp_tool_call",
"a2a_task",
"custom"
],
"description": "what to run: http (GET path or POST prompt), mcp_tools_list, mcp_tool_call (tool + args), a2a_task (message)"
},
"input": {
"default": {},
"description": "http: { path?, method?, body?, prompt? } · mcp_tool_call: { tool, args } · a2a_task: { message }",
"type": "object",
"propertyNames": {
"type": "string"
},
"additionalProperties": {}
},
"golden": {
"type": "object",
"properties": {
"kind": {
"type": "string",
"enum": [
"exact",
"contains",
"regex",
"json_schema",
"rubric",
"baseline"
],
"description": "how the answer is judged; exact/contains/regex/json_schema cost nothing, rubric/baseline use the LLM judge"
},
"value": {
"description": "exact / contains: the expected text",
"type": "string"
},
"pattern": {
"description": "regex: pattern source without slashes",
"type": "string"
},
"caseInsensitive": {
"type": "boolean"
},
"schema": {
"description": "json_schema: JSON Schema (draft 2020-12 subset)",
"type": "object",
"propertyNames": {
"type": "string"
},
"additionalProperties": {}
},
"rubric": {
"description": "rubric: what a passing answer must do, in plain language",
"type": "string"
},
"expected": {
"description": "rubric: a reference answer, when there is one",
"type": "string"
},
"tools": {
"description": "expected tool-call sequence (replay checks)",
"type": "array",
"items": {
"type": "string"
}
}
},
"required": [
"kind"
],
"description": "how the answer is judged. exact / contains / regex / json_schema are free and deterministic; rubric and baseline call the LLM judge, and baseline compares against the last known-good answer."
},
"intervalSec": {
"default": 3600,
"description": "seconds between runs; the tier's interval is the floor (Free hourly, Starter+ 5 min)",
"type": "integer",
"minimum": 60,
"maximum": 604800
}
},
"required": [
"name",
"kind",
"golden"
]
}
}
},
"required": [
"kind"
],
"$schema": "https://json-schema.org/draft/2020-12/schema",
"additionalProperties": false
}🟡agentcheck_add_check(name, kind, input, golden, intervalSec, ...)
Add a check to one of your targets. A check runs an input (http path/prompt, mcp tool call, a2a message) on a schedule and judges the answer with a golden: exact, contains, regex and json_schema cost nothing; rubric and baseline use the LLM judge (baseline = same outcome as the last known-good answer). Returns the check id. Requires an API key.
输入模式
{
"type": "object",
"properties": {
"name": {
"type": "string",
"minLength": 1,
"maxLength": 120,
"description": "short name shown on the status page"
},
"kind": {
"type": "string",
"enum": [
"http",
"mcp_tools_list",
"mcp_tool_call",
"a2a_task",
"custom"
],
"description": "what to run: http (GET path or POST prompt), mcp_tools_list, mcp_tool_call (tool + args), a2a_task (message)"
},
"input": {
"default": {},
"description": "http: { path?, method?, body?, prompt? } · mcp_tool_call: { tool, args } · a2a_task: { message }",
"type": "object",
"propertyNames": {
"type": "string"
},
"additionalProperties": {}
},
"golden": {
"type": "object",
"properties": {
"kind": {
"type": "string",
"enum": [
"exact",
"contains",
"regex",
"json_schema",
"rubric",
"baseline"
],
"description": "how the answer is judged; exact/contains/regex/json_schema cost nothing, rubric/baseline use the LLM judge"
},
"value": {
"description": "exact / contains: the expected text",
"type": "string"
},
"pattern": {
"description": "regex: pattern source without slashes",
"type": "string"
},
"caseInsensitive": {
"type": "boolean"
},
"schema": {
"description": "json_schema: JSON Schema (draft 2020-12 subset)",
"type": "object",
"propertyNames": {
"type": "string"
},
"additionalProperties": {}
},
"rubric": {
"description": "rubric: what a passing answer must do, in plain language",
"type": "string"
},
"expected": {
"description": "rubric: a reference answer, when there is one",
"type": "string"
},
"tools": {
"description": "expected tool-call sequence (replay checks)",
"type": "array",
"items": {
"type": "string"
}
}
},
"required": [
"kind"
],
"description": "how the answer is judged. exact / contains / regex / json_schema are free and deterministic; rubric and baseline call the LLM judge, and baseline compares against the last known-good answer."
},
"intervalSec": {
"default": 3600,
"description": "seconds between runs; the tier's interval is the floor (Free hourly, Starter+ 5 min)",
"type": "integer",
"minimum": 60,
"maximum": 604800
},
"targetId": {
"type": "string",
"minLength": 1,
"description": "id from agentcheck_create_target or GET /api/v1/targets"
}
},
"required": [
"name",
"kind",
"golden",
"targetId"
],
"$schema": "https://json-schema.org/draft/2020-12/schema",
"additionalProperties": false
}🟢agentcheck_run_now(targetId)
Run every check of one of your targets immediately (outside the schedule) and return pass/fail per check with latency, judge cost and any incident opened or closed. Use it right after enrolling, or to confirm a fix. Requires an API key.
输入模式
{
"type": "object",
"properties": {
"targetId": {
"type": "string",
"minLength": 1,
"description": "target id"
}
},
"required": [
"targetId"
],
"$schema": "https://json-schema.org/draft/2020-12/schema",
"additionalProperties": false
}⚪agentcheck_record(targetId, name, input, output, toolCalls, ...)
Record one production interaction with your agent (the prompt or messages, the final answer, the tools it called, optionally the model) as a trace on a target. Call it from your agent or from a proxy in front of it after each task; promote a good trace with agentcheck_promote_trace to replay it nightly and catch regressions. Requires an API key.
输入模式
{
"type": "object",
"properties": {
"targetId": {
"type": "string",
"minLength": 1,
"description": "id from agentcheck_create_target or GET /api/v1/targets"
},
"name": {
"description": "short label, e.g. 'refund for order A-1029'",
"type": "string",
"maxLength": 120
},
"input": {
"anyOf": [
{
"type": "string"
},
{
"type": "array",
"items": {
"type": "object",
"properties": {
"role": {
"type": "string"
},
"content": {
"type": "string"
}
},
"required": [
"role",
"content"
]
}
},
{
"type": "object",
"propertyNames": {
"type": "string"
},
"additionalProperties": {}
}
],
"description": "the prompt, the messages array, or an object with prompt/messages"
},
"output": {
"type": "string",
"minLength": 1,
"description": "the agent's final answer"
},
"toolCalls": {
"description": "tool calls in order",
"type": "array",
"items": {
"type": "object",
"properties": {
"name": {
"type": "string"
},
"args": {}
},
"required": [
"name"
]
}
},
"model": {
"description": "the model that produced this answer, so drift re-runs can compare across models",
"type": "string"
}
},
"required": [
"targetId",
"input",
"output"
],
"$schema": "https://json-schema.org/draft/2020-12/schema",
"additionalProperties": false
}🟢agentcheck_promote_trace(traceId)
Turn an imported or recorded trace into a replayable check whose golden is the recorded outcome (tool sequence + final-answer rubric). The check runs daily and in the nightly replay (Pro). Returns the check. Requires an API key.
输入模式
{
"type": "object",
"properties": {
"traceId": {
"type": "string",
"minLength": 1,
"description": "trace id from agentcheck_record or POST /api/v1/targets/{id}/traces"
}
},
"required": [
"traceId"
],
"$schema": "https://json-schema.org/draft/2020-12/schema",
"additionalProperties": false
}🟢agentcheck_list_incidents(targetId)
Open and recent incidents across your targets (or one target): when they opened/closed, the failing check and the cause. Requires an API key.
输入模式
{
"type": "object",
"properties": {
"targetId": {
"description": "narrow to one target; omit for incidents across every target on the account",
"type": "string"
}
},
"$schema": "https://json-schema.org/draft/2020-12/schema",
"additionalProperties": false
}社区
证据
最近观测
已验证未记录版本8 个工具
已验证未记录版本8 个工具
已验证未记录版本8 个工具