agentcheck
Synthetic checks, nightly regression replay and model-drift alerts for AI agents
Sollte ich dies verwenden
Qualität und Sicherheit
Befunde (2)
- HIGHTool poisoning patterns detected
- MEDIUMTool description contains URL to non-standard domainin agentcheck_get_status
Basierend auf einer automatisierten Analyse der Tool-Definitionen und der Einhaltung des Protokolls.
Kontextkosten
Dies ist die ungefähre Anzahl der Tokens, die jedes Mal verbraucht werden, wenn die Tools des Servers in den Kontext eines Modells geladen werden. Höhere Werte verringern die Aufmerksamkeit, die für andere Aufgaben verfügbar ist.
Installieren
Installation mit einem Klick
Fügen Sie dies Ihrer Datei `claude_desktop_config.json` hinzu:
{
"mcpServers": {
"agentcheck": {
"url": "https://agentwares-agentcheck.vercel.app/api/mcp"
}
}
}Remote-Endpunkte
https://agentwares-agentcheck.vercel.app/api/mcpstreamable-httpWas es kann
Tool-Inventar
Tools (8)
🟢agentcheck_get_status(owner, slug)
Current status of a public monitored target: overall state, uptime over 24h/7d/30d, last check time, last nightly scores, open incidents and the badge/status URLs. Use the owner (GitHub login) and target slug from the status page URL https://agentwares-agentcheck.vercel.app/<owner>/<slug>. No API key needed.
Eingabe-Schema
{
"type": "object",
"properties": {
"owner": {
"type": "string",
"minLength": 1,
"description": "GitHub login of the target's owner, e.g. demo"
},
"slug": {
"type": "string",
"minLength": 1,
"description": "target slug, e.g. demo"
}
},
"required": [
"owner",
"slug"
],
"$schema": "https://json-schema.org/draft/2020-12/schema",
"additionalProperties": false
}🟢agentcheck_get_pricing
Machine-readable pricing for agentcheck: tiers with monthly USD price, target limits, check interval and features, plus per-run add-ons. Same data as /pricing.json. No API key needed.
Eingabe-Schema
{
"type": "object",
"properties": {},
"$schema": "https://json-schema.org/draft/2020-12/schema",
"additionalProperties": false
}🟡agentcheck_create_target(name, slug, kind, url, packageSpec, ...)
Enroll something to monitor: an http endpoint (JSON or OpenAI-style chat), a remote MCP server (Streamable HTTP url) or an A2A agent (origin with /.well-known/agent-card.json). Pass `checks` to create checks in the same call (POST /api/v1/probe proposes three). Returns the target id, the public status page, the badge SVG URL and a README snippet. The first check runs on the next minute tick; call agentcheck_run_now to run immediately. Free tier: 1 target, hourly; Starter+: 5-minute checks. Requires an API key.
Eingabe-Schema
{
"type": "object",
"properties": {
"name": {
"description": "display name; defaults to the host",
"type": "string",
"minLength": 1,
"maxLength": 80
},
"slug": {
"description": "URL slug for /<owner>/<slug>; defaults to a slug of the name",
"type": "string",
"pattern": "^[a-z0-9][a-z0-9-]{0,62}$"
},
"kind": {
"type": "string",
"enum": [
"http",
"mcp",
"a2a"
],
"description": "http (JSON or chat endpoint), mcp (Streamable HTTP url, or npx/uvx package spec), a2a (agent card)"
},
"url": {
"description": "endpoint URL (http/mcp) or the agent's origin (a2a)",
"type": "string",
"format": "uri"
},
"packageSpec": {
"description": "mcp only: `npx @org/server` / `uvx server` — runs on the mcpcheck runner, not the minute checks",
"type": "string"
},
"agentCardUrl": {
"description": "a2a: explicit agent card URL when it is not at /.well-known/agent-card.json",
"type": "string",
"format": "uri"
},
"format": {
"description": "http: openai_chat (POST /chat/completions body), json (raw POST), get (plain fetch)",
"type": "string",
"enum": [
"openai_chat",
"json",
"get"
]
},
"headers": {
"description": "extra request headers",
"type": "object",
"propertyNames": {
"type": "string"
},
"additionalProperties": {
"type": "string"
}
},
"auth": {
"description": "credential the prober sends to your endpoint: {type:'bearer',token} or {type:'header',name,value}. Stored encrypted and never returned or logged.",
"anyOf": [
{
"type": "object",
"properties": {
"type": {
"type": "string",
"const": "bearer"
},
"token": {
"type": "string"
}
},
"required": [
"type",
"token"
]
},
{
"type": "object",
"properties": {
"type": {
"type": "string",
"const": "header"
},
"name": {
"type": "string"
},
"value": {
"type": "string"
}
},
"required": [
"type",
"name",
"value"
]
}
]
},
"modelVar": {
"description": "the model your agent runs on (e.g. claude-sonnet-5); enables model-drift re-runs",
"type": "string"
},
"modelHeader": {
"description": "request header your endpoint accepts to override the model (drift re-runs try the new model)",
"type": "string"
},
"corpusUrl": {
"description": "RAG targets: public corpus URL for the nightly groundedness scorer (Pro)",
"type": "string",
"format": "uri"
},
"alerts": {
"description": "where to send an incident when a check starts failing. Any combination; omit it and incidents are visible only on the status page.",
"type": "object",
"properties": {
"emails": {
"type": "array",
"items": {
"type": "string",
"format": "email",
"pattern": "^(?!\\.)(?!.*\\.\\.)([A-Za-z0-9_'+\\-\\.]*)[A-Za-z0-9_+-]@([A-Za-z0-9][A-Za-z0-9\\-]*\\.)+[A-Za-z]{2,}$"
}
},
"slackWebhookUrl": {
"type": "string",
"format": "uri"
},
"discordWebhookUrl": {
"type": "string",
"format": "uri"
}
}
},
"isPublic": {
"default": true,
"description": "public status page + badge (default true)",
"type": "boolean"
},
"checks": {
"description": "checks to create right away (the probe proposes three)",
"maxItems": 20,
"type": "array",
"items": {
"type": "object",
"properties": {
"name": {
"type": "string",
"minLength": 1,
"maxLength": 120,
"description": "short name shown on the status page"
},
"kind": {
"type": "string",
"enum": [
"http",
"mcp_tools_list",
"mcp_tool_call",
"a2a_task",
"custom"
],
"description": "what to run: http (GET path or POST prompt), mcp_tools_list, mcp_tool_call (tool + args), a2a_task (message)"
},
"input": {
"default": {},
"description": "http: { path?, method?, body?, prompt? } · mcp_tool_call: { tool, args } · a2a_task: { message }",
"type": "object",
"propertyNames": {
"type": "string"
},
"additionalProperties": {}
},
"golden": {
"type": "object",
"properties": {
"kind": {
"type": "string",
"enum": [
"exact",
"contains",
"regex",
"json_schema",
"rubric",
"baseline"
],
"description": "how the answer is judged; exact/contains/regex/json_schema cost nothing, rubric/baseline use the LLM judge"
},
"value": {
"description": "exact / contains: the expected text",
"type": "string"
},
"pattern": {
"description": "regex: pattern source without slashes",
"type": "string"
},
"caseInsensitive": {
"type": "boolean"
},
"schema": {
"description": "json_schema: JSON Schema (draft 2020-12 subset)",
"type": "object",
"propertyNames": {
"type": "string"
},
"additionalProperties": {}
},
"rubric": {
"description": "rubric: what a passing answer must do, in plain language",
"type": "string"
},
"expected": {
"description": "rubric: a reference answer, when there is one",
"type": "string"
},
"tools": {
"description": "expected tool-call sequence (replay checks)",
"type": "array",
"items": {
"type": "string"
}
}
},
"required": [
"kind"
],
"description": "how the answer is judged. exact / contains / regex / json_schema are free and deterministic; rubric and baseline call the LLM judge, and baseline compares against the last known-good answer."
},
"intervalSec": {
"default": 3600,
"description": "seconds between runs; the tier's interval is the floor (Free hourly, Starter+ 5 min)",
"type": "integer",
"minimum": 60,
"maximum": 604800
}
},
"required": [
"name",
"kind",
"golden"
]
}
}
},
"required": [
"kind"
],
"$schema": "https://json-schema.org/draft/2020-12/schema",
"additionalProperties": false
}🟡agentcheck_add_check(name, kind, input, golden, intervalSec, ...)
Add a check to one of your targets. A check runs an input (http path/prompt, mcp tool call, a2a message) on a schedule and judges the answer with a golden: exact, contains, regex and json_schema cost nothing; rubric and baseline use the LLM judge (baseline = same outcome as the last known-good answer). Returns the check id. Requires an API key.
Eingabe-Schema
{
"type": "object",
"properties": {
"name": {
"type": "string",
"minLength": 1,
"maxLength": 120,
"description": "short name shown on the status page"
},
"kind": {
"type": "string",
"enum": [
"http",
"mcp_tools_list",
"mcp_tool_call",
"a2a_task",
"custom"
],
"description": "what to run: http (GET path or POST prompt), mcp_tools_list, mcp_tool_call (tool + args), a2a_task (message)"
},
"input": {
"default": {},
"description": "http: { path?, method?, body?, prompt? } · mcp_tool_call: { tool, args } · a2a_task: { message }",
"type": "object",
"propertyNames": {
"type": "string"
},
"additionalProperties": {}
},
"golden": {
"type": "object",
"properties": {
"kind": {
"type": "string",
"enum": [
"exact",
"contains",
"regex",
"json_schema",
"rubric",
"baseline"
],
"description": "how the answer is judged; exact/contains/regex/json_schema cost nothing, rubric/baseline use the LLM judge"
},
"value": {
"description": "exact / contains: the expected text",
"type": "string"
},
"pattern": {
"description": "regex: pattern source without slashes",
"type": "string"
},
"caseInsensitive": {
"type": "boolean"
},
"schema": {
"description": "json_schema: JSON Schema (draft 2020-12 subset)",
"type": "object",
"propertyNames": {
"type": "string"
},
"additionalProperties": {}
},
"rubric": {
"description": "rubric: what a passing answer must do, in plain language",
"type": "string"
},
"expected": {
"description": "rubric: a reference answer, when there is one",
"type": "string"
},
"tools": {
"description": "expected tool-call sequence (replay checks)",
"type": "array",
"items": {
"type": "string"
}
}
},
"required": [
"kind"
],
"description": "how the answer is judged. exact / contains / regex / json_schema are free and deterministic; rubric and baseline call the LLM judge, and baseline compares against the last known-good answer."
},
"intervalSec": {
"default": 3600,
"description": "seconds between runs; the tier's interval is the floor (Free hourly, Starter+ 5 min)",
"type": "integer",
"minimum": 60,
"maximum": 604800
},
"targetId": {
"type": "string",
"minLength": 1,
"description": "id from agentcheck_create_target or GET /api/v1/targets"
}
},
"required": [
"name",
"kind",
"golden",
"targetId"
],
"$schema": "https://json-schema.org/draft/2020-12/schema",
"additionalProperties": false
}🟢agentcheck_run_now(targetId)
Run every check of one of your targets immediately (outside the schedule) and return pass/fail per check with latency, judge cost and any incident opened or closed. Use it right after enrolling, or to confirm a fix. Requires an API key.
Eingabe-Schema
{
"type": "object",
"properties": {
"targetId": {
"type": "string",
"minLength": 1,
"description": "target id"
}
},
"required": [
"targetId"
],
"$schema": "https://json-schema.org/draft/2020-12/schema",
"additionalProperties": false
}⚪agentcheck_record(targetId, name, input, output, toolCalls, ...)
Record one production interaction with your agent (the prompt or messages, the final answer, the tools it called, optionally the model) as a trace on a target. Call it from your agent or from a proxy in front of it after each task; promote a good trace with agentcheck_promote_trace to replay it nightly and catch regressions. Requires an API key.
Eingabe-Schema
{
"type": "object",
"properties": {
"targetId": {
"type": "string",
"minLength": 1,
"description": "id from agentcheck_create_target or GET /api/v1/targets"
},
"name": {
"description": "short label, e.g. 'refund for order A-1029'",
"type": "string",
"maxLength": 120
},
"input": {
"anyOf": [
{
"type": "string"
},
{
"type": "array",
"items": {
"type": "object",
"properties": {
"role": {
"type": "string"
},
"content": {
"type": "string"
}
},
"required": [
"role",
"content"
]
}
},
{
"type": "object",
"propertyNames": {
"type": "string"
},
"additionalProperties": {}
}
],
"description": "the prompt, the messages array, or an object with prompt/messages"
},
"output": {
"type": "string",
"minLength": 1,
"description": "the agent's final answer"
},
"toolCalls": {
"description": "tool calls in order",
"type": "array",
"items": {
"type": "object",
"properties": {
"name": {
"type": "string"
},
"args": {}
},
"required": [
"name"
]
}
},
"model": {
"description": "the model that produced this answer, so drift re-runs can compare across models",
"type": "string"
}
},
"required": [
"targetId",
"input",
"output"
],
"$schema": "https://json-schema.org/draft/2020-12/schema",
"additionalProperties": false
}🟢agentcheck_promote_trace(traceId)
Turn an imported or recorded trace into a replayable check whose golden is the recorded outcome (tool sequence + final-answer rubric). The check runs daily and in the nightly replay (Pro). Returns the check. Requires an API key.
Eingabe-Schema
{
"type": "object",
"properties": {
"traceId": {
"type": "string",
"minLength": 1,
"description": "trace id from agentcheck_record or POST /api/v1/targets/{id}/traces"
}
},
"required": [
"traceId"
],
"$schema": "https://json-schema.org/draft/2020-12/schema",
"additionalProperties": false
}🟢agentcheck_list_incidents(targetId)
Open and recent incidents across your targets (or one target): when they opened/closed, the failing check and the cause. Requires an API key.
Eingabe-Schema
{
"type": "object",
"properties": {
"targetId": {
"description": "narrow to one target; omit for incidents across every target on the account",
"type": "string"
}
},
"$schema": "https://json-schema.org/draft/2020-12/schema",
"additionalProperties": false
}Community
Nachweis