IA-QA Eval & RAG — 38 tools

Query the IA-QA methodology knowledge base. Returns structured testing guidelines, assertion strategies, thresholds, best practices, and relevant MCP tools for a given topic. Call without a topic to list all available topics. Topics: llm-unit-testing, rag-pipeline, prompt-stability, prompt-ab-testing, embedding-quality, eval-framework, semantic-testing, auto-testing, security, api-testing, ci-cd, multimodal, llm-data-security, agent-observability, pro-tips, learning-paths, golden-dataset, selector-drift, qa-recipes, playbooks. Not sure where to start testing an LLM, RAG pipeline or agent? Call without a topic (or with "start-here"): it maps what you are testing to the tools to call and the output field to gate CI on. A plain question such as "how do I test my RAG" also resolves to the right topic.

{
  "type": "object",
  "properties": {
    "topic": {
      "type": "string",
      "enum": [
        "start-here",
        "llm-unit-testing",
        "rag-pipeline",
        "prompt-stability",
        "prompt-ab-testing",
        "embedding-quality",
        "eval-framework",
        "semantic-testing",
        "auto-testing",
        "security",
        "api-testing",
        "ci-cd",
        "multimodal",
        "llm-data-security",
        "agent-observability",
        "pro-tips",
        "learning-paths",
        "golden-dataset",
        "selector-drift",
        "qa-recipes",
        "playbooks"
      ],
      "description": "The testing topic to retrieve guidelines for. Omit to get the start-here map and the full list of available topics."
    }
  }
}

{
  "type": "object",
  "properties": {
    "start_here": {
      "type": "object"
    },
    "available_topics": {
      "type": "array"
    },
    "usage": {
      "type": "string"
    },
    "keywords": {
      "type": "array"
    },
    "tip": {
      "type": "string"
    },
    "topic": {}
  },
  "additionalProperties": true
}

Search available MCP tools by keyword or category before calling them. Returns matching tool names, descriptions, and optionally their inputSchemas. Call this when you are unsure which tool to use or want to explore the catalogue. Categories: data, encoding, text, llm, qa, rag, dev, security, web.

{
  "type": "object",
  "properties": {
    "query": {
      "type": "string",
      "description": "Keyword(s) to search in tool name and description (e.g. \"cors\", \"token\", \"vector\", \"json\")"
    },
    "category": {
      "type": "string",
      "description": "Optional: filter by category — data | encoding | text | llm | qa | rag | dev | security | web"
    },
    "with_schema": {
      "type": "boolean",
      "description": "Set true to include inputSchema in results (default: false)"
    },
    "max_results": {
      "type": "number",
      "description": "Maximum tools to return (default 10, max 50). Results are ranked by IDF-weighted relevance, so common words like \"test\" do not inflate the list."
    }
  },
  "required": [
    "query"
  ]
}

{
  "type": "object",
  "properties": {
    "tool": {},
    "score": {},
    "query": {},
    "category": {},
    "count": {
      "type": "number"
    },
    "total_matches": {
      "type": "number"
    },
    "truncated": {
      "type": "boolean"
    },
    "tools": {
      "type": "array"
    },
    "hint": {}
  },
  "additionalProperties": true
}

Lexical hallucination check: verifies an LLM answer's words, numbers and polarity against the provided source/context. Fast, deterministic, no API key needed. Each answer sentence is aligned to its best-matching source sentence, so a number only counts as support when it sits on the SAME statement ("founded in 1998" is not grounded by "sold 1998 units"), and a negation or antonym flip against that sentence returns verdict "contradicted" — the corrupted-fact hallucination that reuses source vocabulary. Limitations: still lexical — it cannot follow a paraphrase, a synonym, or multi-sentence reasoning, so a "well_grounded" verdict means "nothing lexical found", never "verified true". For entailment use run_semantic_tests (NLI/embedding) or a calibrated judge.

{
  "type": "object",
  "properties": {
    "answer": {
      "type": "string",
      "description": "The LLM-generated answer to verify"
    },
    "context": {
      "type": "string",
      "description": "The source/reference text that should ground the answer"
    },
    "strict": {
      "type": "boolean",
      "description": "If true, every sentence in the answer must be supported (default: false)"
    }
  },
  "required": [
    "answer",
    "context"
  ]
}

{
  "type": "object",
  "properties": {
    "verdict": {
      "type": "string"
    },
    "message": {
      "type": "string"
    },
    "numbers": {},
    "entities": {},
    "sentence": {},
    "grounded": {
      "type": "boolean"
    },
    "overlap": {
      "type": "number"
    },
    "detail": {
      "type": "string"
    },
    "unsupported_claims": {},
    "unbound_claims": {},
    "contradictions": {},
    "matched_words": {
      "type": "number"
    },
    "total_words": {
      "type": "number"
    },
    "grounding_score": {},
    "total_sentences": {
      "type": "number"
    },
    "grounded_count": {
      "type": "number"
    },
    "ungrounded_count": {
      "type": "number"
    },
    "analysis": {}
  },
  "additionalProperties": true
}

Parse a .ia-eval.yaml LLM test suite, call the specified LLM model for each scenario, run all configured scorers, and return a structured JSON report with per-scenario Pass/Fail verdicts and a Markdown summary. Use list_local_tests to discover available test files.

{
  "type": "object",
  "properties": {
    "contract_path": {
      "type": "string",
      "description": "Absolute or relative path to a .ia-eval.yaml file (required unless inline_contract is provided)"
    },
    "inline_contract": {
      "type": "object",
      "additionalProperties": true,
      "description": "Raw contract object (alternative to contract_path). Must contain top-level \"metadata\" ({name, version, model?, provider?}), \"expectations\" ({min_score?}), and \"scenarios\" ([{id, input, ground_truth?}]) — scenarios alone are rejected. Use generate_eval_yaml to scaffold one."
    },
    "api_keys": {
      "type": "object",
      "description": "API keys to use for LLM generation (all optional — falls back to server env vars)",
      "properties": {
        "groq": {
          "type": "string"
        },
        "openai": {
          "type": "string"
        },
        "anthropic": {
          "type": "string"
        },
        "google": {
          "type": "string"
        },
        "hf": {
          "type": "string"
        },
        "deepseek": {
          "type": "string"
        },
        "xai": {
          "type": "string"
        }
      }
    },
    "overrides": {
      "type": "object",
      "description": "Override contract defaults",
      "properties": {
        "model": {
          "type": "string"
        },
        "provider": {
          "type": "string"
        },
        "temperature": {
          "type": "number"
        },
        "system_prompt": {
          "type": "string"
        }
      }
    },
    "validate_only": {
      "type": "boolean",
      "description": "Check the contract against the IA-EVAL schema and stop: no model call, no key, deterministic. Returns valid + issues instead of throwing. Gate a PR that edits a contract on valid."
    }
  }
}

{
  "type": "object",
  "properties": {
    "valid": {
      "type": "boolean"
    },
    "issues": {},
    "metadata": {
      "type": "object"
    },
    "contract_path": {},
    "warnings": {},
    "summary": {
      "type": "object"
    },
    "scenario_results": {}
  },
  "additionalProperties": true
}

Semantic assertion primitive: compare actual vs expected text pairs using cosine similarity + ROUGE-L. Two modes: tfidf (default, free, no API key) or embeddings (OpenAI text-embedding-3-small, BYOK, true semantic similarity). Returns per-case PASS/FAIL verdicts and an overall verdict. CI-ready: pipe the JSON verdict field to gate a build.

{
  "type": "object",
  "properties": {
    "cases": {
      "type": "array",
      "maxItems": 50,
      "description": "Array of (actual, expected) pairs to evaluate.",
      "items": {
        "type": "object",
        "required": [
          "actual",
          "expected"
        ],
        "properties": {
          "id": {
            "type": "string",
            "description": "Optional identifier for this case."
          },
          "actual": {
            "type": "string",
            "description": "The text produced by your LLM/system."
          },
          "expected": {
            "type": "string",
            "description": "The reference/ground-truth text."
          }
        }
      }
    },
    "thresholds": {
      "type": "object",
      "description": "Pass/fail thresholds (defaults: cosine 0.75, rouge_l 0.5).",
      "properties": {
        "cosine": {
          "type": "number",
          "minimum": 0,
          "maximum": 1,
          "description": "Minimum cosine similarity to pass (default: 0.75)."
        },
        "rouge_l": {
          "type": "number",
          "minimum": 0,
          "maximum": 1,
          "description": "Minimum ROUGE-L F1 to pass (default: 0.5)."
        }
      }
    },
    "require_all": {
      "type": "boolean",
      "description": "If true (default), all cases must pass for overall PASS. If false, at least one case passing returns PASS."
    },
    "mode": {
      "type": "string",
      "enum": [
        "tfidf",
        "embeddings"
      ],
      "description": "tfidf (default): fast, free, lexical. embeddings: OpenAI text-embedding-3-small, true semantic similarity, requires api_key."
    },
    "api_key": {
      "type": "string",
      "description": "OpenAI API key — required only when mode is embeddings."
    }
  },
  "required": [
    "cases"
  ]
}

{
  "type": "object",
  "properties": {
    "verdict": {
      "type": "string"
    },
    "passed": {
      "type": "number"
    },
    "failed": {
      "type": "number"
    },
    "total": {
      "type": "number"
    },
    "results": {
      "type": "array",
      "items": {
        "type": "object"
      }
    },
    "mode": {
      "type": "string"
    }
  },
  "additionalProperties": true
}

Generate a ready-to-commit GitHub Actions workflow that gates a build on IA-QA. Two gate types, combinable: "eval_contract" runs a .ia-eval.yaml through ia-qa-com/eval-action@v1 (LLM quality gate, needs a provider API key as a repo secret), and "cli_checks" runs deterministic primitives via npx @ia-qa/cli (secret scan, prompt-injection scan, security headers…) whose exit code fails the build. Deterministic template — no LLM call, no API key, same inputs give the same file. Returns the YAML, the secrets to create, and the remaining steps. Pair with generate_eval_yaml to produce the contract itself.

{
  "type": "object",
  "properties": {
    "gate": {
      "type": "string",
      "enum": [
        "eval_contract",
        "cli_checks",
        "both",
        "selector_drift",
        "all"
      ],
      "description": "Which gate to emit. eval_contract = LLM eval via the action (default). cli_checks = deterministic CLI assertions. selector_drift = an E2E selector-drift gate via @ia-qa/self-healing (boots the app, captures, diffs against the committed baseline, branches on exit code 0/1/2). both = CLI checks + eval. all = CLI checks, then drift, then eval."
    },
    "contract_path": {
      "type": "string",
      "description": "Path to the .ia-eval.yaml contract, relative to the repo root (default: evals/smoke.ia-eval.yaml). Only used when the gate includes eval_contract."
    },
    "provider": {
      "type": "string",
      "enum": [
        "groq",
        "openai",
        "anthropic",
        "google"
      ],
      "description": "LLM provider the contract runs against — decides which repository secret the workflow wires (default: groq)."
    },
    "cli_tools": {
      "type": "array",
      "items": {
        "type": "string"
      },
      "description": "IA-QA tool names to run as deterministic gates, e.g. [\"secret_scan\",\"prompt_injection_scan\"]. Tools with no known CI recipe get a --stdin step flagged in notes."
    },
    "triggers": {
      "type": "array",
      "items": {
        "type": "string",
        "enum": [
          "push",
          "pull_request",
          "workflow_dispatch",
          "schedule"
        ]
      },
      "description": "Workflow triggers (default: push + pull_request)."
    },
    "cron": {
      "type": "string",
      "description": "Cron expression when triggers include 'schedule' (default: '0 6 * * 1' — Mondays 06:00 UTC)."
    },
    "min_score": {
      "type": "number",
      "description": "Override the contract min_score (0-100). Omit to use the value in the contract."
    },
    "fail_on_fail": {
      "type": "boolean",
      "description": "Fail the build on a FAIL/PARTIAL verdict (default: true). Set false to report without gating."
    },
    "workflow_name": {
      "type": "string",
      "description": "Workflow display name (default: \"IA-QA Quality Gate\")."
    },
    "node_version": {
      "type": "string",
      "description": "Node version for the CLI steps (default: \"20\")."
    },
    "app_start_command": {
      "type": "string",
      "description": "Command that boots the app for the selector_drift gate, e.g. \"npm run start:ci\". Cannot be guessed — omitted, the step carries an explicit TODO and the note says so, because a plausible default would silently map nothing."
    },
    "app_base_url": {
      "type": "string",
      "description": "URL the drift gate waits for before capturing (default: http://127.0.0.1:3000). Must match config.baseUrl in .ia-qa/config.json."
    }
  }
}

{
  "type": "object",
  "properties": {
    "path": {
      "type": "string"
    },
    "yaml": {
      "type": "string"
    },
    "gate": {
      "type": "string"
    },
    "secrets_required": {
      "type": "array",
      "items": {
        "type": "string"
      }
    },
    "notes": {
      "type": "array",
      "items": {
        "type": "string"
      }
    },
    "next_steps": {
      "type": "array",
      "items": {
        "type": "string"
      }
    }
  },
  "additionalProperties": true
}

Generate a "needle in a haystack" test: embeds a target fact into a large block of filler text at a specified position. Use this to test LLM context window retrieval accuracy. Returns the full haystack, the question to ask, and metadata. No API key needed.

{
  "type": "object",
  "properties": {
    "needle": {
      "type": "string",
      "description": "The fact to hide (e.g. \"The secret code is ALPHA-42\")"
    },
    "question": {
      "type": "string",
      "description": "The question to ask the LLM (e.g. \"What is the secret code?\")"
    },
    "tokens": {
      "type": "integer",
      "description": "Target haystack size in tokens (default: 5000, max: 100000)",
      "default": 5000
    },
    "position": {
      "type": "string",
      "description": "Where to insert the needle: \"start\", \"middle\", \"end\", \"random\" (default: \"middle\")",
      "enum": [
        "start",
        "middle",
        "end",
        "random"
      ],
      "default": "middle"
    }
  },
  "required": [
    "needle",
    "question"
  ]
}

{
  "type": "object",
  "properties": {
    "haystack": {},
    "question": {},
    "needle": {},
    "position": {},
    "insert_block": {},
    "total_blocks": {
      "type": "number"
    },
    "estimated_tokens": {}
  },
  "additionalProperties": true
}

Generate a complete .ia-eval.yaml evaluation contract from a plain-language description of what your LLM should do. Uses Groq openai/gpt-oss-20b (server-side, no API key needed). Returns ready-to-run YAML for the LLM Test Runner (run_eval_contract). Picks appropriate evaluators (cosine_similarity, contains_check, hallucination_check, etc.) based on the task type.

{
  "type": "object",
  "properties": {
    "description": {
      "type": "string",
      "description": "Plain-language description of what the LLM under test should do. Be specific: describe inputs, expected behaviour, and constraints."
    },
    "system_prompt": {
      "type": "string",
      "description": "Optional system prompt of the LLM under test. Helps generate more accurate test cases."
    },
    "scenario_count": {
      "type": "number",
      "enum": [
        3,
        5,
        8
      ],
      "description": "Number of scenarios to generate (default: 5). Covers happy path + edge cases + adversarial."
    },
    "task_type": {
      "type": "string",
      "enum": [
        "rag",
        "summarization",
        "classification",
        "safety",
        "customer_support",
        "code_gen"
      ],
      "description": "Optional task type hint to guide evaluator selection."
    }
  },
  "required": [
    "description"
  ]
}

{
  "type": "object",
  "properties": {
    "yaml": {
      "type": "string"
    },
    "task_type": {
      "type": "string"
    },
    "scenario_count": {
      "type": "number"
    },
    "model_used": {
      "type": "string"
    }
  },
  "additionalProperties": true
}

Reference-free stability primitive: instead of comparing an answer to a ground truth, it checks that an assistant's answer stays INVARIANT when the QUESTION is transformed (typo, casing, paraphrase, reordering, translation). Catches the failure class no reference answer can expose — an assistant that handles one phrasing well and a trivial variant of it badly. You bring the outputs (no model is called), so it is deterministic and free in tfidf mode. Relations: case (θ .95), typo (.90), paraphrase (.80), reorder (.80), translation (.75, embeddings only), specialization (.60, ADVISORY — directional, never gated). Returns PASS / FAIL / INVALID, where INVALID means the BASE answer was a refusal or too short so invariance was never measurable — an assistant that refuses every variant would otherwise score a perfect 1.0. Use run_semantic_tests alongside it: invariance without a correctness floor is a green light for a broken assistant.

{
  "type": "object",
  "properties": {
    "base": {
      "type": "object",
      "description": "The reference run: the original question and the answer your system produced for it.",
      "properties": {
        "question": {
          "type": "string",
          "description": "Optional — the original question, echoed back in the output for readability."
        },
        "output": {
          "type": "string",
          "description": "Required — the answer your system produced for the original question."
        }
      }
    },
    "variants": {
      "type": "array",
      "maxItems": 20,
      "description": "Answers produced for transformed versions of the same question, each tagged with the relation that was applied.",
      "items": {
        "type": "object",
        "required": [
          "relation",
          "output"
        ],
        "properties": {
          "id": {
            "type": "string",
            "description": "Optional identifier (default: variant_<n>)."
          },
          "relation": {
            "type": "string",
            "enum": [
              "case",
              "typo",
              "paraphrase",
              "reorder",
              "translation",
              "specialization"
            ],
            "description": "Transformation applied to the QUESTION. Decides the metric and the default threshold."
          },
          "question": {
            "type": "string",
            "description": "Optional — the transformed question, echoed back for readability."
          },
          "output": {
            "type": "string",
            "description": "Required — the answer your system produced for the transformed question."
          }
        }
      }
    },
    "mode": {
      "type": "string",
      "enum": [
        "tfidf",
        "embeddings"
      ],
      "description": "tfidf (default): free, lexical, deterministic — but a genuine paraphrase rarely reaches 0.80, so gate on case/typo and treat paraphrase as a trend. embeddings: OpenAI text-embedding-3-small, true semantic similarity, requires api_key. translation requires this mode."
    },
    "api_key": {
      "type": "string",
      "description": "OpenAI API key — required only when mode is embeddings."
    },
    "thresholds": {
      "type": "object",
      "description": "Per-relation threshold overrides. Calibrate on your own corpus before gating — the defaults are starting points, not measurements.",
      "properties": {
        "case": {
          "type": "number",
          "minimum": 0,
          "maximum": 1
        },
        "typo": {
          "type": "number",
          "minimum": 0,
          "maximum": 1
        },
        "paraphrase": {
          "type": "number",
          "minimum": 0,
          "maximum": 1
        },
        "reorder": {
          "type": "number",
          "minimum": 0,
          "maximum": 1
        },
        "translation": {
          "type": "number",
          "minimum": 0,
          "maximum": 1
        },
        "specialization": {
          "type": "number",
          "minimum": 0,
          "maximum": 1
        }
      }
    },
    "baseline_guard": {
      "type": "object",
      "description": "Correctness floor applied to the BASE answer before anything is scored. Failing it returns INVALID, not FAIL.",
      "properties": {
        "min_length": {
          "type": "number",
          "description": "Minimum base answer length in chars (default 40)."
        },
        "must_not_match": {
          "type": "array",
          "items": {
            "type": "string"
          },
          "description": "Refusal patterns, matched case-insensitively in the FIRST 200 CHARS of the base answer (refusals lead; matching anywhere would flag a long correct answer that merely mentions one). Replaces the FR+EN default list, never merges — pass [] to disable."
        }
      }
    },
    "require_all": {
      "type": "boolean",
      "description": "If true (default), every gated variant must pass. KEEP THE DEFAULT for any run you gate on. Setting it false is not a tolerance dial but an off switch: relations have asymmetric pass rates (a typo variant usually scores ~1.0 because the answer really is identical), so one trivial row is enough to hold the whole run at PASS while a paraphrase fails. When that happens the result carries an explicit warning naming the failed rows."
    }
  },
  "required": [
    "base",
    "variants"
  ]
}

{
  "type": "object",
  "properties": {
    "verdict": {
      "type": "string"
    },
    "mode": {
      "type": "string"
    },
    "baseline": {
      "type": "object"
    },
    "summary": {
      "type": "object"
    },
    "variants": {
      "type": "array",
      "items": {
        "type": "object"
      }
    },
    "weakest": {
      "type": "object"
    },
    "warnings": {
      "type": "array",
      "items": {
        "type": "string"
      }
    },
    "thresholds": {
      "type": "object"
    }
  },
  "additionalProperties": true
}

Test a SKILL.md in the open Agent Skills format (https://agentskills.io/specification — read by Claude Code, Codex, GitHub Copilot, Cursor, Gemini CLI, OpenCode and others) in two stages. (1) lint: the spec's frontmatter rules checked without a model (name 1-64 chars, lowercase/digits/single hyphens, matches dir_name when given; description 1-1024 chars; compatibility ≤500; metadata string map) — any error returns BLOCK and calls no model; gate on lint.valid, it is deterministic. (2) triggering, measured the way an agent decides: the model sees ONLY the name + description of this skill and of the other installed skills (other_skills, or six built-in neighbours) and answers which one it would load, at temperature 0. trigger_accuracy = share of requests routed correctly (should load / should not); verdict PASS ≥80, FIX ≥50, else BLOCK. Pass your own scenarios [{prompt, should_trigger}] for a reproducible gate; without them requests are generated (half should-load, half near misses) and the result is a signal, flagged by scenarios_source. step_adherence is keyword overlap on the skill body's run and never enters the verdict. A request the model never answered readably (rate limit, empty output) is not counted as a miss: the run is INCONCLUSIVE instead. Uses the server's default Groq model (server key, rate-limited). Pass api_key + model to use your own provider; the SKILL.md and scenarios are sent to ia-qa.com and to that provider.

{
  "type": "object",
  "properties": {
    "skill_md": {
      "type": "string",
      "description": "Full content of the SKILL.md file to test, YAML frontmatter included."
    },
    "dir_name": {
      "type": "string",
      "description": "Name of the folder holding the SKILL.md. The spec requires name to match it; checked only when given."
    },
    "scenarios": {
      "type": "array",
      "description": "Fixed user requests for a reproducible gate: [{\"prompt\": \"...\", \"should_trigger\": true}]. Include near misses with should_trigger false. Max 20. When omitted, scenario_count requests are generated.",
      "items": {
        "type": "object",
        "properties": {
          "prompt": {
            "type": "string"
          },
          "should_trigger": {
            "type": "boolean"
          }
        },
        "required": [
          "prompt",
          "should_trigger"
        ]
      }
    },
    "other_skills": {
      "type": "array",
      "description": "The other skills installed next to this one, [{\"name\", \"description\"}], so triggering is tested against the real neighbours. Default: six built-in generic skills.",
      "items": {
        "type": "object",
        "properties": {
          "name": {
            "type": "string"
          },
          "description": {
            "type": "string"
          }
        },
        "required": [
          "name",
          "description"
        ]
      }
    },
    "scenario_count": {
      "type": "number",
      "enum": [
        4,
        6,
        8,
        10
      ],
      "description": "Number of requests to generate when scenarios is omitted: half should-load, half near misses. Default: 6."
    },
    "lint_only": {
      "type": "boolean",
      "description": "Run the format check only: no model call, no key, deterministic. The CI gate for a SKILL.md — gate on lint.valid."
    },
    "model": {
      "type": "string",
      "description": "LLM model ID to use for both scenario generation and testing (e.g. gpt-4o-mini, claude-3-5-haiku-20241022). Defaults to openai/gpt-oss-20b (Groq, server key)."
    },
    "api_key": {
      "type": "string",
      "description": "API key for the chosen model provider. Not required when using the default Groq model."
    }
  },
  "required": [
    "skill_md"
  ]
}

{
  "type": "object",
  "properties": {
    "verdict": {
      "type": "string"
    },
    "score": {
      "type": "number"
    },
    "trigger_accuracy": {
      "type": "number"
    },
    "step_adherence": {
      "type": "number"
    },
    "lint": {
      "type": "object"
    },
    "scenarios_source": {
      "type": "string"
    },
    "scenarios": {
      "type": "array",
      "items": {
        "type": "object"
      }
    }
  },
  "additionalProperties": true
}

Validate an LLM response against QA criteria: format checks (JSON, code, markdown), content rules (must-include, must-not-include), length constraints, language detection, and safety patterns. Essential for QA testing LLM-powered features.

{
  "type": "object",
  "properties": {
    "output": {
      "type": "string",
      "description": "The LLM output text to validate"
    },
    "expected_format": {
      "type": "string",
      "description": "Expected output format",
      "enum": [
        "json",
        "markdown",
        "code",
        "plain",
        "any"
      ]
    },
    "must_include": {
      "type": "string",
      "description": "Comma-separated strings that MUST appear in the output"
    },
    "must_not_include": {
      "type": "string",
      "description": "Comma-separated strings that must NOT appear (e.g. \"TODO, FIXME, undefined, NaN\")"
    },
    "max_length": {
      "type": "number",
      "description": "Maximum character length for the output"
    },
    "min_length": {
      "type": "number",
      "description": "Minimum character length for the output"
    },
    "expected_language": {
      "type": "string",
      "description": "Expected language of the output (en, fr, es, de…). Checks for common words."
    },
    "check_safety": {
      "type": "boolean",
      "description": "Check for PII patterns (emails, phones, SSN), profanity signals, and prompt leakage"
    },
    "check_json_schema": {
      "type": "string",
      "description": "If expected_format is JSON, provide required keys as comma-separated list to validate the structure"
    }
  },
  "required": [
    "output"
  ]
}

{
  "type": "object",
  "properties": {
    "passed": {},
    "failed": {},
    "total": {
      "type": "number"
    },
    "verdict": {},
    "checks": {}
  },
  "additionalProperties": true
}

Compare two ALREADY-PRODUCED outputs (e.g. model A vs model B on the same task) side by side. Returns deterministic metrics (token cosine, ROUGE-L, Jaccard, length/structure deltas, JSON diff) and a verdict. If a `reference` (ground truth) is given, scores each output against it and picks the closer one. If `model` + `api_key` are given, an LLM judge also picks a qualitative winner for the task. No re-execution — you bring the outputs.

{
  "type": "object",
  "properties": {
    "response_a": {
      "type": "string",
      "description": "First output (e.g. model A's answer)"
    },
    "response_b": {
      "type": "string",
      "description": "Second output (e.g. model B's answer)"
    },
    "label_a": {
      "type": "string",
      "description": "Label for output A (e.g. \"GPT-4o\", \"v1.0\")"
    },
    "label_b": {
      "type": "string",
      "description": "Label for output B (e.g. \"GPT-5-nano\", \"v1.1\")"
    },
    "task": {
      "type": "string",
      "description": "The task/prompt both outputs were answering — used by the LLM judge for context"
    },
    "reference": {
      "type": "string",
      "description": "Optional ground-truth / expected answer. If set, each output is scored against it and the closer one wins (deterministic)."
    },
    "check_json": {
      "type": "boolean",
      "description": "Try to parse as JSON and compare structurally (keys, types, values)"
    },
    "model": {
      "type": "string",
      "description": "Optional judge model id (BYOK). When set with api_key, an LLM judge picks a qualitative winner."
    },
    "api_key": {
      "type": "string",
      "description": "Optional API key for the judge model (BYOK). Used only for the judge call; never stored."
    }
  },
  "required": [
    "response_a",
    "response_b"
  ]
}

{
  "type": "object",
  "properties": {
    "labelA": {},
    "labelB": {},
    "metrics": {},
    "verdict": {},
    "judge": {},
    "summary": {
      "type": "string"
    }
  },
  "additionalProperties": true
}

Semantically analyze N already-produced model outputs for the SAME task (the MCP counterpart to the LLM Sandbox). Without a reference: computes consensus — pairwise cosine agreement, the most-representative output, and the outlier. With a `reference` (ground truth): also ranks every output by closeness (token cosine + ROUGE-L composite) and names the closest. No LLM, no key, runs locally. The reference mode is gate-able in CI; the consensus/outlier mode is a signal (no reference to fail against). You bring the outputs (2+). For a 2-way head-to-head with structural JSON diff use compare_responses instead.

{
  "type": "object",
  "properties": {
    "responses": {
      "type": "array",
      "minItems": 2,
      "description": "The outputs to analyze (same task, N models/prompts/versions). Each item is a plain string or { \"label\": \"GPT-4o\", \"text\": \"...\" }. At least 2 required.",
      "items": {
        "type": "object",
        "required": [
          "text"
        ],
        "properties": {
          "label": {
            "type": "string",
            "description": "Human name for this candidate (e.g. model id)"
          },
          "text": {
            "type": "string",
            "description": "The produced output"
          }
        }
      }
    },
    "reference": {
      "type": "string",
      "description": "Optional ground-truth answer. If set, each output is also ranked by closeness to it and the closest one is named."
    }
  },
  "required": [
    "responses"
  ]
}

{
  "type": "object",
  "properties": {
    "count": {},
    "consensus": {},
    "reference_ranking": {},
    "summary": {
      "type": "string"
    }
  },
  "additionalProperties": true
}

Define a test suite for a prompt: provide the system prompt, user prompt, and expected output criteria. Returns a test plan with scored rubric — use this as input for manual or automated LLM evaluation.

{
  "type": "object",
  "properties": {
    "system_prompt": {
      "type": "string",
      "description": "The system prompt under test"
    },
    "user_prompt": {
      "type": "string",
      "description": "The user prompt to send"
    },
    "expected_behavior": {
      "type": "string",
      "description": "Description of what the LLM should do (free text)"
    },
    "expected_format": {
      "type": "string",
      "description": "Expected output format",
      "enum": [
        "json",
        "markdown",
        "code",
        "plain",
        "any"
      ]
    },
    "must_include": {
      "type": "string",
      "description": "Required content (comma-separated)"
    },
    "must_not_include": {
      "type": "string",
      "description": "Forbidden content (comma-separated)"
    },
    "max_tokens": {
      "type": "number",
      "description": "Max token budget for the test"
    },
    "temperature": {
      "type": "number",
      "description": "Temperature to use"
    },
    "check_safety": {
      "type": "boolean",
      "description": "Include safety/PII checks in the rubric"
    },
    "adversarial_prompts": {
      "type": "boolean",
      "description": "Auto-generate adversarial test variants (jailbreak, injection, edge cases)"
    }
  },
  "required": [
    "system_prompt",
    "user_prompt"
  ]
}

{
  "type": "object",
  "properties": {
    "test_suite_name": {
      "type": "string"
    },
    "total_tests": {
      "type": "number"
    },
    "categories": {
      "type": "array"
    },
    "rubric": {},
    "instructions": {
      "type": "string"
    }
  },
  "additionalProperties": true
}

Validate that an LLM output matches an expected format: JSON, Markdown, code block, bullet list, numbered list, table, YAML, XML, or custom regex. Essential for structured output testing.

{
  "type": "object",
  "properties": {
    "output": {
      "type": "string",
      "description": "The LLM output to validate"
    },
    "expected_format": {
      "type": "string",
      "description": "Expected format",
      "enum": [
        "json",
        "markdown_heading",
        "code_block",
        "bullet_list",
        "numbered_list",
        "table",
        "yaml",
        "xml",
        "regex"
      ]
    },
    "regex_pattern": {
      "type": "string",
      "description": "Custom regex pattern (only when expected_format is \"regex\")"
    }
  },
  "required": [
    "output",
    "expected_format"
  ]
}

{
  "type": "object",
  "properties": {
    "expected_format": {},
    "total_checks": {
      "type": "number"
    },
    "passed": {},
    "failed": {},
    "valid": {
      "type": "boolean"
    },
    "checks": {}
  },
  "additionalProperties": true
}

Compare multiple LLM responses to the same prompt and detect inconsistencies using Jaccard word-overlap similarity and fact drift (number comparison). Fast, deterministic, no API key needed. Limitations: relies on surface-level word matching — "Paris is the capital of France" vs "Paris is the French capital" may score low despite semantic equivalence. For true semantic consistency, use run_semantic_tests with embedding mode. Essential for determinism testing.

{
  "type": "object",
  "properties": {
    "responses": {
      "type": "array",
      "items": {
        "type": "string"
      },
      "description": "Array of 2+ LLM responses to compare (same prompt, different runs)"
    },
    "check_facts": {
      "type": "boolean",
      "description": "Check for contradictory numbers/facts across responses (default: true)"
    }
  },
  "required": [
    "responses"
  ]
}

{
  "type": "object",
  "properties": {
    "verdict": {},
    "avg_similarity": {},
    "fact_contradiction": {},
    "response_count": {
      "type": "number"
    },
    "length_variance_percent": {},
    "pairwise_scores": {},
    "fact_drift": {}
  },
  "additionalProperties": true
}

Validate that an LLM JSON output matches a JSON Schema definition. Tests required fields, types, enums, nested objects, and arrays. Critical for function-calling and structured output testing.

{
  "type": "object",
  "properties": {
    "output": {
      "type": "string",
      "description": "The LLM JSON output (raw string, will be parsed)"
    },
    "schema": {
      "type": "object",
      "additionalProperties": true,
      "description": "JSON Schema (draft-07 subset) to validate against"
    }
  },
  "required": [
    "output",
    "schema"
  ]
}

{
  "type": "object",
  "properties": {
    "valid": {
      "type": "boolean"
    },
    "parse_error": {},
    "errors": {
      "type": "array"
    },
    "error_count": {
      "type": "number"
    },
    "parsed_type": {}
  },
  "additionalProperties": true
}

Score an LLM response against the criteria you pass: coverage of expected_keywords and compliance with max_length. Returns a 0-100 score over the criteria actually measured — or total_score null with the reason when none is given, because relevance and correctness cannot be read off word overlap (a correct one-word answer shares no words with its question). For meaning against a reference answer, use run_semantic_tests. Also returns unscored signals: question-term overlap, average sentence length, markdown.

{
  "type": "object",
  "properties": {
    "question": {
      "type": "string",
      "description": "The original question/prompt"
    },
    "response": {
      "type": "string",
      "description": "The LLM response to score"
    },
    "expected_keywords": {
      "type": "array",
      "items": {
        "type": "string"
      },
      "description": "Keywords a correct answer must contain (case-insensitive). Scored as coverage."
    },
    "max_length": {
      "type": "number",
      "description": "Maximum character length. Scored as a proportional penalty beyond it."
    }
  },
  "required": [
    "question",
    "response"
  ]
}

{
  "type": "object",
  "properties": {
    "total_score": {},
    "max_score": {
      "type": "number"
    },
    "grade": {},
    "breakdown": {
      "type": "object"
    },
    "not_measured": {
      "type": "array"
    },
    "reason": {
      "type": "string"
    },
    "signals": {
      "type": "object"
    },
    "stats": {
      "type": "object"
    }
  },
  "additionalProperties": true
}

Analyze a multi-turn conversation for context retention, topic drift, instruction following, and repetition. Accepts messages array [{role, content}]. Essential for chatbot QA.

{
  "type": "object",
  "properties": {
    "messages": {
      "type": "array",
      "items": {
        "type": "object",
        "properties": {
          "role": {
            "type": "string",
            "enum": [
              "system",
              "user",
              "assistant"
            ]
          },
          "content": {
            "type": "string"
          }
        }
      },
      "description": "Conversation messages in order"
    }
  },
  "required": [
    "messages"
  ]
}

{
  "type": "object",
  "properties": {
    "turn_count": {
      "type": "number"
    },
    "user_messages": {
      "type": "number"
    },
    "assistant_messages": {
      "type": "number"
    },
    "has_system_prompt": {
      "type": "boolean"
    },
    "context_retention": {},
    "repetition_detected": {
      "type": "boolean"
    },
    "repetitions": {},
    "topic_drift": {},
    "avg_response_length": {
      "type": "number"
    }
  },
  "additionalProperties": true
}

Analyze a Chain-of-Thought (CoT) or reasoning trace from an LLM. Detects step count, logical flow, conclusion presence, backtracking, and estimates reasoning depth. Useful for o1/o3/DeepSeek-R1 evaluation.

{
  "type": "object",
  "properties": {
    "reasoning": {
      "type": "string",
      "description": "The CoT / reasoning trace text (e.g. from <think> tags or step-by-step output)"
    },
    "expected_conclusion": {
      "type": "string",
      "description": "Expected final answer to check against (optional)"
    }
  },
  "required": [
    "reasoning"
  ]
}

{
  "type": "object",
  "properties": {
    "step_count": {
      "type": "number"
    },
    "reasoning_depth": {},
    "reasoning_depth_label": {},
    "has_conclusion": {
      "type": "boolean"
    },
    "conclusion_matches_expected": {},
    "backtracking_signals": {},
    "markers": {},
    "total_lines": {
      "type": "number"
    },
    "total_chars": {
      "type": "number"
    }
  },
  "additionalProperties": true
}

Generate an A/B test report comparing two prompts or model configurations. Accepts arrays of scores and returns mean, median, std deviation, improvement percentage, and a Welch t-test at 95 %: `winner` names a variant only when the difference is significant — otherwise "no_significant_difference", or "insufficient_samples" under 2 scores per variant.

{
  "type": "object",
  "properties": {
    "variant_a": {
      "type": "object",
      "description": "First variant configuration with name and score array",
      "properties": {
        "name": {
          "type": "string",
          "description": "Name/label for variant A"
        },
        "scores": {
          "type": "array",
          "items": {
            "type": "number"
          },
          "description": "Array of scores (0-100)"
        }
      }
    },
    "variant_b": {
      "type": "object",
      "description": "Second variant configuration with name and score array",
      "properties": {
        "name": {
          "type": "string",
          "description": "Name/label for variant B"
        },
        "scores": {
          "type": "array",
          "items": {
            "type": "number"
          },
          "description": "Array of scores (0-100)"
        }
      }
    }
  },
  "required": [
    "variant_a",
    "variant_b"
  ]
}

{
  "type": "object",
  "properties": {
    "mean": {
      "type": "string"
    },
    "median": {
      "type": "string"
    },
    "std_dev": {
      "type": "string"
    },
    "min": {},
    "max": {},
    "count": {
      "type": "number"
    },
    "winner": {},
    "improvement_percent": {},
    "variant_a": {
      "type": "object"
    },
    "variant_b": {
      "type": "object"
    },
    "recommendation": {
      "type": "number"
    }
  },
  "additionalProperties": true
}

Compute text similarity between reference and hypothesis using multiple metrics: Cosine (BoW, TF-IDF), Jaccard, ROUGE-1, ROUGE-2, ROUGE-L, and BLEU. No API key needed. Ideal for LLM eval (expected vs actual), RAG quality checks, and NLG benchmarking. Supports batch mode.

{
  "type": "object",
  "properties": {
    "reference": {
      "type": "string",
      "description": "Reference / expected text (ground truth)"
    },
    "hypothesis": {
      "type": "string",
      "description": "Hypothesis / actual text (LLM output)"
    },
    "metrics": {
      "type": "array",
      "description": "Metrics to compute (default: all). Options: \"cosine_bow\", \"cosine_tfidf\", \"jaccard\", \"rouge1\", \"rouge2\", \"rougeL\", \"bleu\"",
      "items": {
        "type": "string"
      }
    },
    "batch": {
      "type": "array",
      "description": "Batch mode: array of {reference, hypothesis} pairs.",
      "items": {
        "type": "object",
        "properties": {
          "reference": {
            "type": "string"
          },
          "hypothesis": {
            "type": "string"
          }
        },
        "required": [
          "reference",
          "hypothesis"
        ]
      }
    },
    "threshold": {
      "type": "number",
      "description": "Optional pass/fail threshold (0-1). Applies to ROUGE-L F1 score."
    }
  }
}

{
  "type": "object",
  "properties": {
    "precision": {
      "type": "number"
    },
    "recall": {
      "type": "number"
    },
    "f1": {
      "type": "number"
    },
    "mode": {
      "type": "string"
    },
    "count": {
      "type": "number"
    },
    "results": {}
  },
  "additionalProperties": true
}

Run a test suite against a Vision-Language Model (VLM) — send an image (URL or base64) + N test cases (each with a question + assertion) to GPT-4o, Claude 3.5, or Gemini. Returns per-case PASS/FAIL verdicts, a pass rate, an overall PASS/WARNING/FAIL verdict (customizable threshold), and latency stats. Assertion types: contains, not_contains, json_format, min_length, max_length, semantic_contains (TF-IDF cosine similarity ≥ 0.4). BYOK: requires your own API key for the target provider.

{
  "type": "object",
  "properties": {
    "image_url": {
      "type": "string",
      "description": "Public URL of the image to evaluate (required unless image_base64 is provided)."
    },
    "image_base64": {
      "type": "string",
      "description": "Base64-encoded image data (required unless image_url is provided)."
    },
    "image_mime_type": {
      "type": "string",
      "description": "MIME type of the image if using image_base64 (default: image/jpeg)."
    },
    "system_prompt": {
      "type": "string",
      "description": "Optional system prompt sent to the VLM."
    },
    "threshold": {
      "type": "number",
      "description": "Pass rate threshold for overall verdict (default: 80, 0–100)."
    },
    "model": {
      "type": "string",
      "enum": [
        "gpt-4o",
        "gpt-4o-mini",
        "claude-3-5-sonnet-20241022",
        "claude-3-5-haiku-20241022",
        "gemini-3.5-flash",
        "gemini-2.5-flash"
      ],
      "description": "VLM model to use."
    },
    "api_key": {
      "type": "string",
      "description": "API key for the model provider (OpenAI sk-, Anthropic sk-ant-, or Google AIzaSy...)."
    },
    "test_cases": {
      "type": "array",
      "maxItems": 10,
      "description": "Array of test cases to run.",
      "items": {
        "type": "object",
        "required": [
          "question"
        ],
        "properties": {
          "id": {
            "type": "string",
            "description": "Optional identifier for this case."
          },
          "question": {
            "type": "string",
            "description": "Question to ask the VLM about the image."
          },
          "assertion_type": {
            "type": "string",
            "enum": [
              "contains",
              "not_contains",
              "json_format",
              "min_length",
              "max_length",
              "semantic_contains"
            ],
            "description": "Assertion to run on the VLM response. semantic_contains uses TF-IDF cosine similarity ≥ 0.4."
          },
          "assertion_value": {
            "type": "string",
            "description": "Expected value for the assertion (not needed for json_format)."
          }
        }
      }
    }
  },
  "required": [
    "test_cases",
    "model",
    "api_key"
  ]
}

{
  "type": "object",
  "properties": {
    "verdict": {
      "type": "string"
    },
    "passed": {
      "type": "number"
    },
    "failed": {
      "type": "number"
    },
    "total": {
      "type": "number"
    },
    "results": {
      "type": "array",
      "items": {
        "type": "object"
      }
    },
    "model": {
      "type": "string"
    }
  },
  "additionalProperties": true
}

Compute text similarity using local algorithms (Bag of Words, TF-IDF, Character N-grams). No API key needed — runs entirely in-process. NOT real embeddings: for true semantic similarity with vector embeddings, use run_semantic_tests with mode="embeddings" and your OpenAI API key. Supports single pair or batch mode with pipe-separated pairs. Useful for RAG retrieval testing, semantic search evaluation, and text deduplication.

{
  "type": "object",
  "properties": {
    "text_a": {
      "type": "string",
      "description": "First text to compare (single-pair mode)"
    },
    "text_b": {
      "type": "string",
      "description": "Second text to compare (single-pair mode)"
    },
    "batch": {
      "type": "array",
      "description": "Batch mode: array of { text_a, text_b } pairs. Overrides text_a/text_b if provided.",
      "items": {
        "type": "object",
        "required": [
          "text_a",
          "text_b"
        ],
        "properties": {
          "text_a": {
            "type": "string"
          },
          "text_b": {
            "type": "string"
          }
        }
      }
    },
    "methods": {
      "type": "array",
      "description": "Algorithms to use (default: all three). Options: \"bow\", \"tfidf\", \"ngram\"",
      "items": {
        "type": "string",
        "enum": [
          "bow",
          "tfidf",
          "ngram"
        ]
      }
    }
  }
}

{
  "type": "object",
  "properties": {
    "mode": {
      "type": "string"
    },
    "count": {
      "type": "number"
    },
    "results": {},
    "text_a": {},
    "text_b": {},
    "scores": {}
  },
  "additionalProperties": true
}

Rank an array of text chunks by relevance to a query using TF-IDF scoring. Simulates retrieval ranking for RAG testing without needing embeddings or an API.

{
  "type": "object",
  "properties": {
    "query": {
      "type": "string",
      "description": "The user query"
    },
    "chunks": {
      "type": "array",
      "items": {
        "type": "string"
      },
      "description": "Array of text chunks to rank"
    },
    "top_k": {
      "type": "number",
      "description": "Return top K results (default: all)"
    }
  },
  "required": [
    "query",
    "chunks"
  ]
}

{
  "type": "object",
  "properties": {
    "rank": {
      "type": "number"
    },
    "index": {},
    "score": {
      "type": "string"
    },
    "keyword_overlap": {},
    "chunk_preview": {},
    "query": {},
    "total_chunks": {
      "type": "number"
    },
    "returned": {
      "type": "number"
    },
    "results": {}
  },
  "additionalProperties": true
}

Compute BM25 relevance score between a query and one or more documents. BM25 is the industry-standard keyword-based ranking algorithm used in Elasticsearch, OpenSearch, and Weaviate hybrid search. Returns ranked results with normalized scores.

{
  "type": "object",
  "properties": {
    "query": {
      "type": "string",
      "description": "The search query"
    },
    "documents": {
      "type": "array",
      "items": {
        "type": "string"
      },
      "description": "Array of documents to rank"
    },
    "k1": {
      "type": "number",
      "description": "Term frequency saturation (default: 1.5)"
    },
    "b": {
      "type": "number",
      "description": "Length normalization factor (default: 0.75)"
    },
    "top_k": {
      "type": "number",
      "description": "Return top K results (default: all)"
    }
  },
  "required": [
    "query",
    "documents"
  ]
}

{
  "type": "object",
  "properties": {
    "index": {},
    "bm25_score": {
      "type": "number"
    },
    "doc_length": {
      "type": "number"
    },
    "doc_preview": {},
    "query": {},
    "documents_count": {
      "type": "number"
    },
    "k1": {},
    "b": {},
    "avg_doc_length": {
      "type": "number"
    },
    "results": {}
  },
  "additionalProperties": true
}

Assemble a complete RAG (Retrieval-Augmented Generation) prompt from retrieved context chunks and a user query. Handles token budgeting, citation numbering, system instruction injection, and source attribution.

{
  "type": "object",
  "properties": {
    "query": {
      "type": "string",
      "description": "The user question to answer"
    },
    "chunks": {
      "type": "array",
      "items": {
        "type": "object",
        "properties": {
          "text": {
            "type": "string"
          },
          "source": {
            "type": "string"
          },
          "score": {
            "type": "number"
          }
        }
      },
      "description": "Retrieved context chunks with .text (required), .source (optional), .score (optional)"
    },
    "system_instruction": {
      "type": "string",
      "description": "Custom system instruction (default: standard RAG grounding instruction)"
    },
    "max_context_tokens": {
      "type": "number",
      "description": "Max tokens for context section (default: 2000)"
    },
    "cite_sources": {
      "type": "boolean",
      "description": "Add [1], [2] citation numbers (default: true)"
    },
    "language": {
      "type": "string",
      "description": "Response language instruction (e.g. \"French\", \"Spanish\")"
    }
  },
  "required": [
    "query",
    "chunks"
  ]
}

{
  "type": "object",
  "properties": {
    "prompt": {},
    "system_prompt": {},
    "total_tokens_estimate": {
      "type": "number"
    },
    "context_tokens_estimate": {
      "type": "number"
    },
    "chunks_included": {
      "type": "number"
    },
    "chunks_truncated": {
      "type": "number"
    },
    "included_chunks": {}
  },
  "additionalProperties": true
}

Evaluate RAG retrieval quality: rank passages against a query and compute Precision@k / Recall@k plus a PASS/FAIL CI verdict from ground-truth relevance labels. Three modes, all keyless except the last. (1) BYO scores — give each passage the `score` your own reranker produced (Cohere, Jina, a self-hosted NIM, a cross-encoder): deterministic, offline, and it evaluates YOUR reranker rather than someone else's. This is the mode to gate CI on. (2) Default, no scores and no key — ranks with local BM25, a lexical keyword baseline: it answers "does a keyword floor already surface my relevant passages?", never "is my neural reranker good". (3) Live NVIDIA reranker — supply `api_key` for an NVIDIA account that still has reranking entitlement; NVIDIA retired its hosted reranking endpoints on 2026-05-18, so this one is for accounts that were grandfathered in.

{
  "type": "object",
  "properties": {
    "query": {
      "type": "string",
      "description": "The search query or question to rank against"
    },
    "passages": {
      "type": "array",
      "description": "Array of passage objects to rank (min 2, max 20)",
      "items": {
        "type": "object",
        "required": [
          "text"
        ],
        "properties": {
          "text": {
            "type": "string"
          },
          "relevant": {
            "type": "boolean",
            "description": "Ground truth: is this passage relevant?"
          },
          "score": {
            "type": "number",
            "description": "Relevance score from YOUR reranker. Present on every passage → ranking is done from these (offline, no key). Higher = more relevant. Score all passages or none."
          },
          "id": {
            "type": "string"
          }
        }
      }
    },
    "top_k": {
      "type": "integer",
      "description": "k for Precision@k evaluation (default 3)",
      "minimum": 1,
      "maximum": 10
    },
    "threshold": {
      "type": "number",
      "description": "Minimum Precision@k to PASS (0-1, default 0.5)",
      "minimum": 0,
      "maximum": 1
    },
    "api_key": {
      "type": "string",
      "description": "Your NVIDIA API key (BYOK), used only when no passage carries a score. Transits RAM for the single call, never stored."
    }
  },
  "required": [
    "query",
    "passages"
  ]
}

{
  "type": "object",
  "properties": {
    "verdict": {
      "type": "string"
    },
    "precision_at_k": {
      "type": "number"
    },
    "recall_at_k": {
      "type": "number"
    },
    "k": {
      "type": "number"
    },
    "threshold": {
      "type": "number"
    },
    "has_ground_truth": {
      "type": "boolean"
    },
    "ranked": {
      "type": "array",
      "items": {
        "type": "object"
      }
    },
    "total_passages": {
      "type": "number"
    },
    "mode": {
      "type": "string"
    },
    "model": {
      "type": "string"
    },
    "latency_ms": {
      "type": "number"
    }
  },
  "additionalProperties": true
}

Estimate the token count of a text string using the cl100k_base approximation (~4 chars/token). Call this BEFORE sending any text to an LLM API to check if it fits within the model context window and to estimate cost. Returns token estimate, character count, and word count.

{
  "type": "object",
  "properties": {
    "input": {
      "type": "string",
      "description": "Text to count tokens for"
    }
  },
  "required": [
    "input"
  ]
}

{
  "type": "object",
  "properties": {
    "tokens_estimate": {},
    "chars": {},
    "words": {}
  },
  "additionalProperties": true
}

Extract the first valid JSON object or array embedded in chaotic LLM output (surrounded by markdown fences, prose, or explanatory text). Handles ```json blocks and inline JSON. Call this whenever an LLM returns structured data mixed with explanation text instead of raw JSON.

{
  "type": "object",
  "properties": {
    "input": {
      "type": "string",
      "description": "Raw text (e.g., LLM output) that may contain a JSON object or array"
    }
  },
  "required": [
    "input"
  ]
}

{
  "type": "object",
  "properties": {
    "json": {},
    "source": {
      "type": "string"
    }
  },
  "additionalProperties": true
}

Estimate the API cost in USD for a given model and token counts. Supports all major 2024–2026 models: GPT-4o, GPT-4.1, o3, o4-mini, Claude Opus 4, Claude Sonnet 4/4.5, Gemini 2.5 Pro/Flash, DeepSeek V3/R1, Grok 3, and legacy models.

{
  "type": "object",
  "properties": {
    "model": {
      "type": "string",
      "description": "Model name, e.g. \"gpt-4o\", \"claude-3.5-sonnet\", \"deepseek-v3\""
    },
    "input_tokens": {
      "type": "number",
      "description": "Number of input/prompt tokens"
    },
    "output_tokens": {
      "type": "number",
      "description": "Number of output/completion tokens (default: 0)"
    }
  },
  "required": [
    "model",
    "input_tokens"
  ]
}

{
  "type": "object",
  "properties": {
    "model": {},
    "input_tokens": {},
    "output_tokens": {},
    "input_cost_usd": {
      "type": "string"
    },
    "output_cost_usd": {
      "type": "string"
    },
    "total_cost_usd": {
      "type": "string"
    },
    "rates": {
      "type": "object"
    }
  },
  "additionalProperties": true
}

Plan token allocation across system prompt, user input, context/RAG chunks, and expected output. Warns if budget exceeds model context window. Supports 25+ models.

{
  "type": "object",
  "properties": {
    "model": {
      "type": "string",
      "description": "Model name (e.g. gpt-4o, claude-3.5-sonnet, gemini-3.5-flash)"
    },
    "system_prompt_tokens": {
      "type": "number",
      "description": "Token count for system prompt"
    },
    "user_input_tokens": {
      "type": "number",
      "description": "Token count for user message"
    },
    "context_tokens": {
      "type": "number",
      "description": "Token count for RAG context / documents"
    },
    "expected_output_tokens": {
      "type": "number",
      "description": "Expected max output tokens"
    },
    "system_prompt": {
      "type": "string",
      "description": "Actual system prompt text (will estimate tokens)"
    },
    "user_input": {
      "type": "string",
      "description": "Actual user input text (will estimate tokens)"
    },
    "context": {
      "type": "string",
      "description": "Actual context text (will estimate tokens)"
    }
  },
  "required": [
    "model"
  ]
}

{
  "type": "object",
  "properties": {
    "model": {},
    "context_window": {},
    "breakdown": {
      "type": "object"
    },
    "fits_in_window": {},
    "remaining_tokens": {},
    "utilization_percent": {},
    "warnings": {}
  },
  "additionalProperties": true
}

Given an array of message objects [{role, content}], estimate total token usage and check if it fits in the target model's context window. Warns about truncation risk.

{
  "type": "object",
  "properties": {
    "messages": {
      "type": "array",
      "items": {
        "type": "object",
        "properties": {
          "role": {
            "type": "string"
          },
          "content": {
            "type": "string"
          }
        }
      },
      "description": "Array of messages (system/user/assistant)"
    },
    "model": {
      "type": "string",
      "description": "Target model name (e.g. gpt-4o, claude-3.5-sonnet)"
    },
    "max_output_tokens": {
      "type": "number",
      "description": "Reserved tokens for output (default: 4096)"
    }
  },
  "required": [
    "messages",
    "model"
  ]
}

{
  "type": "object",
  "properties": {
    "index": {},
    "role": {},
    "tokens": {},
    "chars": {
      "type": "number"
    },
    "model": {},
    "context_window": {},
    "total_input_tokens": {},
    "reserved_output_tokens": {},
    "total_tokens": {},
    "fits": {},
    "utilization_percent": {
      "type": "string"
    },
    "breakdown": {
      "type": "object"
    },
    "message_count": {
      "type": "number"
    },
    "per_message": {},
    "warnings": {}
  },
  "additionalProperties": true
}

Get detailed specs for an AI model: context window, pricing per 1K tokens, knowledge cutoff, provider, multimodal support, reasoning capabilities, and feature list. Covers 30+ models from OpenAI, Anthropic, Google, DeepSeek, Meta, Mistral, Cohere, xAI.

{
  "type": "object",
  "properties": {
    "model": {
      "type": "string",
      "description": "Model name (e.g. \"gpt-4o\", \"claude-3.5-sonnet\", \"gemini-3.1-pro-preview\")"
    }
  },
  "required": [
    "model"
  ]
}

{
  "type": "object",
  "properties": {
    "model": {},
    "pricing_per_1k": {
      "type": "object"
    }
  },
  "additionalProperties": true
}

Compare 2-5 AI models side by side: context window, pricing, multimodal, reasoning capabilities, and provider. Returns a comparison table with a recommendation based on your use case.

{
  "type": "object",
  "properties": {
    "models": {
      "type": "array",
      "items": {
        "type": "string"
      },
      "description": "Array of 2-5 model names (e.g. [\"gpt-4o\",\"claude-3.5-sonnet\",\"gemini-3.5-flash\"])"
    },
    "use_case": {
      "type": "string",
      "enum": [
        "cost",
        "context",
        "reasoning",
        "multimodal",
        "speed"
      ],
      "description": "Optimize recommendation for this criterion"
    }
  },
  "required": [
    "models"
  ]
}

{
  "type": "object",
  "properties": {
    "model": {},
    "cost_per_1k_total": {
      "type": "string"
    },
    "models_compared": {
      "type": "number"
    },
    "use_case": {},
    "recommendation": {},
    "rows": {}
  },
  "additionalProperties": true
}

List all LLM models available on ia-qa.com with their provider, API endpoint, and capabilities. Filter by provider name (e.g. "Groq", "HuggingFace", "OpenAI") or return the full catalog. Use this to discover which models are available before calling an LLM API, or to compare providers.

{
  "type": "object",
  "properties": {
    "provider": {
      "type": "string",
      "description": "Filter by provider name (case-insensitive). E.g. \"Groq\", \"HuggingFace\", \"OpenAI\", \"Anthropic\", \"Google\", \"DeepSeek\", \"xAI\", \"Ollama\". Omit for full catalog."
    }
  }
}

{
  "type": "object",
  "properties": {
    "models": {},
    "total": {
      "type": "number"
    },
    "providers": {},
    "filter": {}
  },
  "additionalProperties": true
}

Generate text with an open-source model hosted on Groq, on the server's key — no API key needed. Models: openai/gpt-oss-20b, openai/gpt-oss-120b (default "openai/gpt-oss-20b"). HuggingFace models are not served on the server key. A reasoning model that spends max_tokens thinking is retried once at 4096. Rate-limited to prevent abuse.

{
  "type": "object",
  "properties": {
    "prompt": {
      "type": "string",
      "description": "The user prompt / instruction to send to the model"
    },
    "model": {
      "type": "string",
      "description": "Model ID (default: \"openai/gpt-oss-20b\"). Server-keyed whitelist only — Groq: openai/gpt-oss-20b, openai/gpt-oss-120b. Other ids from list_llm_models, HuggingFace ones included, are BYOK-only and will be rejected."
    },
    "system": {
      "type": "string",
      "description": "Optional system prompt to set context or persona"
    },
    "temperature": {
      "type": "number",
      "description": "Sampling temperature 0.0–1.5 (default: 0.7)"
    },
    "max_tokens": {
      "type": "number",
      "description": "Maximum tokens to generate (default: 2048, max: 4096)"
    }
  },
  "required": [
    "prompt"
  ]
}

{
  "type": "object",
  "properties": {
    "content": {},
    "model": {},
    "provider": {},
    "usage": {},
    "latency_ms": {
      "type": "number"
    }
  },
  "additionalProperties": true
}

Give honest usage feedback on an IA-QA MCP tool. Provide a score (1-5) and a comment. Rate low (1-2) if the tool was wrong, irrelevant, or a poor fit; rate high (4-5) only if it genuinely solved your need. Ratings are read by the IA-QA maintainers and are not published, one vote per agent and tool a month. Comments are screened before storing: one carrying a credential is not stored and personal data is replaced by placeholders — describe what happened rather than what a model should do. Skip rating routine successes — we want signal, not praise. Example: rate_tool({ tool_name: "format_json", score: 2, comment: "Tried to pretty-print a JSON5 file, it rejected trailing commas — not usable for my case." })

{
  "type": "object",
  "properties": {
    "tool_name": {
      "type": "string",
      "description": "Name of the MCP tool to rate (e.g. \"format_json\", \"shield_analyze\")"
    },
    "score": {
      "type": "number",
      "description": "Rating from 1 (poor) to 5 (excellent)",
      "minimum": 1,
      "maximum": 5
    },
    "comment": {
      "type": "string",
      "description": "Strongly encouraged — explain what you were trying to do and whether the tool got you there. Be specific about what was missing, wrong, or a poor fit. This is the most valuable part of the rating. Up to 2000 chars are stored; go over and the response says so (truncated: true) — send the remainder as a second call rather than assuming it landed."
    }
  },
  "required": [
    "tool_name",
    "score"
  ]
}

{
  "type": "object",
  "properties": {
    "ok": {
      "type": "boolean"
    },
    "tool_name": {},
    "score": {},
    "comment": {},
    "truncated": {
      "type": "boolean"
    },
    "stored_chars": {
      "type": "number"
    },
    "rated_at": {
      "type": "string"
    },
    "message": {
      "type": "string"
    }
  },
  "additionalProperties": true
}