Web Search API Eval

Headline findings and the overall scorecard across all 30 queries (90 calls per API): accuracy, fully correct, precision@5, freshness, empty responses, failed calls, latency p50/p95, cost per correct answer. Also the per-use-case summary, which API fits which agent, and developer activation time.

{
  "type": "object",
  "properties": {}
}

Full metric scorecard for one use case (gtm, insurance, finance) or 'overall'. Metrics: search-only latency p50/p95 (server and client), calls under 1 s, accuracy, freshness (% in window, newest relevant result age), content in one call, tokens, search cost.

{
  "type": "object",
  "properties": {
    "use_case": {
      "type": "string",
      "enum": [
        "overall",
        "gtm",
        "insurance",
        "finance"
      ],
      "description": "overall, gtm, insurance or finance"
    }
  },
  "required": [
    "use_case"
  ],
  "$schema": "http://json-schema.org/draft-07/schema#"
}

Compare two or more of You.com, Tavily, Exa and Parallel side by side, overall or within one use case, optionally filtered to metrics whose name contains a keyword (e.g. 'latency', 'accuracy', 'fresh', 'cost', 'empty').

{
  "type": "object",
  "properties": {
    "providers": {
      "minItems": 2,
      "type": "array",
      "items": {
        "type": "string",
        "enum": [
          "You.com",
          "Tavily",
          "Exa",
          "Parallel"
        ]
      },
      "description": "Two or more of: You.com, Tavily, Exa, Parallel"
    },
    "use_case": {
      "default": "overall",
      "description": "overall, gtm, insurance or finance",
      "type": "string",
      "enum": [
        "overall",
        "gtm",
        "insurance",
        "finance"
      ]
    },
    "metric_filter": {
      "description": "Optional keyword to keep only matching metrics",
      "type": "string"
    }
  },
  "required": [
    "providers"
  ],
  "$schema": "http://json-schema.org/draft-07/schema#"
}

Which web search API fits an agent, based on this benchmark: live voice/chat turns, deep research, semantic or long-tail queries, docs lookups, or a specific use case (GTM, insurance, finance). Returns the guidance with the supporting numbers.

{
  "type": "object",
  "properties": {
    "agent_need": {
      "description": "What the agent does, e.g. 'voice agent answering in under a second' or 'finance research'",
      "type": "string"
    }
  },
  "$schema": "http://json-schema.org/draft-07/schema#"
}

The 30 benchmark questions (G1-G10 GTM, I1-I10 insurance, F1-F10 finance): ID, what the agent is asked, the search call and freshness.

{
  "type": "object",
  "properties": {
    "use_case": {
      "default": "all",
      "description": "all, gtm, insurance or finance",
      "type": "string",
      "enum": [
        "all",
        "gtm",
        "insurance",
        "finance"
      ]
    }
  },
  "$schema": "http://json-schema.org/draft-07/schema#"
}

One benchmark question by ID (e.g. G3, I4, F5): the task, search call, parameters, answer key with sources, and what each API returned with its accuracy score (0-2, mean of 3 runs).

{
  "type": "object",
  "properties": {
    "id": {
      "type": "string",
      "description": "Question ID, e.g. G3, I4 or F5"
    }
  },
  "required": [
    "id"
  ],
  "$schema": "http://json-schema.org/draft-07/schema#"
}

The published graphs: overall scorecard, key metrics and per-query grids for GTM, insurance and finance, and developer activation. Without chart_id, lists them with titles and image links; with chart_id, also returns the image.

{
  "type": "object",
  "properties": {
    "chart_id": {
      "description": "e.g. overall_scorecard, finance_key_metrics, gtm_query_grid, time_steps_activation",
      "type": "string"
    },
    "use_case": {
      "default": "all",
      "description": "Filter the list",
      "type": "string",
      "enum": [
        "all",
        "gtm",
        "insurance",
        "finance",
        "setup"
      ]
    }
  },
  "$schema": "http://json-schema.org/draft-07/schema#"
}

How the benchmark was run and graded: query design, identical request parameters, 3 runs per query, latency timing, SimpleQA and FreshQA grading, the 10 metrics and the standards they follow, and limits.

{
  "type": "object",
  "properties": {}
}

How to rerun this benchmark on your own machine with your own API keys, including on your own queries. Nothing runs on this server.

{
  "type": "object",
  "properties": {}
}

Search the Web Search API Eval benchmark (results, questions, methodology, charts, how to run). Returns matching document IDs, titles and URLs.

{
  "type": "object",
  "properties": {
    "query": {
      "type": "string",
      "description": "What to look for"
    }
  },
  "required": [
    "query"
  ],
  "$schema": "http://json-schema.org/draft-07/schema#"
}

Full text of one benchmark document by its ID (for example question-F5, scorecard-finance, overall, methodology).

{
  "type": "object",
  "properties": {
    "id": {
      "type": "string",
      "description": "Document ID from search"
    }
  },
  "required": [
    "id"
  ],
  "$schema": "http://json-schema.org/draft-07/schema#"
}