OpenMark

How to write an OpenMark task: the YAML structure, all 18 scoring modes, and tips. Call this before writing a task. Pass a topic for the full detail on one subject (the app's Task Guide): fields, scoring, composite, chaining, documents, examples, best_practices, or a scoring mode name.

{
  "type": "object",
  "properties": {
    "topic": {
      "type": "string",
      "description": "Optional: fields, scoring, composite, chaining, documents, examples, best_practices, or a scoring mode name such as regex_match"
    }
  }
}

Check a task YAML the way OpenMark will before running it: structure, scoring settings, and the test limit of the user's plan. Free, saves nothing. Returns the problems to fix and warnings worth a look.

{
  "type": "object",
  "properties": {
    "task_yaml": {
      "type": "string",
      "description": "The full task YAML (a JSON object with the same fields also works)"
    }
  },
  "required": [
    "task_yaml"
  ]
}

Current OpenMark model ids with list prices ($ per 1M tokens), as text and as a structured list. Use these ids in run_benchmark. Append @low, @medium or @high to a reasoning model id to set its reasoning effort.

{
  "type": "object",
  "properties": {
    "search": {
      "type": "string",
      "description": "Filter by id, name or provider, e.g. 'claude', 'gemini flash'"
    },
    "max_output_price": {
      "type": "number",
      "description": "Only models at or below this $ per 1M output tokens"
    },
    "images_only": {
      "type": "boolean",
      "description": "Only models that accept image attachments"
    }
  }
}

Get an upload_uid and a one-hour upload link (no API key needed) for images, PDFs or text documents, with the exact curl command. Upload the files, reference them by file name in each test's attachments list, then pass the upload_uid to estimate_cost / run_benchmark.

{
  "type": "object",
  "properties": {
    "file_names": {
      "type": "array",
      "items": {
        "type": "string"
      },
      "maxItems": 20,
      "description": "Optional: the local file paths to upload, to get a ready-to-run command"
    }
  }
}

Credits a run_benchmark call would use, before spending anything: the expected total, the split per model, the balance needed to start, the worst case, and the user's balance. Same inputs and run settings as run_benchmark. Use it to drop an expensive model, or to compare settings (e.g. max_tokens), before running.

{
  "type": "object",
  "properties": {
    "task_yaml": {
      "type": "string",
      "description": "The full task YAML (a JSON object with the same fields also works)"
    },
    "models": {
      "type": "array",
      "items": {
        "type": "string"
      },
      "minItems": 1,
      "maxItems": 20
    },
    "upload_uid": {
      "type": "string",
      "description": "From prepare_attachments, so image and document tokens are counted"
    },
    "preset": {
      "type": "string",
      "enum": [
        "standard",
        "thorough"
      ],
      "description": "Like the app's presets: standard = 2 passes + snappy timeouts (the default), thorough = 3 passes + patient timeouts. Explicit stability_runs / timeout_profile override it"
    },
    "stability_runs": {
      "type": "integer",
      "minimum": 1,
      "maximum": 5,
      "default": 2,
      "description": "Passes per model; 2+ shows which models are consistent"
    },
    "max_tokens": {
      "type": "integer",
      "minimum": 0,
      "maximum": 16000,
      "default": 500,
      "description": "Output token limit per answer, 16 to 16000; 0 = no limit. Keep the default unless results show it was too low: a higher limit raises the credits needed to start. get_results flags models whose answers came back empty or near the limit; re-run those with a higher value"
    },
    "temperature_discovery": {
      "type": "boolean",
      "default": false,
      "description": "Find each model's best temperature for this task. Costs about 6x as much"
    },
    "temp_preference": {
      "type": "string",
      "enum": [
        "low",
        "high"
      ],
      "default": "low",
      "description": "With temperature_discovery: which temperature wins a tie"
    },
    "timeout_profile": {
      "type": "string",
      "enum": [
        "snappy",
        "patient",
        "custom"
      ],
      "default": "snappy",
      "description": "How long to wait per answer: snappy 45s, patient 90s, custom = custom_timeout"
    },
    "custom_timeout": {
      "type": "integer",
      "minimum": 120,
      "maximum": 600,
      "description": "Seconds per answer, with timeout_profile custom"
    },
    "fail_fast": {
      "type": "boolean",
      "default": true,
      "description": "Stop a model after repeated errors to save credits. Set false to let every model finish, e.g. to diagnose failures"
    }
  },
  "required": [
    "task_yaml",
    "models"
  ]
}

Save a task (YAML, see get_task_format) to the user's OpenMark account and run it on the given models with real API calls. Returns a job_id and the expected cost; poll get_results. Spends the user's credits; pass max_credits to only start within a budget. Optional run settings match the app's Configure & Run panel (preset, stability_runs, max_tokens, temperature_discovery, temp_preference, timeout_profile, custom_timeout, fail_fast).

{
  "type": "object",
  "properties": {
    "task_yaml": {
      "type": "string",
      "description": "The full task YAML (a JSON object with the same fields also works)"
    },
    "models": {
      "type": "array",
      "items": {
        "type": "string"
      },
      "minItems": 1,
      "maxItems": 20,
      "description": "Model ids from list_models"
    },
    "upload_uid": {
      "type": "string",
      "description": "From prepare_attachments, when tests use attachments"
    },
    "max_credits": {
      "type": "integer",
      "minimum": 1,
      "description": "Budget: if the expected cost is higher, nothing runs and you get the quote instead"
    },
    "preset": {
      "type": "string",
      "enum": [
        "standard",
        "thorough"
      ],
      "description": "Like the app's presets: standard = 2 passes + snappy timeouts (the default), thorough = 3 passes + patient timeouts. Explicit stability_runs / timeout_profile override it"
    },
    "stability_runs": {
      "type": "integer",
      "minimum": 1,
      "maximum": 5,
      "default": 2,
      "description": "Passes per model; 2+ shows which models are consistent"
    },
    "max_tokens": {
      "type": "integer",
      "minimum": 0,
      "maximum": 16000,
      "default": 500,
      "description": "Output token limit per answer, 16 to 16000; 0 = no limit. Keep the default unless results show it was too low: a higher limit raises the credits needed to start. get_results flags models whose answers came back empty or near the limit; re-run those with a higher value"
    },
    "temperature_discovery": {
      "type": "boolean",
      "default": false,
      "description": "Find each model's best temperature for this task. Costs about 6x as much"
    },
    "temp_preference": {
      "type": "string",
      "enum": [
        "low",
        "high"
      ],
      "default": "low",
      "description": "With temperature_discovery: which temperature wins a tie"
    },
    "timeout_profile": {
      "type": "string",
      "enum": [
        "snappy",
        "patient",
        "custom"
      ],
      "default": "snappy",
      "description": "How long to wait per answer: snappy 45s, patient 90s, custom = custom_timeout"
    },
    "custom_timeout": {
      "type": "integer",
      "minimum": 120,
      "maximum": 600,
      "description": "Seconds per answer, with timeout_profile custom"
    },
    "fail_fast": {
      "type": "boolean",
      "default": true,
      "description": "Stop a model after repeated errors to save credits. Set false to let every model finish, e.g. to diagnose failures"
    }
  },
  "required": [
    "task_yaml",
    "models"
  ]
}

Status of a benchmark job, and once completed the ranked results (score, stability, cost per run, latency), the tests each model missed with its answer, a recommendation for the chosen objective and quick picks for the others.

{
  "type": "object",
  "properties": {
    "job_id": {
      "type": "string"
    },
    "optimize_for": {
      "type": "string",
      "enum": [
        "accuracy",
        "cost",
        "speed",
        "stability"
      ],
      "default": "accuracy",
      "description": "accuracy (same order as the app's results table): best score, then cheapest, then fastest. cost / speed / stability: cheapest / fastest / most consistent model within `tolerance` points of the top score"
    },
    "tolerance": {
      "type": "number",
      "minimum": 0,
      "maximum": 50,
      "default": 2,
      "description": "Score points below the top score that still count as good enough"
    },
    "show_misses": {
      "type": "boolean",
      "default": true,
      "description": "List the tests each model got wrong, with the expected answer and the model's reply"
    }
  },
  "required": [
    "job_id"
  ]
}

The user's plan, credit balance and limits, and what a credit pack, Pro or Expert would unlock. Use it before large runs, or when a run is refused for credits or limits.

{
  "type": "object",
  "properties": {}
}

Watch the task of a completed job: when new or cheaper models ship, OpenMark re-tests up to 4 of them next to the user's pick and emails the user only when one beats it. Each check uses the user's credits at the normal rate.

{
  "type": "object",
  "properties": {
    "job_id": {
      "type": "string"
    },
    "model": {
      "type": "string",
      "description": "The model to keep as the pick (default: the best-scoring model). Use the model you recommended."
    }
  },
  "required": [
    "job_id"
  ]
}