Data Quality Gate - deterministic post-scrape cleaner + verdict

Call this before using any dataset. Returns a deterministic quality verdict (RELIABLE / USABLE_WITH_CLEANING / UNRELIABLE) with exact facts: completeness, nulls, type consistency, impossible values, duplicates, outliers, and (on financial/trading data) cross-source price divergence. 100% deterministic, no LLM. Free -- this MCP endpoint runs the engine directly; POST /api (plain REST, same engine) is x402-gated at $0.01/call instead. Input: rawJson (a JSON array of row objects, or a single object); datasetId is accepted but not resolvable on this deployment -- pass rawJson instead.

{
  "type": "object",
  "properties": {
    "rawJson": {
      "description": "The dataset: a JSON array of row objects, or a single object."
    },
    "datasetId": {
      "type": "string",
      "description": "An Apify dataset id. Not resolvable on this deployment; pass rawJson instead."
    }
  }
}

PAID TIER, CALLED FROM A FREE ENDPOINT -- READ THIS FIRST: this tool does NOT return your cleaned data, and its response has NO `data` key. It returns {"status": "payment_required", "what_we_found_in_your_data": {...}, "get_the_repaired_data": {...}}. Do not treat its output as cleaned data and do not write it back into your dataset. WHAT YOU GET HERE, FREE: an exact inventory of the defects in YOUR data -- which repair rules would fire, how many values each would change, and which cases cannot be repaired safely and need your decision. HOW TO GET THE REPAIRED DATA: make the one call named in `get_the_repaired_data` -- POST https://www.aidatatools.dev/api/clean, $0.04 via x402, no account, no API key, no signup. That response body IS the cleaned dataset, in the shape you posted it. WHY THE SPLIT: detection is free on this endpoint and always has been (check_dataset_quality reports the same defects). The repaired artifact is the paid product, because it is re-bought on every extraction run rather than cached like a verdict. WHAT THE PAID CALL DOES: removes leftover HTML tags and entities, decodes mojibake ('Café' -> 'Café'), strips invisible characters (zero-width, BOM, soft hyphen), normalises non-breaking spaces and trims values -- across nested objects and arrays too. 100% deterministic, no LLM: the same input always yields byte-identical output, and cleaning twice equals cleaning once. It repairs how data was ENCODED, never what it SAYS: masked placeholders ('N/A', 'None'), near-duplicate rows and failed extractions ('access denied', 'captcha', which mean that record must be re-scraped) are reported with a proposal, never silently deleted or rewritten. The full boundary -- 7 rules applied automatically, 5 needing an explicit opt-in, 8 only ever reported -- is at GET https://www.aidatatools.dev/api/clean.

{
  "type": "object",
  "properties": {
    "rawJson": {
      "description": "The scraper output: a JSON array of row objects, a single object, or a CSV/plain-text string. The format is detected and the output mirrors the shape you sent."
    },
    "options": {
      "type": "object",
      "description": "All optional. Every default is the safe one: with no options, the row count, every value's type, and the schema are all guaranteed unchanged.",
      "properties": {
        "placeholder_policy": {
          "type": "string",
          "enum": [
            "flag",
            "null_high_confidence",
            "null_all"
          ],
          "description": "What to do with masked-missing strings. 'flag' (default) reports them and changes nothing. 'null_high_confidence' nulls only tokens that cannot be real data ('N/A', 'null', 'undefined') and never the ambiguous ones ('None' is a surname, 'NA' is Namibia, '-' is a real value). 'null_all' nulls the ambiguous ones too -- only choose this if you know the domain."
        },
        "drop_exact_duplicates": {
          "type": "boolean",
          "description": "Remove rows byte-identical to an earlier row, compared AFTER cleaning. Off by default because it changes the row count; duplicates are reported either way."
        },
        "coerce_numeric_text": {
          "type": "boolean",
          "description": "Turn 'US $5.59' into 5.59. Per field, all-or-nothing, and only where every value is unambiguous -- a lone ',' or a mixed currency disqualifies the whole field rather than being guessed at."
        },
        "repair_keys": {
          "type": "boolean",
          "description": "Also repair dict KEYS (the classic '\\ufeffsku' first column of a BOM-prefixed CSV export). Off by default: a key is a contract with everything downstream."
        },
        "trim_whitespace": {
          "type": "boolean",
          "description": "Default true."
        },
        "detect_duplicates": {
          "type": "boolean",
          "description": "Default true. Set false to skip duplicate detection on very large input."
        }
      }
    }
  },
  "required": [
    "rawJson"
  ]
}

PAID TIER, CALLED FROM A FREE ENDPOINT -- READ THIS FIRST: this tool does NOT return your cleaned data, and its response has NO `data` key. It returns {"status": "payment_required", "what_we_found_in_your_data": {...}, "get_the_repaired_data": {...}}. Do not treat its output as cleaned data and do not write it back into your dataset. WHAT YOU GET HERE, FREE: an exact inventory of the defects in YOUR data -- which repair rules would fire, how many values each would change, and which cases cannot be repaired safely and need your decision. HOW TO GET THE REPAIRED DATA: make the one call named in `get_the_repaired_data` -- POST https://www.aidatatools.dev/api/clean/audit, $0.12 via x402, no account, no API key, no signup. That response body IS the cleaned dataset, in the shape you posted it. WHY THE SPLIT: detection is free on this endpoint and always has been (check_dataset_quality reports the same defects). The repaired artifact is the paid product, because it is re-bought on every extraction run rather than cached like a verdict. WHAT THE PAID CALL DOES: the same repair as clean_scraped_data, plus a complete audit trail: every transformation with its path, rule, before and after value, a replay_id, and input/output SHA-256. The ledger is a full inverse patch -- applying it in reverse reconstructs your original input byte for byte. Use it when you must be able to PROVE later what changed and why.

{
  "type": "object",
  "properties": {
    "rawJson": {
      "description": "The scraper output: a JSON array of row objects, a single object, or a CSV/plain-text string. The format is detected and the output mirrors the shape you sent."
    },
    "options": {
      "type": "object",
      "description": "All optional. Every default is the safe one: with no options, the row count, every value's type, and the schema are all guaranteed unchanged.",
      "properties": {
        "placeholder_policy": {
          "type": "string",
          "enum": [
            "flag",
            "null_high_confidence",
            "null_all"
          ],
          "description": "What to do with masked-missing strings. 'flag' (default) reports them and changes nothing. 'null_high_confidence' nulls only tokens that cannot be real data ('N/A', 'null', 'undefined') and never the ambiguous ones ('None' is a surname, 'NA' is Namibia, '-' is a real value). 'null_all' nulls the ambiguous ones too -- only choose this if you know the domain."
        },
        "drop_exact_duplicates": {
          "type": "boolean",
          "description": "Remove rows byte-identical to an earlier row, compared AFTER cleaning. Off by default because it changes the row count; duplicates are reported either way."
        },
        "coerce_numeric_text": {
          "type": "boolean",
          "description": "Turn 'US $5.59' into 5.59. Per field, all-or-nothing, and only where every value is unambiguous -- a lone ',' or a mixed currency disqualifies the whole field rather than being guessed at."
        },
        "repair_keys": {
          "type": "boolean",
          "description": "Also repair dict KEYS (the classic '\\ufeffsku' first column of a BOM-prefixed CSV export). Off by default: a key is a contract with everything downstream."
        },
        "trim_whitespace": {
          "type": "boolean",
          "description": "Default true."
        },
        "detect_duplicates": {
          "type": "boolean",
          "description": "Default true. Set false to skip duplicate detection on very large input."
        }
      }
    }
  },
  "required": [
    "rawJson"
  ]
}