Ask-Data Trust Layer

Natural-language analytics that refuses to guess

Prototype
Aug – Oct 2026
AI PMData / AnalyticsEvaluationGovernance
79→5%
Silent-Error Rate
97%
Exec. Accuracy
0%
Over-refusal

Documents

Product TeardownPDF · 5 pages

How natural-language analytics will be won on trust: 3 vendors compared, 3 ranked product proposals, and the prototype as evidence.

Open the Product Teardown PDF

Problem Statement

Business users have questions that live in the data warehouse but depend on an analyst to write SQL. Natural-language-to-SQL demos look magical, but adoption stalls on one failure: a confident wrong answer. One wrong number in a QBR and users stop asking.

User Personas & Pain Points

Business Analysts / RevOps

Waiting on analysts; inconsistent metric definitions across teams

Data Governance Owners

Fear of ungoverned AI access exposing PII or inventing metrics

Data Engineers

Acting as the human SQL API for ad hoc requests

Solution

A semantic layer that resolves business terms to governed definitions and shows its assumptions, an engine that abstains on PII, writes, causal questions and undefined metrics, and policy enforced in the database authorizer so any engine — rules or LLM — inherits it.

Goals & Success Metrics

  • Make silent-error rate the north-star metric, tracked alongside accuracy
  • Keep over-refusal at 0% so safety doesn't cost usefulness
  • Gate release on silent-error rate below 2% on a customer's own eval set

Feature Set

Semantic Model: Governed metrics, dimensions and synonyms; every answer states the definition it used
Policy Guard: Read-only access, table allow-list, and PII columns blocked even through SELECT *
Graceful Abstention: Explains why it won't answer instead of guessing
Eval Harness: 38 hand-written questions with independent gold SQL, scored on result-set match
Scorecard UI: Task success, silent errors, over-refusal and latency, with the SQL behind every answer

Tools & Stack

PythonSQLiteSQLClaude APIHTML / JS

Outcomes & Results

Versus a keyword baseline: silent-error rate 79% → 5%, execution accuracy 17% → 97%, with 0% over-refusal. The eval set is small and author-written, so these numbers validate the harness rather than claim a benchmark; the two remaining failures (ratio metrics, trend questions) are next on the roadmap.

Want to discuss this project?

I'd love to walk you through the decisions, trade-offs, and what I'd do differently.