LLMs shouldn't write your numbers

The easy way to build an AI dashboard tool is to hand a language model the spreadsheet. It writes the queries, reads the results, and writes a summary with the numbers in it. It demos well. We chose not to.

In the design we're building, the model describes the dashboard, and code does everything that produces a number. Here's why, with the research behind it.

3 ways a model gets a number wrong

The query is wrong, and it still runs

Spider 2.0, a benchmark presented at ICLR 2025, tests text-to-SQL on 632 real enterprise workflows, with databases that often have more than 1,000 columns. In the paper, the best agent (built on o1-preview) solved 21.3% of tasks. On the older Spider 1.0 and BIRD benchmarks, GPT-4-based methods had reached 91.2% and 73.0%. On Spider 2.0's self-contained settings, Snow and Lite, the paper's best scores were under 24%.

Agents built for the benchmark have closed much of that gap since. As of October 17, 2026, the top scores on the official leaderboard are 96.70 out of 100 on Spider 2.0-Snow and 76.23 on Spider 2.0-Lite. But a wrong query often doesn't fail loudly. It runs and returns a plausible number. Even at the best Snow score, a miss rate of about 3%, a model that writes 7 queries a week gets at least one wrong in about 1 week in 5, and nothing says which.

Snowflake's engineering team, writing about its own text-to-SQL product, says fan traps and chasm traps (joins that count the same amount more than once) are more prevalent when a model writes the SQL.

The same question gets a different answer

Temperature 0 is supposed to make a model pick its most likely next token every time. The output still isn't repeatable. Thinking Machines Lab sampled 1,000 completions of one prompt at temperature 0 and got 80 different ones. Their diagnosis: server load changes the batch size, which changes the order of floating-point additions inside the model, and with it the result. With batch-invariant kernels they wrote, all 1,000 matched.

Anthropic's API reference, for the API we use, says results are not fully deterministic even at temperature 0.0. It also lists temperature as deprecated: models released after Claude Opus 4.6 accept only the default value of 1.0. So a model in the path at view time could show a different number on Tuesday than on Monday, with no change in the data.

The sentence misstates the number

DataTales, a benchmark from EMNLP 2024, pairs 4.9k financial market reports with the market data behind them. The authors gave models the data and an expert's report up to each number, then checked whether the model produced the number the expert wrote. No model in the main results reached 30%, even with fine-tuning; the best was 28.93%. The models in the main results (GPT-3.5 Turbo, GPT-4, and Llama 2) are older than today's. A 5-sample spot check on Claude 3 Opus and Claude 3.5 Sonnet pointed the same way, but treat the figures as dated. The mechanism hasn't changed: a model writing a sentence is predicting text, and nothing in that step checks the number against a calculation.

What the model does instead

Apart from suggesting metrics, the model in our design writes one thing: a specification of the dashboard, in a small typed JSON format we call the dashboard IR. It has a title, a goal (with audience, decision, and cadence), filters, and a list of tiles. Each tile is one of 5 kinds (kpi, trend, breakdown, table, or note), and tiles that show a number refer to a metric by name and version, such as paid_revenue@3. A whole tile can be as small as {"id": "t1", "kind": "kpi", "metric": "paid_revenue@3"}.

It leaves out 2 things on purpose: SQL and numbers. A note can say Paid revenue was {{tile.value}} this week, and the placeholder is filled from the calculation. A sentence can't get a number any other way.

In the pilot, a person does the compiler's job by hand, and a database engine still does the counting.

How code does the counting

In the product, a small compiler will turn the spec into SQL. Because the compiler writes every query, the same spec will always produce the same SQL.

DuckDB will run that SQL with 2 settings in place. enable_external_access is false, which per DuckDB's documentation stops queries from attaching database files, reading or writing files with COPY, or reading outside files with functions like read_csv. lock_configuration is true, so no setting can be changed after that point. DuckDB's own security guide calls settings like these defense in depth, not a substitute for proper sandboxing. The bigger protection will sit upstream: the SQL will come from our compiler, not from a model.

After launch, when you open a dashboard, the model won't be called at all. The same spec compiles to the same SQL, DuckDB runs it on the same data, and you get the same number.

What the model sees

To suggest metrics, the model gets your column names, their types, and your goal. Never the rows. A text column called status, a number column called amount, and the goal "see which clients are active" are enough to propose a metric. You approve every definition before anything is built.

We send those through Anthropic's API, and Anthropic says that by default it doesn't use inputs or outputs from commercial products like the API to train its models. Our security page covers what else we keep, and for how long.

Why combining tables can't inflate totals

A fan trap is how a query that looks right returns the wrong total. Say an order of $100 has 3 line items. Join orders to line items, sum the order amount, and the order counts once per line: $300. Looker's documentation shows 3 orders worth $124.84 that add up to $223.44 once joined to their items, and fixes it with symmetric aggregates, which count rows with the same primary key only once.

We use 2 rules instead. First, joins only go many-to-one: each row of the table being measured looks up at most one row in another table, the way a line item looks up its order or an invoice looks up its client. A join like that can't multiply the rows being added up. Second, each metric is aggregated on its own table before results are combined. Order revenue is summed over orders, line items are counted over line items, and only then are the two put side by side. An order of $100 with 3 line items is $100 in every tile.

The 6 rules

Everything above comes down to 6 rules.

1. Same data, same number. No model runs at view time, so the same spec on the same data always gives the same result.
2. No number without a receipt. Each number carries its definition, filters, rows used, rows left out, source, and the date the data runs through. See Anatomy of a receipt.
3. The AI never types a number. Sentences get numbers only through placeholders.
4. The AI never sees your rows. It gets column names, column types, and your goal.
5. The AI never touches your data. It describes the dashboard. A database engine does the counting.
6. Combining tables never inflates totals. Joins go many-to-one, and each metric is aggregated first.

The compiler is what we're building, and it isn't live yet. Today, in the pilot, a person follows these rules by hand. How it works shows the same rules from the customer's side.

The trade-off: you can't ask it an open-ended question and get a fresh query back. If you need that, a text-to-SQL tool may fit better, but check its numbers against a calculation you trust. Why dashboard numbers don't match lists what to look for.