# Small local model handles half of database questions, cutting agent costs

A 7B model running on one graphics card answered about half of the questions sent to a costly AI agent. In the authors' tests this cut list-price spending by a fifth to a third, with no measurable drop in accuracy. The work is a preprint and has not been peer reviewed.

- Category: Efficiency
- Readiness level: III (Open source) on Science Underway's four-step scale, as proposed
- Published: 2026-11-16
- Editor: Stefan Perdal
- URL: https://scienceunderway.com/articles/small-local-model-handles-half-of-database-questions-cutting-agent-costs

## The study

LuthorSQL: A One-Pass Text-to-SQL Model as System 1 for a Slower LLM Agent (preprint, not peer reviewed). arXiv 2611.17780. https://arxiv.org/abs/2611.17780

## At a glance

- What it shows: A small model that runs on a company's own computer can answer the easier questions about its databases. It passes only the harder ones to an expensive AI service. In tests, this cut the service bill by roughly a fifth to a third, and accuracy did not measurably change.
- What it does not show: It does not show the savings in a real company with real invoices. The tests were small, and "no measurable change" is not proof that accuracy is the same.
- Why it matters: Many organisations want to let staff ask questions of their data in plain English. Sending every question to a hosted AI is slow and costly, and it moves sensitive questions off site. A local first filter could reduce all three.

## What the study shows

The authors built LuthorSQL, a system that turns a plain-English question into SQL, the language used to query databases. It does not write SQL word by word. Instead, a 7-billion-parameter model reads the question and the database layout once. It then fills in the blanks of a fixed query template. A rule-based compiler turns each filled template into SQL and works out how tables connect. The system runs up to 32 candidate queries and uses a second trained scorer to pick one.

On BIRD, a benchmark of large, messy real-world databases, the BIRD team ran it on their hidden test set. It got 68% of questions right, meaning its query returned the correct rows. It took a median of 0.84 seconds per question on one GPU. On BIRD's public development set it scored about 59%. That is below the 62.6% the authors measured for the model it was built on, which writes SQL word by word.

The main experiment pairs LuthorSQL with a Claude Opus 5.5 agent, an AI that explores the database with tools before answering. LuthorSQL answers alone when its scorer is confident. Otherwise the question goes to the agent, along with LuthorSQL's candidate answers. The routing rule and its threshold were written down before each test.

Across three held-out test sets, the local model answered about half the questions. Against the agent alone, list-price cost fell by about 30% on held-out BIRD questions and about 20% on two other public benchmarks. These figures treat local computing as free. When the agent was called directly through its API, on 227 new questions, cost fell by a third. Accuracy stayed within about one point of the agent alone, and no difference was statistically significant.

The authors also tested whether Opus had memorised the benchmarks. Given only a question and a database name, with no access to the data, it reproduced between about half and nine tenths of the official answer queries word for word. Even so, its accuracy as a tool-using agent held up when every table and column was renamed. It also held up on new databases built from data published after its training cutoff.

## Our reading

In our reading, the useful idea here is the gate, not the small model's raw score. LuthorSQL alone is clearly weaker than the agent: about 59% against about 77% on the authors' main development split. What makes it useful is that its confidence score tracks when it is right. That lets it hand off the hard questions and keep the easy ones. The authors also show the handoff threshold cost little accuracy with five other, weaker agents.

Consider an organisation asking 10,000 data questions a day. The authors put the agent's bill at over $70,000 a year at list price. If half those questions stay on a local GPU, the questions that stay local never leave the building. The money saved is less than half, because the hard questions that remain are the expensive ones. The tests suggest a fifth to a third.

The memorisation finding deserves attention beyond this paper. Public text-to-SQL benchmarks appear to be partly in Opus's memory. Scores from lightly prompted large models on these sets may overstate what they would do on a company's own data. The authors' remedy, renaming everything and building fresh databases, is a practical check others could copy. They tested only one model, though. The readiness rating is III on our four-level scale. Code and model weights are public, and a public benchmark result exists. No named organisation is reported using it.

The rest of the article (what it means, limits, what to watch) is free for members: https://scienceunderway.com/articles/small-local-model-handles-half-of-database-questions-cutting-agent-costs

## How this was made

An AI model read the study and wrote this article in the structure Science Underway uses. An editor reviewed it and approved it for publication. Check the original study before you rely on the details.

---
Source: Science Underway, https://scienceunderway.com/. Please link to the article when you cite it.
