Small local model handles half of database questions, cutting agent costs
A 7B model running on one graphics card answered about half of the questions sent to a costly AI agent. In the authors' tests this cut list-price spending by a fifth to a third, with no measurable drop in accuracy. The work is a preprint and has not been peer reviewed.

AI illustrationWhat the study shows
The authors built LuthorSQL, a system that turns a plain-English question into SQL, the language used to query databases. It does not write SQL word by word. Instead, a 7-billion-parameter model reads the question and the database layout once. It then fills in the blanks of a fixed query template. A rule-based compiler turns each filled template into SQL and works out how tables connect. The system runs up to 32 candidate queries and uses a second trained scorer to pick one.
On BIRD, a benchmark of large, messy real-world databases, the BIRD team ran it on their hidden test set. It got 68% of questions right, meaning its query returned the correct rows. It took a median of 0.84 seconds per question on one GPU. On BIRD's public development set it scored about 59%. That is below the 62.6% the authors measured for the model it was built on, which writes SQL word by word.
The main experiment pairs LuthorSQL with a Claude Opus 5.5 agent, an AI that explores the database with tools before answering. LuthorSQL answers alone when its scorer is confident. Otherwise the question goes to the agent, along with LuthorSQL's candidate answers. The routing rule and its threshold were written down before each test.
Across three held-out test sets, the local model answered about half the questions. Against the agent alone, list-price cost fell by about 30% on held-out BIRD questions and about 20% on two other public benchmarks. These figures treat local computing as free. When the agent was called directly through its API, on 227 new questions, cost fell by a third. Accuracy stayed within about one point of the agent alone, and no difference was statistically significant.
The authors also tested whether Opus had memorised the benchmarks. Given only a question and a database name, with no access to the data, it reproduced between about half and nine tenths of the official answer queries word for word. Even so, its accuracy as a tool-using agent held up when every table and column was renamed. It also held up on new databases built from data published after its training cutoff.
Our reading
In our reading, the useful idea here is the gate, not the small model's raw score. LuthorSQL alone is clearly weaker than the agent: about 59% against about 77% on the authors' main development split. What makes it useful is that its confidence score tracks when it is right. That lets it hand off the hard questions and keep the easy ones. The authors also show the handoff threshold cost little accuracy with five other, weaker agents.
Consider an organisation asking 10,000 data questions a day. The authors put the agent's bill at over $70,000 a year at list price. If half those questions stay on a local GPU, the questions that stay local never leave the building. The money saved is less than half, because the hard questions that remain are the expensive ones. The tests suggest a fifth to a third.
The memorisation finding deserves attention beyond this paper. Public text-to-SQL benchmarks appear to be partly in Opus's memory. Scores from lightly prompted large models on these sets may overstate what they would do on a company's own data. The authors' remedy, renaming everything and building fresh databases, is a practical check others could copy. They tested only one model, though. The readiness rating is III on our four-level scale. Code and model weights are public, and a public benchmark result exists. No named organisation is reported using it.
Read the rest of this story
Membership is free. You get every article in full and the Issue every Sunday.
We send a sign-in link, no password needed. Already a member? Sign inReadiness assessment
| Level | Status | Evidence |
|---|---|---|
| I Lab only | Met | A working system with controlled experiments in a research setting. |
| II Benchmarked | Met | A BIRD hidden-test result of 68.03% EX, run by the BIRD team, plus public-benchmark results on BIRD dev, KaggleDBQA and Spider-Realistic. The test score is reported by the authors and not yet on the public leaderboard. |
| III Open source | Met | The paper links code, registrations, scorers and prompts on GitHub, adapters on Hugging Face (two-adapter and single-adapter) and the fresh evaluation set (research use). The backbone, OmniSQL-7B, is Apache-2.0. The adapter licences are not stated in the paper. The BIRD-Platinum labels used in validator training have no licence the authors could find. |
| IV In production | Not yet | No named organisation is reported using it in production. All cost savings are list-price estimates on benchmark questions. The cascade has not run on BIRD's hidden test set, and the agent was tested on SQLite only. |
What would move it up
To reach level 4, the authors need a documented deployment by a named organisation on its own databases. This should include invoiced cost, accuracy checks against real user questions and a rule for routing out-of-grammar questions without gold. Independent reproduction of the cascade on BIRD's hidden test set and on larger held-out sets with tighter intervals would also strengthen the case.
The study
Ibrahim Memon, Armin Karamzade, M. Hanif Jhaveri et al.. LuthorSQL: A One-Pass Text-to-SQL Model as System 1 for a Slower LLM Agent. arXiv, 13 November 2026. Read the paper
Notes
- Memon, Karamzade, Jhaveri and Fox, "LuthorSQL: A One-Pass Text-to-SQL Model as System 1 for a Slower LLM Agent", arXiv preprint, 13 November 2026, CC BY 4.0. Source
- Code, registrations, scorers, result files and prompts. Source
- Model adapters (two-adapter version; single-adapter version as luthorsql-bird-single). Licence not stated in the paper. Source
- Fresh post-cutoff evaluation set, for research use. Source
- Exact figures: BIRD hidden test 68.03% execution accuracy (1,789 questions); BIRD dev 59.39% (911 of 1,534); OmniSQL-7B 62.6% in the authors' runs; median 0.84 s per question on one H100 PCIe; $0.64 per 1,000 questions batched.
- Cascade at threshold 3.0: cost saving against Opus alone, local compute counted as zero, 30.1% pooled on RESERVED (348 questions, 8 databases) and 19.9% on KaggleDBQA plus Spider-Realistic (693 questions, 27 databases). Direct API on 227 fresh questions: 33.3% saving, 95.15% vs 95.59% accuracy (p = 1.0). Direct API on RESERVED: 35.3% saving, 2.0 points lower accuracy (p = 0.03).
- Memorisation: with no schema and no tool, Opus 5.5 reproduced 48.3% to 92.3% of gold queries verbatim across four public splits. Renamed schemas: 72.88% vs 72.32% for the routed agent. Fresh set: cascade 94.27%, Opus alone 97.36%.
- Other engines, BIRD dev, no retraining: 57.1% to 59.3% on Snowflake, BigQuery, MySQL, PostgreSQL and DuckDB, against 60.17% in SQLite on the same files. 8-bit laptop run: about 9 GB, 159 vs 158 correct on 239 RESERVED questions.
- Readiness: level III, open source available. Evidence: public code and adapters, and a BIRD hidden-test result. No documented production use by a named organisation.
Edited and signed by
Stefan PerdalEditor and publisher86 stories editedStefan read the study, set the readiness level and approved this story. An AI system found the paper and wrote the first draft.
How this was made
An AI model read the study and wrote this article in the structure Science Underway uses. An editor reviewed it and approved it for publication. Check the original study before you rely on the details.
How we workNo corrections

