LLM juries cut false SQL accepts without ditching human review
AI coding tools like GitHub Copilot, Cursor, and Replit now let developers describe what they want in plain language and get working SQL back. The bottleneck is not generation. It’s verification: how do you know the query is safe to run without routing every single one to a human reviewer?
New research from Muhammad Aziz Ullah and Abdul Serwadda proposes a specific answer: a unanimous LLM jury.
How the study was structured
The researchers benchmarked 15 open models on 82 MySQL text-to-SQL tasks using an execution-grounded protocol, meaning correctness was judged by actually running the queries rather than comparing them to a reference string. That approach gives a cleaner baseline on which models are genuinely strong versus which just look plausible.
From the six best-performing models, they built unanimous committees ranging in size from 1 to 6 members. Each committee sees the original prompt, the schema, and the candidate SQL. It accepts the query only when every member agrees it is correct. One dissenting vote and the query gets flagged for human review.
What the results showed
Single-model judges were inconsistent. The variance between models was high enough that picking the wrong one as your sole gatekeeper would leave real risk on the table.
Small unanimous committees of strong models reduced false accepts while still passing a meaningful proportion of good queries. The tradeoff is intentional: the design targets safety-first deployments where a false accept (a bad query that runs) costs more than a false reject (a good query that gets held for review).
Committee composition also mattered significantly. It was not enough to grab any combination of the top six models. Which specific models sat on the committee changed the outcome in ways the researchers measured using true positive rate, false positive rate, and Youden J.
The operator takeaway
If you are building any pipeline where AI-generated SQL touches a production database, the unanimous jury pattern is worth understanding. The key design choice is accepting more false rejects in exchange for fewer dangerous false accepts. For most production contexts, that is the right tradeoff. The study gives you a concrete benchmarking framework to evaluate which open models belong on your committee.
I’m founder of BizStack at Cagri Sarigoz LLC and a passionate advocate for entrepreneurs.
With over 14 years in tech, marketing, and AI, including my role as Head of SEO at CitizenShipper and co-founder of TaleBot at Intale AI, I’m dedicated to sharing genuine, useful product insights and tips.
At BizStack, I aim to cut through the digital noise to provide clear, actionable advice.
And more than all else, I’m a father to a (always) little girl and a husband.
Contact me at [email protected] for assistance.
