Genie demos really well. You point it at a catalog, type "What was production last month?" and a clean answer comes back in a few seconds. Then you put it in front of an actual operations team and you find out pretty quickly that capability was never the problem. Reliability is.
The Real Problem
Over the past year, my team shipped a number of production Databricks apps with Genie embedded: a transmission grid console for a large utility, an upstream well-intelligence hub for an oil & gas operator, midstream asset surveillance, load forecasting, and revenue-leakage detection in pharma distribution, among others. The same failure mode showed up in almost every one.
Ask Genie the same question two slightly different ways and you can get two different SQL statements, and sometimes two different answers. Once in a while it returns no SQL at all, or a confident-looking number that came off a join that was subtly wrong. In a demo, none of that matters. In production, one wrong number in front of a VP and trust in the whole platform takes a hit. A Genie that is usually right turns out to be harder to live with than a dashboard that is boringly right every time.
What We Tried First
The obvious first move: create a Genie Space, point it at the catalog, and hand users a chat box. It looked great in demos and got shaky in production. Answers were hard to explain, the UI would sometimes sit there spinning while a query ran, and we had no real way to tell a bad day from a bad question.
What Actually Worked
So we stopped treating Genie as the whole system and started treating it as one (very good) component inside a reliability layer we built around it. A few things moved the needle.
1. Separate generation from execution. We let Genie author the SQL, then we run that SQL ourselves against a governed SQL warehouse. The generated query becomes something we can log, inspect, and show the user when they push back. The answer stops being a black box and becomes reproducible and auditable.
2. Bounded, status-aware polling with a graceful fallback. Every Genie call polls the message and handles each state explicitly ( COMPLETED , EXECUTING_QUERY , COMPLETED_WITH_ERROR , FAILED ), caps how long it will wait, and falls back to a plain "I couldn't answer that, try rephrasing" message or a known-good view. The UI never hangs, and a dispatcher never sees a stack trace.
3. Constrain the semantic surface. We keep spaces narrow, one domain per space, with a curated set of tables, certified metrics, and explicit Space instructions. Genie is far more reliable answering questions over 30 well-described columns than 300 ambiguous ones. Honestly, most of our reliability gains came from removing tables, not adding them.
4. Measure it. We read system.access.audit (service aibiGenie ) to track query volume, success rate, the questions people actually ask, and who is getting value. We built a small Genie Space Manager on top of those logs so "Genie feels flaky" turns into a number on a chart instead of a hallway complaint. You can't improve a reliability you don't measure.
5. Watch it continuously, and route smartly. As we added more spaces, we put an Agent Supervisor in front of them: it reads the question, routes it to the right scoped space, and enforces a few guardrails before anything runs. Because the supervisor is served on Model Serving, we turn on Unity Catalog inference tables, so every request and response is logged to a governed Delta table automatically. That gives us the thing audit logs can't: real P50/P95 latency, and a base to run Lakehouse Monitoring plus a nightly LLM-as-judge eval on a sample of answers for actual correctness, not just a 200 status. When a score slips, we see it the next morning, not after a user complains.
One Honest Caveat
Genie is probabilistic, so we treat reliability as something you build rather than a switch you flip, and the good news is that it compounds. One nuance worth knowing: the HTTP-200 success rate in the audit logs tells you the call succeeded, not that the answer was correct. So we pair that number with a quick weekly human review of the top questions. Put that together with tightly scoped spaces, treating the generated SQL as reviewable code, and retuning Space instructions as the questions evolve, and Genie gets steadily, and measurably, more dependable over time.
Get that part right and Genie crosses a line no demo can show you. It goes from "neat" to the thing the operations team opens first thing every morning.
