Scoped and led the rollout of an AI feature into a $8B-revenue, multi-tenant enterprise platform. The hard part wasn't the model — it was making the tenant-security boundary structurally unbypassable, and staging a rollout around an honest 91% accuracy number instead of waiting for perfect. Read for: cross-functional risk ownership, AI product judgment, staged rollout under real stakes.
ScopeWorker runs contract and procurement workflows for companies like T-Mobile, where buyers award billions of dollars in contracts to suppliers and routinely hit business questions the platform had no way to answer — Is a specific buyer favoring one supplier? How much did competitive bidding save versus direct awards last year? Answering meant exporting raw data and rebuilding the calculation in Excel, every time. As product owner, I scoped and led the rollout of a natural-language query layer: a buyer types the question in plain English, and the system translates it into a live Databricks SQL query against a curated analytics layer, runs it, and returns the answer — no export, no spreadsheet, no engineering ticket.
The risk wasn't accuracy alone — ScopeWorker is multi-tenant, so an LLM free to write its own database queries could just as easily hand one client another client's contracts. We closed that boundary structurally, not with a prompt, tested the model against 57 hand-verified business questions, and launched to internal admins first — read-only, with the accuracy baseline disclosed — turning real usage into the feedback loop a perfect score would have made us wait for.
Roughly 900,000 contracts get created on ScopeWorker in a single year — together representing about $8B in annual procurement spend. That volume is exactly why the same handful of questions kept surfacing, not as hypotheticals but as real asks, over and over, from real people:
In practice, answering any of them meant someone exporting raw data out of the platform, dropping it into multiple Excel sheets, and building the calculation by hand — every time. Admins reporting platform usage up to T-Mobile's CXO and finance teams were doing this regularly just to answer routine finance questions like bidding savings or change-order spend by market. The same questions kept resurfacing in user interviews and product-launch conversations with both T-Mobile and Samsung — the ask was almost always some version of "can we just see this?" The honest answer was: not without an export and an afternoon in Excel.
At the scale we were operating at — nearly a million contracts a year, $8B in spend — the inability to ask and answer questions like these directly on the platform was a missing core enterprise capability, not a minor inconvenience. A platform processing that much contract volume was expected to have an equivalent depth of insight readily available. It didn't.
Instead of adding another static report for every new question — a cycle that would never keep pace with how many different questions kept surfacing — we built a layer that could answer questions it had never seen a report for.
ScopeWorker already had its own analytics tool, and extending it to cover this was on the table. But it was a legacy system — upgrading it enough to carry a natural-language layer was its own ~6-month project. Rather than gate this feature behind that rebuild, we built the natural-language layer on top of a new curated Databricks lakehouse instead, decoupling the launch from the legacy tool's timeline.
A buyer or admin types the question directly, in plain English, into a familiar chat interface. Behind that single sentence, the system reads the question and the asking user's context and translates it into a Databricks SQL query — the same query an engineer would have had to hand-write for a one-off report — executes it against a curated analytics layer, and returns a plain-language answer alongside the underlying numbers. Not a cached report, not an approximation: the real answer to the real question asked.
ScopeWorker's transactional data spans both relational (MySQL) and document (MongoDB) stores. Querying either one directly from a chat feature would risk latency and performance bottlenecks on systems that also have to serve the live product — the last thing we wanted was a slow business question degrading contract creation for everyone else. So instead of teaching the model two query languages against two production systems, we routed around the problem entirely: ETL pipelines continuously ingest both sources into a single, unified Databricks Lakehouse, and the natural-language layer only ever talks to that curated copy.
tenant_id clause isn't something the model chose to write — the system forces it onto every query before execution. More on why that had to be structural, not a prompt, below.No export. No Excel. No waiting on an engineering ticket for a question that might only ever get asked once.
One decision made this viable to ship rather than just prototype: we didn't build a chat UI from scratch. We built on top of OpenAI's existing chat interface — thread history, search, a familiar layout — so admins had zero learning curve, and engineering spent zero time reinventing a front-end that already exists. Every hour saved there went straight into the part that actually mattered: getting the query translation right.
ScopeWorker is multi-tenant — T-Mobile, Samsung, and every other client's data lives in the same underlying system. Letting an LLM freely generate and run database queries against that system introduced one non-negotiable risk: a buyer at Company A asking a question and getting back Company B's contracts.
This couldn't be handled as a prompt instruction telling the model to "only look at the user's own data" — a model can be talked past, misinterpret a query, or simply get it wrong under the wrong phrasing, and the cost of being wrong here isn't a bad answer, it's a data breach. So the boundary was built structurally instead of behaviorally.
The result: the model is free to be creative about how it answers a question, but structurally incapable of reaching outside the asking company's own data. Everything else in this project could ship at 91% and improve in production. This had to be closed before day one.
Given the stakes — these are the same kinds of cross-entity questions described above, now being answered by a model instead of a person — we needed an honest read on reliability, including whether the same question asked differently would get the same answer.
We tested 57 distinct business questions, each phrased 2–3 different ways with the same underlying intent — e.g., "How many contracts were created in the last 1 year?" vs. "Number of contracts created in the last 365 days?" For each attempt, we ran a hand-written query to establish the known-correct answer and compared it against the LLM-generated query's result.
The failures weren't random — they clustered on phrasing. The model could reliably answer a direct version of a question but sometimes failed on a semantically identical rephrasing. Given that these questions inform decisions about buyer conduct, bidding versus direct-award savings, and change-order approvals, an admin getting a different answer depending on how they worded the question wasn't an acceptable risk to launch broadly.
A 9-in-10 hit rate sounds strong until you notice the misses aren't spread evenly — they hide inside specific phrasings, invisible until an admin happens to word a question the wrong way. That inconsistency, not the raw percentage, was the actual risk.
Rather than shelve the project or wait for a perfect score, we launched to internal admins only, in read-only mode, with the accuracy baseline disclosed to them explicitly — turning real usage into a feedback loop to identify which phrasing patterns broke the model, ahead of a broader rollout.
We don't yet have usage logging on the tool itself, so this rating data is currently the closest thing we have to an instrumented adoption signal — closing that logging gap is one of the first things planned post-launch.
| Decision area | What we chose | Why we chose it |
|---|---|---|
| Interface | Reused OpenAI's chat architecture | Avoided building a UI from scratch; leveraged a familiar layout and chat history admins already knew. |
| Permissions | Read-only access only | Avoided any risk of data mutation while query-accuracy and validation guardrails were still unproven. |
| Multi-tenancy | Programmatic query injection, enforced structurally | Prompt-level instructions weren't good enough for a zero-tolerance risk; the tenant boundary had to be unbypassable, not just recommended. |
| Rollout scope | Internal admins first, accuracy disclosed | Delivered real value immediately while turning early usage into a feedback loop instead of waiting for a perfect score. |
| Validation approach | Manual comparison against 57 hand-verified questions, phrased multiple ways | Proportionate rigor for a small, admin-only beta — not a formal benchmark suite, but enough to catch a real failure pattern (phrasing sensitivity) before wider exposure. |
57 questions with 2–3 phrasing variants each was enough to detect that phrasing sensitivity was a real failure mode, but not enough to fully characterize it — which phrasings break, how often, and why. Given the chance again, I'd rely less on a pre-built test set and more on instrumenting real usage from day one, so we could learn which phrasings actually trip up the model from real admin behavior rather than from what we anticipated testing for.
Right now, an admin who wants the same answer regularly — say, the duplicate-work rate for a specific market — has to re-ask the chat every time, re-exposing that query to the same phrasing risk we found in testing. A natural next step is letting admins save a verified query and its visualization to a personal analytics page: once a question has been asked, translated, and confirmed correct, it becomes a durable artifact that gets re-executed directly going forward, rather than re-derived from language each time it's asked.
This depends on two things we don't have yet: usage logging, so we can tell which queries are actually common enough to be worth saving, and investment in the analytics platform itself, since the current stack isn't well-suited to hosting ad-hoc saved views.
Conceptually, this turns the chat tool from a one-off Q&A interface into a discovery layer that feeds a better, permanent analytics platform — the same pattern as the multi-tenant boundary: let the model be creative where creativity is safe, and make the trusted, repeated parts structural rather than re-derived from language every time.