For one of T-Mobile's procurement programs, buyers purchased work across 540,000 cell sites — about 5,000 standard line items in the catalogue, plus custom ones created whenever the needed work didn't exist yet. Every time a buyer awarded a contract, they had to answer one question: has this work already been purchased for this site? The platform had no reliable way to answer it. A single site could carry years of history spread across contracts, so duplicate work got approved without anyone noticing — until billing.
We set out to add a simple duplicate check to the procurement workflow. It grew into a two-phase system — rules for the obvious matches, AI for the ambiguous ones — that cut contract awards by 9.5%. The order mattered: two-thirds of that came from the simple rules, before any AI ran. And because every catch could be traced to a cancelled contract, we could put a real number on it: $2.74M of duplicate work stopped before it was paid.
Answering that at the moment of award was harder than it sounds. A buyer working across hundreds of sites would have to open years of past contracts for the one site in front of them, read every line item, and decide whether any of it matched what they were about to buy — for every award, under deadline.
The history itself worked against them. The same job could have been entered by a different buyer, on a different contract, worded a different way. There was nothing to line up against, no single view of "what this site already has."
So in practice the check didn't happen. Buyers awarded the work and moved on — which is exactly how the same job got bought twice.
A duplicate that slips past the award stage doesn't stay a mistake on a screen. The contractor does the work once and bills against both contracts, and the client pays twice. By the time anyone notices, the money is already gone — catching it before the award was the only point where it was still free to fix.
Many line items had a standard ID and a fixed price. For those, detection was simple. When a buyer tried to award one, ScopeWorker checked whether that same line-item ID had already been purchased for the site. If it found a match, it surfaced the earlier order and asked the buyer to review it before continuing.
We shipped this in about a month. It cut contract awards by 6%.
The more telling number was what happened next: 74% of the contracts it flagged were cancelled. In most of those cases the buyer looked at the earlier order and decided the new one wasn't needed. A basic ID check was already stopping a real amount of duplicate work — and it settled the question of whether this was a genuine problem. These weren't hypothetical duplicates. They were making it all the way to a contract award before anyone caught them.
A cancelled contract isn't an estimate. It has a value recorded in the system. So every time a flag led to a cancellation, we could read the exact amount that would otherwise have been paid and add it to the total. That's the entire basis for the $2.74M — real cancelled contracts, not a model.
Not every line item had a standard ID or a fixed price. A large share of the work was variable-cost, entered as free text — and its name, description, and price shifted with the buyer, the supplier, the quote, the contract.
That broke the ID check completely. Two orders could describe the same job at the same site in entirely different words, with different prices and different IDs — and the rule would treat them as unrelated. The reverse happened too: two orders could read almost identically and be separate, valid jobs.
Phase 1 was good at matching data. This needed something that could read meaning.
We didn't replace Phase 1 with AI. The ID check was fast, cheap, and reliable — there was no reason to hand a model a job a simple rule already did better. So the two ran as layers, hardest-case-last:
The AI never cancelled anything. Its only job was to narrow thousands of past line items down to the few a person should actually look at. The decision stayed with the buyer — the model just found the right orders to put in front of them.
We didn't know yet whether AI could judge these orders well enough to be useful — and the live product is the most expensive place to find that out. So we built a small testing tool outside the platform, ran real line-item pairs through it, and reworked the approach in hours instead of release cycles. Phase 1 kept delivering value the whole time.
Our first version asked the model one broad question — "Are these two line items duplicates?" — for every kind of work. It wasn't good enough. Different work needs different judgment: a crane rental turns on capacity, duration, and whether an operator is included; an antenna install turns on equipment and sector count; fibre work needs to know that 400 metres and ~1,300 feet are the same distance.
So we moved from one generic prompt to guidance written for each type of line item. The results jumped — the model was finally weighing the details that actually mattered for the work in front of it.
The hardest part wasn't picking a model. It was agreeing on what counted as a duplicate — and that definition couldn't come from product and engineering alone. Two descriptions could look identical to us and be different jobs; two very different descriptions could be the same work.
So we sat with T-Mobile's procurement experts and reviewed real pairs of line items. They labelled each one — duplicate work, separate work, or unclear without more context. Those reviewed pairs became our golden dataset: a trusted set of answers we could measure every version of the AI against.
The disagreements were the most useful part. When experienced buyers read the same pair differently, it meant our definition wasn't clear enough yet — for them or for the model. Each hard case sharpened the rules.
| Signal | How much it counts | The rule |
|---|---|---|
| Purpose | Decisive | Are both orders trying to complete the same job? A different purpose means different work. |
| Scope | Supporting | Differences in equipment, capacity, duration, quantity, or labour separate two similar-looking jobs. |
| Price | Minor | Allowed to differ within reason — never decisive on its own, since quotes and costs move. |
Most of these experts had never been asked to sit as the judge in an exercise like this. A real part of the job was bringing them along — why the golden dataset mattered, how we'd break it into manageable batches, what we were building. Their disagreements weren't friction; they were the signal that showed me exactly where my definition was still fuzzy.
The AI introduced two kinds of mistake, and they pulled in opposite directions. A false alarm is two valid, separate jobs flagged as duplicates — it wastes the buyer's time, and if it happens too often, buyers stop trusting the queue and ignore it, which quietly kills the whole check. A missed duplicate is a real repeat the AI doesn't catch; it lets unnecessary work continue toward billing.
To choose a setting, I measured two things against the golden set at every threshold: precision — of the orders it flags, how many are genuinely duplicates (few false alarms) — and recall — of all the real duplicates, how many it catches (few misses). Raising the threshold pushes precision up and recall down. No setting maxes both.
I set it at 0.78 — not because two lines cross there, but because it's where precision is high enough that buyers keep trusting the list, while the daily volume of alerts stays inside the review team's capacity. It's an operating threshold, chosen for trust and workload and recalibrated as production data comes in — not a mathematical optimum the chart discovers on its own.
The two pull apart: a stricter setting buys trust (precision) by catching fewer (recall). 0.78 isn't where the lines cross — it's the point where precision is high enough that buyers keep using the queue while the daily alert volume stays inside the review team's capacity. An operating choice, recalibrated as production data lands.
The final experience was deliberately simple. When the AI found a likely duplicate, the buyer saw the new order beside the earlier one and compared them directly before deciding. Nothing was cancelled automatically.
The AI narrowed thousands of past line items down to a few strong matches. It didn't make the procurement call — it removed the manual work of finding the right orders to compare, and left the judgment where it belonged.
The two phases produced one clear result. The ID-based rules cut contract awards by 6%. The AI layer found another 3.5% among the variable-cost, free-text orders no rule could catch. Together: 9.5% fewer contract awards, stopped as duplicates before billing.
That split is the whole story. Almost two-thirds of the result came from simple rules — AI was never the starting point, and we didn't reach for it everywhere. We shipped the smallest thing that could create value, measured what it did, found where it broke, and used AI only for the part that actually needed to understand meaning.
And because every catch traced back to a cancelled contract with a real value, the impact wasn't a guess. $2.74M of duplicate work stopped before it was paid — a number pulled from the system, not modelled.
This looked like a data-matching problem at the start. It turned into a product, evaluation, and change-management problem — and the model was only one piece of it. The real work was:
The strongest AI product isn't the one that automates the most. Here it was rules for the obvious matches, AI for the ambiguous ones, and human judgment for the decision — and the thing that made it defensible wasn't the model, it was being able to prove, in dollars, that it worked.