As the sole PM across two of three squads, I don't hand testing off — I run a lot of it myself, and so do the six QAs I work alongside. The manual part is always the same: read a story, turn its acceptance criteria into a sensible set of tests, run them one at a time, write up the bugs, and post a summary back to Jira. So I built a tool that does the mechanical parts and leaves the judgment to a person — a full-stack app with Claude wired through five different jobs, and I designed every prompt, model choice, and failure path in it myself. This one has no dollar figure. It's here to show how I build applied-AI products.
You take a story — background, a handful of acceptance criteria, some corner cases someone flagged in refinement — and turn it into a concrete set of tests. Then you run them, one after another, keeping track of what passed. Then, for anything that failed, you write a bug ticket a developer can act on. Then you post a summary comment back to Jira so the team can see the story was tested.
None of that is hard. All of it is tedious — and the tedium is where it goes wrong. You write one test per AC even when three ACs describe the same behavior. You test in an order that makes you re-set-up state five times. You file a bug with half the reproduction steps missing. And the Jira summary never gets written, because the story's already merged.
I wanted the mechanical parts handled and my attention on the one thing that matters: did this actually work? So I built the companion — a tool that turns a pasted story into a structured, walkable test session and stays out of the way of the judgment.
I'm a PM, and I built the whole thing — the Next.js app, the schema, every prompt, the model choices, the deploy. That's the point of putting it here. The interesting decisions in an AI product aren't in the model; they're in how you shape the prompt, where you spend tokens, which model you trust with what, and what you do when the model returns garbage. I made all of those calls, in code.
Everything persists to a Postgres database, organized into projects and sessions — so a test run isn't a throwaway chat, it's a record you can come back to.
The app is deployed and running. Paste a user story, generate the plan, walk the tests, mark a couple as failed, draft a bug, and produce the Jira summary — the real flow, on live models, end to end.
The most important surface in this tool isn't a screen — it's the test-plan prompt. A naive version ("write tests for this story") produces exactly the low-value output I was trying to escape: one shallow test per acceptance criterion, in whatever order they happened to be written.
So the prompt encodes how a good tester actually thinks:
That's product design done in a prompt. The output quality lives or dies on those instructions, and getting them right was most of the work.
When you paste a blob of several stories, the obvious approach is to hand the whole thing to the model and ask for everything at once. That's slow, expensive, and fragile — one big generation that fails as a unit. I split it into two passes instead.
Cheaper (tiny pass-1 output), faster (parallel pass-2), and more robust (one story failing doesn't sink the batch). This is the kind of decision that doesn't show up in a demo but decides whether a feature is usable at scale.
Not every job needs the strongest model. Plan generation, bug drafting, and the summary all run on Claude Haiku — roughly three times cheaper than Sonnet and more than good enough for structured, well-specified output. Sonnet is wired in and one constant away if I ever decide a job needs the extra quality.
Choosing the cheapest model that clears the quality bar — and being able to prove it clears it — is exactly the eval-and-cost discipline I bring to AI features at work.
Every one of these calls asks the model for strict JSON, and every one then trusts that JSON to build a database record. Smaller, cheaper models occasionally wrap their JSON in a markdown fence, add a sentence of preamble, or leave a trailing comma. If you don't handle that, the feature breaks in production on inputs that worked fine in testing.
So the parsing layer is deliberately tolerant and self-correcting:
This is the part that decides whether a clever prompt survives contact with real input. I built it because I've watched AI features that demo perfectly fall over on the first weird paste.
It would have been easy to let the tool auto-mark tests or auto-file bugs. I didn't, on purpose — the same principle that governs the production AI features I ship at work. The model does the mechanical work of drafting; a person always makes the call.
The tool removes the typing. It doesn't remove the judgment. That line is the whole reason a tester can trust it.
This project has no metric attached, and I've kept it that way. It's a personal build, not a shipped feature with adoption I can measure. What it demonstrates is the how:
The hard part of an AI product is almost never the model. It's the prompt design, the token and cost decisions, the failure handling, and knowing exactly where the human has to stay in the loop. I make those calls in code — which is why, when I set a confidence threshold or review a prompt on a production feature, I know what I'm looking at.