Practice 01
QA for AI
Testing an AI feature is not testing. There is no deterministic pass, so confidence has to be expressed as a distribution across repeated runs. Someone has to produce it. We do, and we hand you the evidence pack rather than an opinion.
-
ReadinessWhere the system stands against the behavior you say it must hold to.
-
ValidationTesting against those bounds at scale, with quantified coverage and named gaps.
-
Assurance opsContinuous re-grading after every model, prompt, or data change.
Method
Confidence Is a Distribution, Not a Checkmark.
We generate synthetic ground-truth data with answer keys, run it repeatedly, and report the shape of the result: how often behavior lands inside agreed bounds, where it drifts, and which conditions were never covered.
Synthetic-first Ground Truth
Clean-room datasets with known answers, so grading isn't circular and no production data has to leave your environment to build the test set.
Vertical Packs
Insurance, healthcare, legal and financial variants carrying the regulatory flavour and the domain traps that generic evaluation sets miss entirely.
Trajectory Grading
For agents, the answer is not the only thing that can be wrong. We grade the path: tool calls, retries, escalation, refusal, and where authority ends.
What a confidence distribution looks like
Illustrative. Your distribution comes from your own runs. A single pass/fail number would report this system as working. The shape is the finding.
Failure taxonomy
What Actually Goes Wrong in Production.
Five patterns account for most of the AI incidents we are called in after. A generic evaluation suite catches roughly one of them.
- Silent Drift A model or prompt change shifts behavior on a subset of cases nobody re-tested. Suites still pass. Different system.
- Confident Wrong Answers The system answers in-scope questions correctly and out-of-scope questions with exactly the same confidence.
- Phantom Actions The agent reports completing an action it never completed. Common in scheduling, booking, ticketing, and claims flows.
- Untested Conditions Accented speech, background noise, adversarial phrasing, and very long context. Fine in the demo, absent from the test set.
- Boundary Erosion Multi-turn pressure walks the agent outside the limits it held on turn one, one small concession at a time.
Boundaries
What We Will Not Claim.
This matters more here than anywhere else, and vagueness is how vendors in this category get people hurt.
- We do not certify anything, and we are not a certification body.
- We do not assert compliance with any regulation on your behalf.
- We do not guarantee safety, correctness or production outcomes.
- We do not assume your production risk.
- We produce evidence under tested conditions with known gaps stated. You make the release decision.
Verification Gap Diagnostic
Seven questions, two minutes. A scored read on where your gap is widest, an estimate of your unverified merge ratio, and what to fix first, on screen before we ask who you are.
Three-week Readiness Assessment
We measure change volume against verified change, find where coverage claims and production reality diverge, and hand you a costed 60–90 day plan you can take to a board.
Thirty Minutes With a QA Lead
An engineering lead is in the room, not just a salesperson. Bring the number you have to hit and we will tell you whether it is reachable, including when it is not.