500 Questions to Find Where an AI Assistant Breaks
HypothesisWhen the assistant skips retrieval, it fills the gap by improvising. Cutting tool skips should cut made-up answers too.
In 30 seconds
- Problem: the team said the AI assistant's made-up answers were fixed. Nobody had measured it.
- What I did: asked it about 500 questions and traced every tool call behind each answer.
- What I found: 67 made-up answers (13.4%), 34 answers given without calling the right tool (6.8%), and 21 off-topic answers (4.2%).
- What changed: the skipped-tool problem is partly fixed, the rest is in progress, and I proposed grey-box testing to find root causes.
The thinking, drawn
3 diagrams
What I'd Do Next
- Turn the 500 questions into a test suite that runs before every release, so no fix is ever declared done without a number.
- Track the three failure rates on one dashboard, release over release.
- Agree a release threshold with engineering: the assistant doesn't ship if made-up answers go above it.
"Fixed" without a number isn't fixed. And an AI that skips its tools still sounds completely confident, which is exactly why you have to measure instead of reading answers and trusting your gut.
The full deep dive
For interviewers who want every detail and source.
Context
The assistant helps support staff answer customer questions. It works by retrieval: it should call a tool, pull the right information from the company's own data, and answer from that. After a round of fixes, the team believed the hallucinations were gone. I'd been in the role for under a month, and the fastest way to be useful was to check.
How I Tested It
- Built the question set: about 500 questions. Two sets: questions staff had actually asked, and a second set I prepared with AI help to cover edge cases and tricky phrasing.
- Traced every answer: this was a black-box setup, so I could see the question, the answer and which tools were called, but not the system's internals.
- Ran three checks on each answer: did it call the right tool? Does the answer match the data it pulled? Is it even on topic?
- Sorted failures into three groups: made up, answered without the right tool, and off topic.
What I Found
| Failure type | Count | Share of 500 |
|---|---|---|
| Made-up answers | 67 | 13.4% |
| Answered without the right tool | 34 | 6.8% |
| Off-topic answers | 21 | 4.2% |
About one in seven answers was made up, on a system the team believed was fixed.
Why Black-Box Testing Wasn't Enough
Black-box testing tells you that an answer failed, not why. I could see the assistant skip a tool, but not whether retrieval returned the wrong data, the right data ranked too low, or nothing at all. So I proposed grey-box testing: give the test access to what was retrieved at each step, so every failure can be traced to its cause.
Outcome
- The skipped-tool problem is partly fixed with engineering.
- Made-up and off-topic answers are still being worked on.
- Grey-box testing is planned as the next step.
PM Deep Dive
How the three failure types map to standard RAG evaluation
| What I measured | Standard eval metric | What it tells you |
|---|---|---|
| Made-up answers | Groundedness (faithfulness) | Is every claim backed by what was retrieved? |
| Answered without the right tool | Tool-call accuracy | Did the agent fetch data at all, and from the right source? |
| Off-topic answers | Answer relevance | Did it answer the question that was actually asked? |
Metric tree
- North Star: staff questions resolved without escalating to a human. (My proposed North Star; the team tracks its own.)
- Input metrics: groundedness rate, tool-call accuracy, answer relevance.
- Guardrails: refusal rate and response time. An assistant that refuses everything scores zero hallucinations, so quality can't be judged without them.
Black box, grey box, white box
| Level | What you can see | What you can conclude |
|---|---|---|
| Black box (what I had) | Question, answer, tool calls | That it failed |
| Grey box (what I proposed) | Plus retrieved chunks and their ranking | Where it failed: retrieval, ranking or generation |
| White box | Plus prompts, model settings, code | Why, at the code level |
Root-cause hypotheses for tool skips, each testable with grey-box access:
- Tool descriptions are vague, so the model doesn't know when a tool applies.
- Retrieval returns nothing useful, so the model falls back on its own training data.
- The system prompt doesn't make tool use mandatory for factual questions.
Making it a release process, not a one-off
- Golden dataset: the 500 questions become a fixed regression suite, re-run on every prompt or model change.
- Severity tiers: a wrong price is a P0; a slightly awkward phrasing is a P2. The release gate is zero P0s, not a blended average.
- Scale with LLM-as-judge: an AI grader scores every answer, and a human spot-checks a sample each run to make sure the grader itself stays accurate.