A chatbot test lies when it measures something that looks like the answer instead of the answer. Two rules cover most of it: read the assistant's message, not the whole screen, and compare literally only what the model does not decide, leaving the free text out. The numbers below come from a chat assistant I tested in September 2026, in an internal system.
What the documentation says
Anthropic's guide to testing LLM applications recommends automating grading whenever possible, and gives exact match as the example for closed-category answers. It also says to build test cases that reflect real usage, including edge cases such as empty, overly long or ambiguous input. There is no recipe there for a chat UI, only the principle.
Playwright, the tool that drives the browser in tests, documents that before clicking it waits for the element to be visible, stable and enabled, and that assertions (checks like "this text appeared") retry until the condition is true. That solves waiting for the button. It does not solve waiting for the model's reply, which is text that grows a bit at a time.
First mistake: the test reads the footer
On September 2, 2026, my test said: "after the question, there is a text longer than 15 characters". It passed green without the assistant having answered anything. The reason was the fixed footer on the screen ("Enter sends…"), which comes after the question and is longer than 15 characters. The test measured the presence of text, not the answer.
The fix was to measure the bubble: the last element in the message area that does not belong to the user. And to wait for the "typing" indicator to disappear before reading, because reading earlier means reading an answer that does not exist yet.
Second mistake: asking the assistant for something its scope forbids
To get a short answer that is easy to check, I used to ask "reply with just ok". The assistant had a locked scope and refused, correctly. The probe question has to be inside the scope, with an answer you can verify: a value that exists in the test report, not a formatting instruction.
Third mistake: comparing the whole text
On September 8, 2026, I needed to know whether a change to the system had altered its behavior. I compared pairs of answers. The refusal that the prompt dictates for out-of-scope questions was identical in 21 of 21 pairs. The full answer, though, varied by up to 86% between one run and the next, almost all of it from a free-text offer at the end ("can I help with anything else?" and variations). A loose similarity threshold failed the test with no defect present.
What fixed it was splitting the two parts:
- The deterministic part, which the prompt pins down (the refusal sentence, the format), is compared with literal equality.
- The free part stays out of the equality check. If it matters, it gets its own check, such as "did it cite a number that exists in the data?".
All of this is measured on my case, on a single assistant.
What to do
- Measure the assistant's message, not the page. Identify the answer element and wait for the "typing" indicator to end.
- Ask probe questions from inside the scope, with a checkable answer.
- Separate what is fixed from what is free. Literal equality on the fixed part. On the free part, a specific rule (number present, source cited), not a similarity percentage.
- Include the case that must fail: an out-of-scope question, empty input, an injection attempt. A test that only sees the happy path passes on any system.
- When the test turns green, break something on purpose. Replace the answer with empty text and see whether the test fails. If it does not, it is not measuring the answer.
If your chat is already live and nobody knows whether the answers get better or worse with each change, that is the kind of work I do in production AI agents.