Testing an AI Product Without Burning Credit: A Playwright Approach
Zephyr Cloud explains how it tests an AI course product built on The AI Platform by intercepting model calls while keeping the real UI and flow intact.
Building a course product with Kent C. Dodds on top of The AI Platform (TAP), Zephyr Cloud's team found that every chat step—routing, specialist replies, and a rubric evaluator that reads the whole conversation—triggers a real, billable model call. Testing five full exercises end to end the naive way meant costs scaling with every suite run. Instead of mocking the whole app or the network, they intercepted only the most expensive piece, the model, at the chat provider level, leaving routing logic, message plumbing, and the UI fully real.
This let Playwright tests drive the actual desktop app, create real chat rooms, and verify the rubric genuinely fills to 100%—with only the invoice missing. The key safeguard is strictness: the harness refuses to let any unscripted message reach the real provider, and tests explicitly assert the evaluator call was truly intercepted. The team even had to ship a fix after one unmatched message quietly leaked through and kept costing money on every run.
The broader lesson for engineers: testing AI products is now as much about cost as correctness. The fix is to find the single most expensive call in the stack, intercept it as close to the model as possible, and keep everything else genuinely running—preserving honest end-to-end coverage while decoupling cost from test count.