AI Testing Isn't One Problem: Selectors, Search, Streaming, and Review
An analysis of how AI-driven testing tools must separately address self-healing selectors, search quality, reranking, and streaming settings panels—each with distinct risks.
The term "AI testing" loosely bundles two very different activities: using AI to generate or repair tests, and testing an AI-powered product feature. Conflating them leads teams to ask whether a tool merely "has AI" rather than which decision is being delegated, what evidence is preserved, and how a wrong outcome gets caught.
Self-healing selectors can cut maintenance work, but a wrong inference lets a suite report success after clicking the wrong element. A trustworthy healing system logs the original and replacement locators, match evidence, confidence scores, and human approval—and fails visibly when the match is ambiguous. For apps with heavy DOM churn, the real fix isn't cleverer selectors but stable test IDs and accessible component contracts.
For AI-powered search and reranking, checking that results merely appear isn't enough. Interface correctness (filters, loading states) and result quality (relevance, consistency, safety) need separate evaluation. Rigid exact-order assertions break under legitimate model improvements, so threshold-based checks—required results in the top N, banned results excluded—hold up better while still catching real regressions.
AI help centers and streaming settings panels reintroduce classic frontend race conditions in new guises: escalation links, source citations, late-arriving model lists, and stale state after navigation all need end-to-end coverage. A strong AI-generated answer doesn't make up for a broken escalation path or an unstable settings panel.