Why AI visibility dashboards are mostly useless
AI search visibility trackers have become a $100M industry, but research shows most of these measurements are statistically meaningless noise dressed as precision.
Tools tracking brand visibility across ChatGPT, Claude and Google's AI features have exploded into an industry now estimated at over $100 million a year in spend. But a rigorous study by Rand Fishkin with Gumshoe.ai, backed by several arXiv papers, shows that the visibility percentages and ranking positions these dashboards report are largely meaningless. Running the same prompt hundreds of times produces the same brand list less than one percent of the time, with ordering, brand count, and content all shifting between runs.
The root cause is architectural: language models generate answers token by token with built-in randomness, and adding live web search doubles the variance through query fan-out and citation selection. The one stable signal Fishkin found was the 'consideration set' — whether a brand appears in the pool at all — but rankings and positions proved unreliable. Compounding this, most trackers query bare API endpoints lacking the memory, location data, account history, and system prompts that shape real consumer-facing chat experiences, meaning they measure a population of users that barely exists.
For engineers and marketers, the takeaway is that decimal-precision scores with confidence intervals create a false sense of scientific rigor over what is fundamentally noise. The piece calls this 'precision laundering': averaging a hollow measurement enough times to cancel out noise doesn't manufacture validity. Before paying for these dashboards, teams should question whether the thing being measured is actually measurable at all.