Half My Traffic Was Bots: A 4-Day Layered Defense Stack
A developer discovered headless bots executing JS made up half his traffic, then built a 4-day, five-layer defense using robots.txt, ISR, Redis, and Vercel BotID.
The maintainer of a Next.js/Vercel side project that scores and roasts GitHub profiles noticed a suspicious traffic surge in July and traced it to its real cause: roughly half of all requests came from large-scale scrapers running headless browsers on rotating datacenter proxies. These bots executed JavaScript, fired real analytics events, and injected fake 'google.com' referrers, while a burst from Alibaba Cloud IPs exhausted the database connection pool and knocked the API offline with 500 errors.
Rather than blocking all bots, the author adopted a cost-based classification principle: user-agents and referrers cost nothing to fake, but residential IP space, verified bot signatures, and cryptographic attestation do. robots.txt was configured to allow retrieval-oriented AI crawlers (like ChatGPT-User and PerplexityBot) while blocking training crawlers (GPTBot, ClaudeBot, CCBot, and similar).
On the API side, rate limiting was moved ahead of cache lookups, a Redis SET NX shield was placed in front of hot database writes, and input validation was hardened against malformed payloads. The heaviest crawled pages were converted from force-dynamic rendering to ISR, eliminating server cost for every bot hit. The most expensive, LLM-backed endpoint was protected with Vercel BotID, letting humans and self-identifying bots through while returning a helpful 403 with API documentation links to unverified impersonators — and analytics were cleaned by filtering out requests flagged via navigator.webdriver.
The episode is a practical reminder for engineers that bot traffic can no longer be reliably filtered using free-to-fake signals like user-agents or referrers, and that targeted infrastructure fixes — caching order, Redis-backed dedup, ISR, and edge-level bot classification — can restore both system performance and data integrity at relatively low engineering cost.