Addressing the Benchmark Problem in the Web Scraping Industry
An open-source benchmark has been developed to address issues in web scraping evaluations.
Evaluating web scraping companies reveals that their benchmarks are often unverifiable and designed to showcase success. In response, we created an open-source benchmark aimed at representing the frontier of web data. Our goal is to develop a comprehensive test set across major anti-bot providers and industries, and we welcome community feedback for improvement.