The web scraping industry has a benchmark problem

These can’t all be true

Vendor marketing and published rankings, captured July 2026, uncropped.

Ask any web scraping vendor about their success rate and the answer lands somewhere between 98% and 100%. The numbers are not exactly lies. They are measurements of test sets each vendor chose for itself. We are a vendor too, so instead of adding one more trust-us number, we built the benchmark we wanted to exist: an open target set, an open harness, 15 providers, 6,750 live requests, and results anyone can reproduce or dispute.

Problem one: bias

When a web scraping vendor publishes a benchmark, they are marketing. There is nothing wrong with that, but it makes comparison nearly impossible. Each company controls which URLs get tested, how success is defined, how retries are handled, and what configuration competitors are run with, and nobody publishes the harness. A 99% from one vendor and a 98% from another describe two different tests, not two products. The problem is not dishonesty so much as unfalsifiability: there is nothing to check, which is how the screenshots above can all exist at once.

It goes deeper than first-party numbers. Two of the top results when you google, or ask a chatbot, for independent scraper benchmarks are Scrapeway and ScrapingTest. Scrapeway advertises independent benchmarks with “no affiliate links, no sponsors, just data”; ScrapingTest brands itself an “independent web scraping benchmark.” Each ranks a single provider first. Scrapeway’s domain traces back to Joam Intelligence, LLC, the company behind Scrapfly, its number one. ScrapingTest is registered to Batuhan Özyön, the founder of Scrape.do, its number one. Even the word independent is not a signal you can rely on.

Problem two: breadth

The genuinely independent benchmarks are scarce. Proxyway and AIMultiple were the only ones we could find published this year; both use affiliate links, and Proxyway tells vendors ahead of time what the test will be. They are still the closest thing the industry has to neutral measurement, and the bigger limitation is what they cover.

Proxyway has not run in 2026, and their suite covers 15 sites. They fire thousands of requests at each page, which is good statistics but narrow coverage: with roughly one site per anti-bot vendor, a single leniently configured deployment becomes the entire grade for DataDome, Kasada, or Akamai. Their own 2024 write-up said as much about Indeed, which is protected by Shape: “We either failed to trigger it or Indeed is using a lenient configuration.”

AIMultiple is much better. They ran in May of this year and cover quite a few sites, but they report anti-bot results only for Cloudflare, PerimeterX, and DataDome. Their target list includes Akamai-protected sites without reporting Akamai as a category, and Kasada and Imperva are not touched at all. Industry coverage has the same shape: no airlines and flights, financial market data, news publishers, live-event ticketing, electronic components, telecom carriers, automotive listings, healthcare and drug pricing, government sites, or B2B software reviews.

So we built one anyone can run

The Web Data Frontier Benchmark is an open-source suite of 90 real sites - Amazon, Walmart, Zillow, Ticketmaster, and 86 others - run through one fixed harness that gives every provider identical URLs. The pass criteria are binary: the request must come back successful and the body must contain a known marker, like a product title, that would not appear on a block page. Five attempts per site, per provider. The target list, the success markers, and the per-provider adapters are all in the repo, and a run emits the machine-readable output.json that every number in this post is generated from.

We are not the first vendor to try this. Firecrawl open-sourced a similar effort about seven months ago, scrape-evals, with a public write-up of the results. Since then the repository and the blog post have been taken down, and what remains visible is the CI workflow that now runs the evals privately. We mention it as precedent rather than criticism - keeping a public benchmark current is real, ongoing work.

Methodology

The set spans every major anti-bot vendor (Akamai, Cloudflare, DataDome, Kasada, PerimeterX, AWS WAF) along with sites that run their own defenses, like Google, LinkedIn, and Temu. On each site we target a high value page: a product page, search results, listings, or reviews - the pages people actually scrape.

Every provider is run in the strongest anti-bot mode it offers - the premium or stealth tier, not the default - and the exact request shape for each one lives in an adapter file in the repo, so the configuration is auditable rather than taken on faith. Requests go out live with a 90 second timeout, 5 attempts per page, and at most two concurrent requests per provider with 500ms between dispatches, so no provider is rate limited by test pressure and a miss is attributable to the target rather than the harness.

A pass requires more than a successful status code. A 200 that returns a captcha interstitial or an empty JavaScript shell is a failure: the response body must contain a marker - a product title, a listing address - that only appears on the real page. An error status, a timeout, or a body without the marker all count as failures. The run makes real, billable API calls against every provider; we bought the competitor credits ourselves.

Results

The full leaderboard, verbatim from the run’s output.json (completed July 15, 2026 - treat every number here as a snapshot of that date). The range is wide: success rates span 33.3% to 95.8%, with the middle of the field clustered between 65% and 80%. None of these numbers look like a marketing page, ours included; against targets selected to be difficult, they should not.

Success rate and latency only make sense read together, because latency here is the mean across all attempts, failures included. Browserbase posts the fastest average at 2.43s largely because it fails fast on half its requests - speed means little when the payload is a block page - while Decodo averages over 31 seconds per attempt to pass 67.8%. The upper left corner is the one that matters: String averages 9.70s while passing 95.8% of requests.

Success by anti-bot vendor

Averages hide where each provider actually breaks, so the same run sliced by the anti-bot system defending each target is more informative. Kasada separates the field most sharply: ScraperAPI and Scrapingdog passed 0% of Kasada-protected requests and ScrapingBee 6.7%, while four providers cleared it at 93% or above. Cloudflare shows a similar split, with Nimble at 15.4% and Browserbase at 10.8%. String passed 100% of requests against DataDome, AWS WAF, and Kasada targets; our weakest category was PerimeterX at 86.7%. One caveat to read with the chart: Kasada covers three targets and AWS WAF six, so a single site moves those columns more than it moves DataDome’s nineteen.

Success by industry

Slicing by vertical shows how unevenly the frontier is distributed. News and finance pages are broadly reachable, while live-event ticketing is the hardest vertical in the set: across Ticketmaster, SeatGeek, and StubHub, Decodo and ZenRows passed 0% of requests and Bright 6.7%. Social platforms split the field in a different place - Firecrawl passed 10% there while clearing real estate at 97.1%. This is the case for breadth: a single overall number flattens exactly the differences that decide whether a provider fits your use case.

Where even we failed

An honest benchmark cuts both ways, so here is our own failure ledger for this run: 19 of 450 requests. Two targets account for half of it. Temu went 0 for 5 - it serves a custom interactive captcha that we do not yet solve (support is coming to the Web Access API). Skyscanner also went 0 for 5 behind PerimeterX. The remaining nine misses were scattered timeouts and rate limits across five sites we otherwise passed, including Instacart and Crunchbase.

Numbers like these are the entire point of the benchmark. A published 100% means the test set stopped being hard; the failures are what tell you where the frontier actually is.

Limitations

A few things to read this with. It is a snapshot in time: anti-bot defenses and provider capabilities both change monthly, and these numbers will drift. Ninety sites is far more coverage than the independent benchmarks manage, but it is not the whole web. And while every provider was run in its strongest anti-bot mode through its standard API, a provider might do better on a given site with hand-tuned, site-specific configuration than any general benchmark can show.

Then the obvious one, in plain words: we built this benchmark, and we sell the product that ranks first in it. We have tried to answer that the only way that actually works - by publishing the harness, the targets, the success criteria, and our own failures - but the remedy for any remaining doubt is not to take our word for it. It is to run the suite.

Reproduce it, or prove us wrong

The quick start in the repo takes your own API keys and emits an output.json in the same format behind every chart on this page. Fair warning: a full run makes thousands of billable API calls across every provider you test.

If we misconfigured your product, open a pull request against your adapter and we will rerun it. If there is a site you think belongs in the set, open an issue - we are happy to run suggested targets and add them. And if your numbers come out different from ours, publish them; that is what an open harness is for.

Cheers,
String team