What Screaming Frog will not tell you

Screaming Frog is a good crawler. Its extraction is trusted, its filters are thorough, and for a skilled analyst with an afternoon and a spreadsheet there is almost nothing it cannot eventually surface. That is not the criticism.

The criticism is that "eventually" is doing a lot of work in that sentence. A crawler ranks its findings from a catalogue. Screaming Frog labels over three hundred of them High, Medium or Low, and then says, in its own documentation, that those labels “lack context”. That is an honest thing to write, and it is the whole gap. The four checks below are ones where the interpretation is the whole point. Where knowing the number without knowing what Google does with it tells you nothing at all.

In 2025 Google published its own breakdown of the over-crawling problems people report to it. Faceted navigation accounted for 50%. Action parameters (the add-to-cart and add-to-wishlist links that perform something rather than showing something) accounted for another 25%. Three quarters of the problem, two causes.

A desktop crawler will happily show you a list of URLs containing a question mark. What it will not tell you is that one listing path has fanned out into four hundred crawlable variants from a few dozen products, that the same filter selection is linked in two different parameter orders and therefore counts twice, or that your add-to-cart links are still crawlable because rel="nofollow" was never a way to stop crawling. Only a robots.txt disallow is.

• We count the parameterized variants of every path, including ones the crawl only discovered but never fetched.
• Action parameters are flagged wherever they are linked, nofollowed or not.
• The same filter set linked in several orders is reported as the duplication it is.

Google clarified something in 2026 that had been true for years: Googlebot downloads the first 2 MB of a resource, header included, and ignores every byte after it. It does not reject the oversized file. It truncates the download and hands the fragment to indexing as though it were the complete page.

The median mobile page is now around 2.5 MB. On a heavy product page, that means the title, the canonical link or the JSON-LD block can sit past the line Google stops reading at. Perfectly valid markup that Google never sees. So we measure where those tags actually are, in bytes, and flag the ones that fall off the edge. Screaming Frog will tell you the document is over 2 MB, and marks it high priority. What no crawler we know of tells you is which tag fell off the edge, because page weight and tag position have always been treated as separate questions.

A page that sends an ETag or Last-Modified header can answer an unchanged re-crawl with a 304 and no body at all. Google asked for exactly this in its "Crawling December" series and reported that the share of cacheable fetches had fallen over a decade from 0.026% to 0.017%. We are collectively getting worse at it.

It matters more than it used to. Cloudflare found that over half of all AI-crawler traffic is re-fetching pages that never changed: pure overhead that HTTP caching would remove. Desktop crawlers can export the response headers, so the data is technically available, but none of them tell you that a missing validator is why your server is doing unnecessary work.

An honest tool should be as clear about what it refuses to claim. Crawl budget is not a ranking factor. Crawling is necessary for a page to appear in results, but it is not a ranking signal, and any tool promising higher positions from "more crawl budget" is selling a myth Google explicitly denies. For over 90% of sites, this whole topic does not apply.

Nor will we recommend llms.txt as leverage: Google has said it does not use it and has no plans to, and large log studies have found no correlation with AI citations. We will not suggest crawl-delay for Googlebot, which ignores it. And we will never advise blocking a URL in robots.txt while also putting noindex on it, that is not double protection, it is mutual cancellation, because a blocked page can never be crawled and the noindex is therefore never seen.