Where your crawl budget actually goes
Google’s own breakdown puts faceted navigation at 50% of reported over-crawling and action parameters at another 25%. These checks measure that space directly: how many URLs your filters generate, which links only perform an action, whether an unchanged page can answer with a cheap 304, and whether the tags Google needs sit inside the 2 MB it reads.
Sources: Google crawling documentation, the “Crawling December” series (2024), “Inside Googlebot” (2026), and Gary Illyes on Search Off the Record (2025).
Title, canonical or structured data past the 2 MB fetch limit
Move the title, canonical link and JSON-LD to the top of the document — Googlebot stops reading at 2 MB and ignores everything after it.
- Id
- HEAD_ELEMENTS_BEYOND_FETCH_LIMIT
- Fires at
- > 2 MB
- Fix estimate
- 60 min per page
Crawlable add-to-cart or wishlist links
Disallow the action parameter in robots.txt — these URLs perform an action rather than showing content, so nothing is lost by keeping crawlers out.
- Id
- ACTION_PARAM_CRAWLABLE
- Fix estimate
- 20 min per page
Filters generate a very large crawlable URL space
Unless the filtered views need to rank on their own, disallow the filter parameters in robots.txt and leave one clean listing plus the product pages crawlable.
- Id
- FACETED_URL_EXPLOSION
- Fires at
- ≥ 25 variants per path
- Fix estimate
- 45 min per page
HTML larger than the 2 MB Googlebot fetches
Trim the HTML below 2 MB — Googlebot truncates the download there and indexes the fragment as if it were the complete page.
- Id
- HTML_EXCEEDS_FETCH_LIMIT
- Fires at
- > 2 MB
- Fix estimate
- 45 min per page
Same filters linked in different parameter orders
Emit filter parameters in one fixed order so the same selection is always one URL rather than several.
- Id
- INCONSISTENT_PARAM_ORDER
- Fix estimate
- 25 min per page
Query string uses a non-standard separator
Separate query parameters with & — commas, semicolons and brackets are not parsed as separators and turn one filter set into an opaque single value.
- Id
- NONSTANDARD_PARAM_SEPARATOR
- Fix estimate
- 25 min per page
Internal search results are open to crawlers
Disallow the search path in robots.txt: every query is a new address, and a results page is not something anybody wrote.
- Id
- SEARCH_RESULTS_CRAWLABLE
- Fires at
- ≥ 3 links on one page
- Fix estimate
- 20 min per page
Critical head tags sit deep in the document
Keep the title, canonical link and structured data near the start of the HTML so they stay comfortably inside the 2 MB Googlebot reads.
- Id
- HEAD_ELEMENTS_LATE_IN_DOCUMENT
- Fires at
- > 1 MB
- Fix estimate
- 30 min per page
No ETag or Last-Modified header
Send an ETag or Last-Modified header so unchanged pages can answer with a cheap 304 instead of the full body.
- Id
- MISSING_CACHE_VALIDATORS
- Fix estimate
- 20 min per page