Where your crawl budget actually goes

Title, canonical or structured data past the 2 MB fetch limit

Move the title, canonical link and JSON-LD to the top of the document — Googlebot stops reading at 2 MB and ignores everything after it.

Googlebot reads the first two megabytes of a page's code and puts the rest down. The title, canonical or structured data of this page sit further than that. Everything beyond that does not exist for Google. The page has a title and a canonical, but Google does not see them, because it stopped reading earlier. Usually it is a huge block of styles or scripts pasted into the head, an image embedded in base64, or the whole application state written into the code. Slim the head: critical tags at the very top, heavy things into separate files.

Id

HEAD_ELEMENTS_BEYOND_FETCH_LIMIT

Fires at

> 2 MB

Fix estimate

60 min per page

Crawlable add-to-cart or wishlist links

Disallow the action parameter in robots.txt: these URLs perform an action and show no content of their own, so nothing is lost by keeping crawlers out.

The "add to cart", "save for later" and "compare" links are ordinary addresses a crawler can follow. And it does: Googlebot clicks every one of them, creating carts nobody needs along the way and reading thousands of addresses that differ by a single parameter. That wastes its time on your site and can load the server. Actions belong on buttons, and if they must be links, the parameter has to be blocked in robots.txt.

Id

ACTION_PARAM_CRAWLABLE

Fix estimate

20 min per page

Filters generate a very large crawlable URL space

Unless the filtered views need to rank on their own, disallow the filter parameters in robots.txt and leave one clean listing plus the product pages crawlable.

The filters in the shop create more addresses than you have products. Colour times size times price times sort order gives thousands of combinations, and each is a separate page for Google to read. The crawler has limited time for your site, and instead of reading new products it wanders through "red, size M, cheapest first, page 7". In Search Console this shows as a growing mountain of "discovered, not indexed". This is not a problem to fix page by page. Decide which combinations belong in Google because people search for them (usually a single filter, such as brand), and close the rest: a canonical to the category, a noindex, or a parameter block in robots.txt. A ready robots.txt block is on the action card.

Id

FACETED_URL_EXPLOSION

Fires at

≥ 25 variants per path

Fix estimate

45 min per page

HTML larger than the 2 MB Googlebot fetches

Trim the HTML below 2 MB — Googlebot truncates the download there and indexes the fragment as if it were the complete page.

The code of this page is larger than two megabytes, which is how much Googlebot fetches before it stops. Everything below that line does not exist for Google: the further content, the links, the footer. The page looks normal, because the browser fetches it all. Usually it is a product list without pagination, styles and scripts pasted into the code instead of separate files, or images embedded directly in the HTML. Move the heavy things into separate files and split very long lists.

Id

HTML_EXCEEDS_FETCH_LIMIT

Fires at

> 2 MB

Fix estimate

45 min per page

Same filters linked in different parameter orders

Emit filter parameters in one fixed order so the same selection always has a single URL.

Id

INCONSISTENT_PARAM_ORDER

Fix estimate

25 min per page

Query string uses a non-standard separator

Separate query parameters with & — commas, semicolons and brackets are not parsed as separators and turn one filter set into an opaque single value.

Id

NONSTANDARD_PARAM_SEPARATOR

Fix estimate

25 min per page

Internal search results are open to crawlers

Disallow the search path in robots.txt: every query is a new address, and a results page is not something anybody wrote.

The result pages of your internal search are open to crawlers. Every query is a new address, and nobody wrote those addresses: they are automatic lists, often empty or built from junk terms typed by bots. Google reads them, sometimes indexes them, and your site starts appearing in results for phrases you would not want, while spending crawler visits on pages without value. Block the search path in robots.txt and add a noindex on the results page; two lines that tidy up thousands of addresses.

Id

SEARCH_RESULTS_CRAWLABLE

Fires at

≥ 3 links on one page

Fix estimate

20 min per page

Critical head tags sit deep in the document

Keep the title, canonical link and structured data near the start of the HTML so they stay comfortably inside the 2 MB Googlebot reads.

Id

HEAD_ELEMENTS_LATE_IN_DOCUMENT

Fires at

> 1 MB

Fix estimate

30 min per page

No ETag or Last-Modified header

Send an ETag or Last-Modified header so unchanged pages can answer with a cheap 304 instead of the full body.

Id

MISSING_CACHE_VALIDATORS

Fix estimate

20 min per page