Every check, written down.
The complete list of what a reading looks at, with the threshold behind each one. Generated from the crawler itself, so it says what the code does rather than what the marketing wishes it did.
Where your crawl budget actually goes
Google’s own breakdown puts faceted navigation at 50% of reported over-crawling and action parameters at another 25%. These checks measure that space directly: how many URLs your filters generate, which links only perform an action, whether an unchanged page can answer with a cheap 304, and whether the tags Google needs sit inside the 2 MB it reads.
Read the crawl-budget checks →What to allow through the firewall
A firewall that answers our reader with 403 makes your site look like a tiny healthy one: nothing is read, so nothing is found. If your readings come back with a handful of pages and a refusal, these are the addresses the crawl leaves from.
The crawl reads robots.txt first and honours it, fetches one page at a time per site, and slows down on its own when a site pushes back. It reads as an ordinary browser, so allow by address rather than by user agent.
By category
Whether a page can be indexed at all: canonicals, noindex directives, soft 404s and the size limits Googlebot works within.
What crawlers can reach and how much of it is worth reaching — robots.txt, sitemaps, pagination and the parameter space your filters create.
Titles, descriptions, headings and the substance of the page itself — what a searcher sees before they click, and what they find after.
Broken links, redirect chains, orphan pages and how internal link equity moves around the site.
Server response time, payload weight, image sizes, Core Web Vitals and the HTTP caching that decides what a re-crawl costs.
Pages that say the same thing twice — by fingerprint, and by meaning.
Viewport, JavaScript rendering and the gap between the HTML you serve and the page a crawler finally sees.
Hreflang: return links, self-references, language codes and the pages they point at.
What a keyboard and a screen reader run into: labels, landmarks, tab order and link text.
Mixed content and the response headers a browser expects from a site served over HTTPS.