Crawling

Homepage answers a server error

Get the homepage answering 200 first: a homepage returning 5xx makes search engines slow their crawling of the whole site, not just this page.

Id

HOMEPAGE_SERVER_ERROR

Fix estimate

5 min per page

Crawlable add-to-cart or wishlist links

Disallow the action parameter in robots.txt — these URLs perform an action rather than showing content, so nothing is lost by keeping crawlers out.

Id

ACTION_PARAM_CRAWLABLE

Fix estimate

20 min per page

Linked page blocked by robots.txt

Remove the robots.txt block if the page should appear in search; if the block is intentional, no fix is needed.

Id

BLOCKED_BY_ROBOTS_BUT_INDEXABLE_INTENT

Fix estimate

8 min per page

Filters generate a very large crawlable URL space

Unless the filtered views need to rank on their own, disallow the filter parameters in robots.txt and leave one clean listing plus the product pages crawlable.

Id

FACETED_URL_EXPLOSION

Fires at

≥ 25 variants per path

Fix estimate

45 min per page

Page did not answer

The page sent nothing within the time limit twice in one reading, once at the crawl's pace and once on its own: find what makes this URL slow, a crawler gives up long before a reader does.

Id

PAGE_NOT_ANSWERING

Fix estimate

20 min per page

Page answers 404

Restore the page or redirect it to the nearest live one, then fix whatever still points at it.

Id

PAGE_NOT_FOUND

Fix estimate

10 min per page

Page answers a server error

Look at the server log for this URL: a 5xx that lasts is read as the page being gone.

Id

PAGE_SERVER_ERROR

Fix estimate

20 min per page

Redirect with nowhere to go

A 3xx must carry a Location header; without one the browser shows a blank page and a crawler follows nothing.

Id

REDIRECT_WITHOUT_TARGET

Fix estimate

5 min per page

Relative links nest the same path deeper and deeper

Start internal hrefs with a slash (or the full URL): a bare href="stelaze" resolves against the current page’s path, so every page spawns a new nested address that does not exist.

Id

RELATIVE_LINK_TRAP

Fix estimate

30 min per page

robots.txt blocks the whole site

Disallow: / under User-agent: * keeps the entire site out of search. Confirm it is deliberate, or remove it today.

Id

ROBOTS_TXT_BLOCKS_EVERYTHING

Fix estimate

15 min per page

The site refused our reader

Nearly every address answered 401 or 403 to us, so this reading is empty. Allowlist our crawler by its egress address, or the next reading comes back blank as well.

Id

SITE_REFUSES_CRAWLER

Fix estimate

30 min per page

Session id in the address

Keep the session in a cookie: in the address it turns every visit into a new page with the same content.

Id

URL_HAS_SESSION_ID

Fix estimate

25 min per page

AI answer engines are blocked

Decide whether the answer engines may read the site: this block is often inherited from a template or a CDN preset rather than chosen.

Id

AI_CRAWLER_BLOCKED

Fix estimate

15 min per page

Date archive generates endless addresses

Stop the archive generating a page per day for ever: cap the range, or keep the generated addresses out of the crawl.

Id

CALENDAR_TRAP

Fix estimate

30 min per page

Same filters linked in different parameter orders

Emit filter parameters in one fixed order so the same selection is always one URL rather than several.

Id

INCONSISTENT_PARAM_ORDER

Fix estimate

25 min per page

Non-200 URL listed in sitemap

Id

NON_200_IN_SITEMAP

Fix estimate

4 min per page

Query string uses a non-standard separator

Separate query parameters with & — commas, semicolons and brackets are not parsed as separators and turn one filter set into an opaque single value.

Id

NONSTANDARD_PARAM_SEPARATOR

Fix estimate

25 min per page

Page answers 410

Confirm the page is meant to be gone for good, and remove the links and sitemap entries that still list it.

Id

PAGE_GONE

Fix estimate

10 min per page

The site refused the crawler

Raise the rate limit for search crawlers, or slow the reading down: pages refused with 429 are read as unavailable.

Id

PAGE_RATE_LIMITED

Fix estimate

20 min per page

Page refuses the crawler

Check whether the page is meant to be public: a login wall on a page you want indexed keeps it out of search entirely.

Id

PAGE_UNAUTHORIZED

Fix estimate

15 min per page

Pagination points to a broken or non-indexable page

Id

PAGINATION_BROKEN

Fix estimate

6 min per page

robots.txt hides the stylesheets or scripts

Let crawlers fetch the CSS and JavaScript: the page is rendered before it is judged, and a blocked stylesheet means it is judged unstyled.

Id

ROBOTS_TXT_BLOCKS_RESOURCES

Fix estimate

20 min per page

Internal search results are open to crawlers

Disallow the search path in robots.txt: every query is a new address, and a results page is not something anybody wrote.

Id

SEARCH_RESULTS_CRAWLABLE

Fires at

≥ 3 links on one page

Fix estimate

20 min per page

Maintenance page never says when to come back

Send a Retry-After header with the 503 so a crawler knows when to return instead of guessing, and guessing long.

Id

SERVICE_UNAVAILABLE_NO_RETRY_AFTER

Fix estimate

5 min per page

Parameterized URL without canonical

Id

URL_WITH_PARAMETERS_NO_CANONICAL

Fix estimate

5 min per page

Meta refresh used as a redirect

Replace the meta refresh with a proper 301 server redirect.

Id

META_REFRESH_REDIRECT

Fix estimate

5 min per page

URL is long or uses unfriendly characters

Prefer short lowercase URLs with hyphens; avoid uppercase, underscores and spaces in new URLs.

Id

NON_SEO_FRIENDLY_URL

Fires at

> 120 characters

Fix estimate

10 min per page

Page answers an unexpected status

Check what this URL is meant to answer, because search engines treat anything outside 200 and 301 as a page they cannot keep.

Id

PAGE_UNEXPECTED_STATUS

Fix estimate

10 min per page

Pagination only links forwards

Add the matching rel=prev on the page being pointed at, so the sequence can be walked in both directions.

Id

PAGINATION_SEQUENCE_ONE_WAY

Fix estimate

15 min per page

robots.txt has lines nothing reads

Fix or remove the unrecognised lines: a misspelled directive is skipped in silence, so the rule you wrote never applied.

Id

ROBOTS_TXT_INVALID_LINE

Fix estimate

15 min per page

No robots.txt

Serve a robots.txt, even a permissive one: without it every crawler guesses, and the file is where the sitemap is announced.

Id

ROBOTS_TXT_MISSING

Fix estimate

15 min per page

robots.txt does not name the sitemap

Add a Sitemap line to robots.txt: it is the one place every crawler looks for it without being told.

Id

ROBOTS_TXT_NO_SITEMAP

Fix estimate

10 min per page

Page missing from sitemap

Id

SITEMAP_MISSING_URL

Fix estimate

4 min per page

Double slash in the path

Collapse the repeated slash and redirect the old form: both addresses answer, so both get crawled.

Id

URL_DOUBLE_SLASH

Fix estimate

10 min per page

Same parameter twice in one address

Emit each parameter once: repeated keys make addresses that differ only by order of the same values.

Id

URL_REPEATED_PARAMETER

Fix estimate

10 min per page

Address is very long

Shorten the path to the words that identify the page: an address nobody can read is an address nobody links to.

Id

URL_TOO_LONG

Fix estimate

10 min per page

Address contains unsafe characters

Replace spaces and punctuation with hyphens: the address works in a browser and breaks everywhere it is pasted as text.

Id

URL_UNSAFE_CHARACTERS

Fix estimate

10 min per page