Crawling

Homepage answers a server error

Get the homepage answering 200 first: a homepage returning 5xx makes search engines slow their crawling of the whole site, every other page included.

A homepage answering a server error is a locked front door. Googlebot starts most visits here, and when it meets a 5xx it not only misses the homepage, it slows its visits to the whole site, assuming the server is in trouble. Every other page then waits longer to be refreshed, and a reader who typed your company name sees an error instead of your offer. This is one of the few things in this panel that really is for today. Check whether the homepage answers at all, and if it answers only sometimes, the server log will say what knocks it over.

Id

HOMEPAGE_SERVER_ERROR

Fix estimate

5 min per page

The address does not resolve

The domain name has no address in DNS, so nobody reaches the site, search engines included: check that the domain is paid for, that its name servers are the right ones, and that the A or AAAA record for this host exists.

Id

HOST_NOT_RESOLVING

Fix estimate

15 min per page

robots.txt blocks the whole site

Disallow: / under User-agent: * keeps the entire site out of search. Confirm it is deliberate, or remove it today.

The robots.txt file tells every crawler: "go nowhere." That is a "closed" sign on the door of the whole site. Google will not read a single page, and the ones it holds in the index it begins to show without a description and, in time, to withdraw. Almost always this is left over from a staging copy, where the block protected against accidental indexing, carried to production with the files. If the site is meant to be in Google, this is for today: one line in one file.

Id

ROBOTS_TXT_BLOCKS_EVERYTHING

Fix estimate

15 min per page

A search engine is shut out

robots.txt refuses a search engine by name: narrow the rule to the private paths, or the site cannot be found at all.

The crawler of one of the search engines is barred from the site. This is a crawler that builds a results index, Googlebot, Bingbot or kin, and the block on it has nothing to do with AI crawlers. A barred crawler reads not a single page, so for that search engine the site ceases to exist over time. Bing also powers search in ChatGPT and DuckDuckGo, so a blocked Bingbot is more than one search engine. Usually it is a rule added while blocking "all bots except Google", or a ready-made robots.txt copied from the internet. Check whether anyone meant it.

Id

SEARCH_CRAWLER_BLOCKED

Fix estimate

15 min per page

The site has moved to another domain

Add the site under the address it now lives at; this address only forwards there, so nothing here can be read or improved.

The address you added answers with a redirect to a different domain, and everything a reader or a crawler gets comes from that other site. There is nothing to improve under this address: it forwards, and that is all it does. If the move is yours, add the site under its new domain and read it there. If it is not, the domain has lapsed or been taken over, and the redirect is somebody else's.

Id

SITE_MOVED_TO_OTHER_HOST

Fix estimate

5 min per page

Crawlable add-to-cart or wishlist links

Disallow the action parameter in robots.txt: these URLs perform an action and show no content of their own, so nothing is lost by keeping crawlers out.

The "add to cart", "save for later" and "compare" links are ordinary addresses a crawler can follow. And it does: Googlebot clicks every one of them, creating carts nobody needs along the way and reading thousands of addresses that differ by a single parameter. That wastes its time on your site and can load the server. Actions belong on buttons, and if they must be links, the parameter has to be blocked in robots.txt.

Id

ACTION_PARAM_CRAWLABLE

Fix estimate

20 min per page

Answer engines are shut out

robots.txt refuses the crawlers that cite sources in AI answers: decide whether that is what you meant, because it is usually inherited from a template or a CDN preset.

The crawlers that answer questions in ChatGPT, Perplexity and similar assistants are barred. The distinction matters: some crawlers train models, others look for answers live when someone asks a question. Blocking the second kind protects nothing from training; it only strikes you from the answers an assistant gives with a named source. If you want to be cited, let the answer crawlers in, and decide about the training crawlers separately.

Id

ANSWER_CRAWLER_BLOCKED

Fix estimate

15 min per page

Linked page blocked by robots.txt

Remove the robots.txt block if the page should appear in search; if the block is intentional, no fix is needed.

You link to this page as if it belonged in Google, and robots.txt forbids crawlers to read it. A robots.txt block does not deindex: Google may show such an address in the results, but without a title or description, because it was not allowed in. Worse, a blocked page cannot tell Google about its noindex or canonical, because the crawler will not read them. If the page should be visible, lift the block. If it should vanish, noindex first, and only then a block if you still want one. If the block is deliberate and the page unimportant, mark it so and the matter returns to housekeeping.

Id

BLOCKED_BY_ROBOTS_BUT_INDEXABLE_INTENT

Fix estimate

8 min per page

A firewall refuses a crawler robots.txt welcomes

A bot rule at the CDN answers 403 or 429 to a crawler the site allows: no correction to robots.txt will reach it.

robots.txt says "come in" and the firewall says "no". The second one wins, because it stands closer to the door. That crawler never sees robots.txt or the page; it gets a 403 or a challenge screen from the CDN. If it is a search engine or an answer engine crawler, your pages will not reach its index however good they are. Usually it is a default bot management rule that blocks whatever it does not know. One exception for the named crawler settles it. One refusal in one reading is only a hint; the server log will confirm whether the refusals repeat.

Id

CRAWLER_REFUSED_BY_EDGE

Fix estimate

30 min per page

A hostname points back at itself in DNS

The CNAME chain for this name returns to a name already in it, so it resolves for nobody; point the record at a host with an address.

Id

DNS_CNAME_LOOP

Fix estimate

20 min per page

Filters generate a very large crawlable URL space

Unless the filtered views need to rank on their own, disallow the filter parameters in robots.txt and leave one clean listing plus the product pages crawlable.

The filters in the shop create more addresses than you have products. Colour times size times price times sort order gives thousands of combinations, and each is a separate page for Google to read. The crawler has limited time for your site, and instead of reading new products it wanders through "red, size M, cheapest first, page 7". In Search Console this shows as a growing mountain of "discovered, not indexed". This is not a problem to fix page by page. Decide which combinations belong in Google because people search for them (usually a single filter, such as brand), and close the rest: a canonical to the category, a noindex, or a parameter block in robots.txt. A ready robots.txt block is on the action card.

Id

FACETED_URL_EXPLOSION

Fires at

≥ 25 variants per path

Fix estimate

45 min per page

Verified Googlebot refused by the server or the firewall

Your own server logs, the last seven days: a share of verified Googlebot requests got 5xx, 429 or 403. A 5xx is the server failing, a 429 is rate limiting, a 403 is a firewall treating Googlebot as an intruder; none of them has a page problem behind it. Fix the cause with the hosting or allow Googlebot in the firewall, then check the next reading.

Your logs show that the real Googlebot came and was turned away. A 403 or 429 for a verified Google crawler is no small thing: every refusal is a page Google did not see, and after repeats the address drops out of the index. Neither robots.txt nor content counts here; the protection at the edge of the site says "no" before the page gets a chance. Most often it is a bot management rule on the CDN, a request limit set too tight, or a blocked IP list that happens to contain a Google range. The fix is an exception for verified search engine crawlers; the protection itself can stay on.

Id

GOOGLEBOT_REQUESTS_REFUSED

Fix estimate

30 min per page

Server slowed down and Googlebot came less often

Your own server logs, the last seven days against the seven before: verified Googlebot waited longer for each answer and made fewer requests. Google backs off a struggling host, so new pages reach the index later. Ask the hosting for the slow hours, check caching and database load, and watch the next reading.

Googlebot comes less often because the server answers more slowly. That is courtesy on the crawler’s part: it sees the response time climbing and eases off so as not to bring the server down. The effect is the same as a penalty. New pages wait to be discovered, changed pages wait to be refreshed, and in the Crawl stats of Search Console two lines drift in opposite directions. Usually the culprit is hosting that stopped being enough, no caching, a plugin that recomputes something on every visit, or a database without indexes. Start with the response time of the homepage and the categories: those are what Googlebot reads most.

Id

GOOGLEBOT_SLOWED_BY_SERVER

Fix estimate

60 min per page

The page cannot be decompressed

The Content-Encoding header promises gzip or br and the body is something else; check the compression settings on the server or the proxy in front of it, browsers show an error page for this.

Id

HTML_DECOMPRESSION_FAILED

Fix estimate

20 min per page

Indexable page that verified Googlebot has not fetched

Your own server logs, the last 30 days: no request from a verified Googlebot address reached this page, so whatever the page says, it is not being read. Link to it from pages Googlebot does visit, make sure the sitemap lists it, and check the next reading.

The page is ready for Google, and in the logs Google never came for it. From the site's side everything is fine: a 200, no noindex, a canonical to itself. From the index's side the page does not exist, because nobody read it. Usually too few links lead to it, it sits too deep, or it is new and missing from the sitemap. If there are many such pages, Googlebot is probably not keeping up, because its visits go to filters, parameters and redirects. The road first: links from strong pages, the sitemap, less litter in the address space.

Id

INDEXABLE_NOT_FETCHED_BY_GOOGLEBOT

Fix estimate

10 min per page

Page did not answer

The page sent nothing within the time limit twice in one reading, once at the crawl's pace and once on its own: find what makes this URL slow, a crawler gives up long before a reader does.

The server did not answer at all: the connection dropped or the time ran out. That is worse than an error, because an error at least says what happened. To Google it is as if the page were not there, and if it repeats on the next visits, the address drops out of the index. Usually it is an overloaded server, a firewall quietly dropping connections from some addresses, or a page that takes so long to generate that it exceeds the time limit. Check whether the address answers from somewhere other than your office: that separates a server problem from a firewall problem.

Id

PAGE_NOT_ANSWERING

Fix estimate

20 min per page

Page answers 404

Restore the page or redirect it to the nearest live one, then fix whatever still points at it.

A 404 page is no tragedy, as long as nothing leads to it. A deleted product with no successor, an old post, a category that stopped existing: the server honestly says "not here", Google strikes the address after a while, and everything works as it should. The trouble starts when the address still sits in the menu, in an article, in the sitemap or in an ad. Then a reader clicks into a wall, and Googlebot spends a visit on something that does not exist. For every address the question is one: does this page have a successor? If so, redirect to it and fix the links. If not, keep the 404 and remove whatever still points at it.

Id

PAGE_NOT_FOUND

Fix estimate

10 min per page

Page answers a server error

Look at the server log for this URL: a 5xx that lasts is read as the page being gone.

A 5xx error is a fire in the building, even when only one room is burning. The server tells Google: "I am here, but I could not answer." A handful of such addresses is enough for Googlebot to decide the server cannot cope and to slow down across the whole site, every page included. New products and articles then wait longer for the index, and a page that keeps answering with an error eventually drops out of it. Usually one of three things hides here: a plugin or script that crashes on one page type, a traffic peak that runs out of memory, or a nightly export that locks the database. Read the server log by the hour as well as by the address: that shows whether the errors come in waves. Take it to your host today.

Id

PAGE_SERVER_ERROR

Fix estimate

20 min per page

Redirect with nowhere to go

A 3xx must carry a Location header; without one the browser shows a blank page and a crawler follows nothing.

A redirect with no target is a signpost with no board. The server says "go elsewhere" but not where, so the browser and the crawler stand still. The reader sees an error, Google sees a page leading nowhere. Usually it is a redirect rule with an empty variable, a plugin that lost the record of its target, or a template building the Location header from a missing field. Find the rule responsible for this address and give it a target or remove it.

Id

REDIRECT_WITHOUT_TARGET

Fix estimate

5 min per page

Relative links nest the same path deeper and deeper

Start internal hrefs with a slash (or the full URL): a bare href="stelaze" resolves against the current page’s path, so every page spawns a new nested address that does not exist.

The links on this page glue the path onto the previous one: /blog/ becomes /blog/blog/, then /blog/blog/blog/, without end. They are written relative, with no leading slash, so every click goes one floor down. The browser shows a page, because the server answers at each of these addresses, so nobody notices. The crawler notices: an infinite site with the same content at ever longer addresses. Usually it is a template link missing its leading slash. Fix the template, then check whether the server should not answer 404 to nested paths.

Id

RELATIVE_LINK_TRAP

Fix estimate

30 min per page

robots.txt answers with a server error

Make /robots.txt answer 200 (or 404 if there is none); while it answers 5xx or 429, Google treats the whole site as off limits and crawls nothing.

The robots.txt file answers with a server error, and to Google that is a special signal. When robots.txt does not answer, Google does not know what is allowed, so to be safe it stops reading the site altogether until the file returns. A 404 would mean "no rules, come in"; a 5xx means "I do not know, wait". If it lasts, pages begin dropping out of the index. Check whether the error affects only this address or the whole server; in the first case a plugin generating the file dynamically is usually at fault.

Id

ROBOTS_TXT_SERVER_ERROR

Fix estimate

20 min per page

The site refused our reader

Nearly every address answered 401 or 403 to us, or served a firewall's verification page instead of the site, so this reading is empty. Allowlist our crawler by its egress address, or the next reading comes back blank as well.

The site did not let our reader in, so this reading is nearly empty. Almost every address answered 401, 403 or a firewall challenge screen instead of a page. This finding is about the door, and says nothing about your pages yet: until it opens, there is no telling what is behind it. Usually it is Cloudflare, Imunify360 or similar protection blocking unknown crawlers wholesale. Add our egress address to the allow list in that protection and the next reading will go through. While there, check whether the same rule is also stopping the Google and Bing crawlers.

Id

SITE_REFUSES_CRAWLER

Fix estimate

30 min per page

Open to training, closed to being quoted

The training crawler is allowed and the search crawler of the same vendor is refused: the two are separate controls, and in this order the site is learned from without ever being cited.

You give your content away for training and forbid it to be quoted. Training crawlers may enter, while the crawlers that look for answers live and name their source are barred. That is the reverse of what most owners want: you give the content away for training and get nothing back. Usually it is a block list built piece by piece, with new crawler names added without distinguishing what they are for. Reverse the proportion: let the answer crawlers in and decide about the training ones knowingly.

Id

TRAINING_ALLOWED_SEARCH_BLOCKED

Fix estimate

15 min per page

Session id in the address

Keep the session in a cookie: in the address it turns every visit into a new page with the same content.

The addresses carry a session id, so every visitor, the crawler included, gets its own version of every page. To Google that is an endless number of copies: the same content at addresses differing only by a string nobody will remember. Each copy collects signals separately, and the results can end up showing an address with a year-old session. This is left over from older shops and applications that kept the session in the address instead of a cookie. Move the session to a cookie and block the session parameter.

Id

URL_HAS_SESSION_ID

Fix estimate

25 min per page

Date archive generates endless addresses

Stop the archive generating a page per day for ever: cap the range, or keep the generated addresses out of the crawl.

The date archive creates an endless number of addresses. The calendar has a "next month" link and the crawler follows it into 2031, 2032 and beyond, because every further page has the link again. Each of those pages is empty, and Googlebot reads them anyway, spending time meant for your content. Block the archive outside the range that holds posts, or remove the "next" link once there is nothing further.

Id

CALENDAR_TRAP

Fix estimate

30 min per page

A hostname points at a name that no longer exists

The CNAME target does not resolve: the old hosting was removed and the record was not; delete or repoint it, a dangling name can be claimed by someone else.

Id

DNS_CNAME_DANGLING

Fix estimate

20 min per page

The www (or bare) spelling of the domain does not exist

Add a record for the other spelling and redirect it to the site; visitors and links that use it currently reach nothing.

Id

DNS_SIBLING_HOST_MISSING

Fix estimate

20 min per page

Same filters linked in different parameter orders

Emit filter parameters in one fixed order so the same selection always has a single URL.

Id

INCONSISTENT_PARAM_ORDER

Fix estimate

25 min per page

Indexable page that verified Bingbot has not fetched

Your own server logs, the last 30 days: no request from a verified Bingbot address reached this page. Bing’s index is what ChatGPT search answers from, so until Bingbot reads it the page cannot be cited there. Link to it from pages Bingbot does visit, make sure the sitemap lists it, and submit it through IndexNow with the one-line request we draft. We draft the request; we do not send it.

Bingbot has never fetched this page, and that matters more today than it used to. Search in ChatGPT answers from Bing's index, so a page Bing has not read cannot be cited there, however well it is written. Bing has a smaller budget for your site than Google and gives up sooner on deep or weakly linked pages. Add the site to Bing Webmaster Tools if it is not there, submit the sitemap, and consider IndexNow: the one channel where you tell the search engine about new pages instead of waiting for it to find them.

Id

INDEXABLE_NOT_FETCHED_BY_BINGBOT

Fix estimate

10 min per page

Non-200 URL listed in sitemap

The sitemap reports addresses to Google that do not answer with a page: they redirect, return 404 or a server error. The map is the list of pages you want shown, so every such address is an invitation to an empty room. Google reads the map with less trust when part of it is stale, and wastes visits on addresses that do not exist. Usually it is a map generator taking everything from the database, withdrawn products and pre-redirect addresses included. The map should hold only addresses that answer 200 and belong in the index.

Id

NON_200_IN_SITEMAP

Fix estimate

4 min per page

Query string uses a non-standard separator

Separate query parameters with & — commas, semicolons and brackets are not parsed as separators and turn one filter set into an opaque single value.

Id

NONSTANDARD_PARAM_SEPARATOR

Fix estimate

25 min per page

Page answers 410

Confirm the page is meant to be gone for good, and remove the links and sitemap entries that still list it.

A 410 is a 404 said with conviction: "this page is gone and will not return." Google takes it more seriously than a plain 404 and strikes the address from the index sooner, which makes it a good way to be rid of pages you truly do not want. The question is whether you meant it. If a page answers 410 by mistake while it is linked or earning traffic, it has to come back quickly, because Google will not wait.

Id

PAGE_GONE

Fix estimate

10 min per page

The site refused the crawler

Raise the rate limit for search crawlers, or slow the reading down: pages refused with 429 are read as unavailable.

The server said "too much, slow down" with a 429. That is overload protection, and it cannot tell a search engine crawler from an attack. Our reader was refused, and the same happens to Googlebot, which then simply takes fewer pages at a time and comes back later. On a large site that means new content waits days for the index. Usually it is a request limit on the CDN or in a security plugin, tuned for people, and crawlers trip it. Keep the protection on, make an exception for search engine crawlers and raise the threshold for the rest.

Id

PAGE_RATE_LIMITED

Fix estimate

20 min per page

Page refuses the crawler

Check whether the page is meant to be public: a login wall on a page you want indexed keeps it out of search entirely.

The page answered "not letting you in" with a 401 or a 403. Sometimes that is fine: an admin panel, a customer area, a staging copy. The problem is when a page meant to sell gets the same answer. More and more often the server is fine and the refusal comes from overzealous protection: a firewall, bot management on the CDN or a security plugin that took the crawler for an intruder. Google reacts to a 403 the way it reacts to a page disappearing. Check whether the address answers normally in a browser, then whether it answers to crawlers. If the browser sees the page and the crawler gets a 403, the protection is at fault and the page itself is fine.

Id

PAGE_UNAUTHORIZED

Fix estimate

15 min per page

Pagination points to a broken or non-indexable page

Id

PAGINATION_BROKEN

Fix estimate

6 min per page

robots.txt hides the stylesheets or scripts

Let crawlers fetch the CSS and JavaScript: the page is rendered before it is judged, and a blocked stylesheet means it is judged unstyled.

The robots.txt file keeps crawlers out of the stylesheets and scripts. The pages are reachable, but Google sees them the way a browser without CSS and JavaScript would: text without layout, a menu that does nothing, sometimes without the content a script loads. Google also judges whether a page works on a phone, and without the styles that cannot be told. This is an old habit from the days of blocking wp-includes and similar directories. Today the crawler should see the page as a person does, so unblock the files the page needs to display.

Id

ROBOTS_TXT_BLOCKS_RESOURCES

Fix estimate

20 min per page

robots.txt is an HTML page

The address serves a web page (usually a catch-all route) instead of the text file; serve the file itself, a page holds no rules and announces no sitemap.

At the robots.txt address there is an HTML page instead of a file of rules, usually an error page or the homepage. Crawlers asking for the rules get page code and find nothing in it they can read. Google treats that as no file and goes everywhere, which may be fine, or not. Usually it is a rule redirecting every unknown address to the homepage that caught robots.txt too, or a CMS answering that address itself. The server should serve a plain text file at this address, or a clean 404.

Id

ROBOTS_TXT_IS_HTML

Fix estimate

15 min per page

Internal search results are open to crawlers

Disallow the search path in robots.txt: every query is a new address, and a results page is not something anybody wrote.

The result pages of your internal search are open to crawlers. Every query is a new address, and nobody wrote those addresses: they are automatic lists, often empty or built from junk terms typed by bots. Google reads them, sometimes indexes them, and your site starts appearing in results for phrases you would not want, while spending crawler visits on pages without value. Block the search path in robots.txt and add a noindex on the results page; two lines that tidy up thousands of addresses.

Id

SEARCH_RESULTS_CRAWLABLE

Fires at

≥ 3 links on one page

Fix estimate

20 min per page

Maintenance page never says when to come back

Send a Retry-After header with the 503 so a crawler knows when to return. Without it the crawler guesses, and it guesses long.

A maintenance page without a Retry-After header is a closed shop with no "back at 2 pm" sign. A 503 tells Google "temporarily down", which in itself is fine, because Google does not strike the page, it comes back later. Without being told when, the crawler guesses, and after a few failed tries it starts treating the page as gone. If the outage is planned, add the header with the return time. If 503s appear without a plan, the site is overloaded, and the cause is found the way 5xx errors are.

Id

SERVICE_UNAVAILABLE_NO_RETRY_AFTER

Fix estimate

5 min per page

A sitemap is not valid sitemap XML

Open the sitemap in a browser: it is not XML, or not a <urlset> or <sitemapindex>; Google skips the whole file, and every URL it should have announced.

Id

SITEMAP_INVALID_XML

Fix estimate

20 min per page

A sitemap has output before its XML declaration

Something prints before <?xml (a PHP notice, a blank line in a template, a byte order mark); strict readers stop there, so find what echoes first and silence it.

Id

SITEMAP_OUTPUT_BEFORE_XML

Fix estimate

15 min per page

A declared sitemap does not answer

A sitemap named in robots.txt or in a sitemap index answers with an error; fix the address or the file, Google stops at it.

robots.txt or Search Console names a sitemap that cannot be fetched: it answers 404, an error, or nothing. Google asks for the list of your pages and finds a closed door, so it relies on links alone and discovers new pages more slowly. Usually it is a map address from before a plugin change, a map switched off during a tidy-up, or a file generated on demand that stopped generating. Open the address in a browser: if it does not show XML, fix the path in robots.txt or restore the generator.

Id

SITEMAP_UNREACHABLE

Fix estimate

15 min per page

The response stops before the end of the page

The HTML ends before its closing tags, twice in one reading: look for a script that runs out of time or memory mid-page, the fragment is what gets indexed.

Id

TRUNCATED_RESPONSE

Fix estimate

20 min per page

Parameterized URL without canonical

An address with a parameter after the question mark and no canonical is a separate page to Google. Sort order, page number, colour, session id, campaign tag: every variant collects its own signals and any of them can enter the index instead of the clean address. On a shop there are thousands of such variants, and they eat the Googlebot visits that should go to new products. A canonical to the clean address settles most cases; what should never be in the index is better blocked in robots.txt as well.

Id

URL_WITH_PARAMETERS_NO_CANONICAL

Fix estimate

5 min per page

Meta refresh used as a redirect

Replace the meta refresh with a proper 301 server redirect.

The page redirects through a tag in its code, with no part played by the server: the tag tells the browser, after a moment, to go elsewhere. The reader sees the old page flash, sometimes a countdown, and the crawler gets a 200 and a page saying "not here". Google usually understands the trick, but more slowly and with no guarantee the signals carry over. It is left over from the days without access to the server configuration. Replace it with an ordinary 301 redirect on the server.

Id

META_REFRESH_REDIRECT

Fix estimate

5 min per page

URL is long or uses unfriendly characters

Prefer short lowercase URLs with hyphens; avoid uppercase, underscores and spaces in new URLs.

Id

NON_SEO_FRIENDLY_URL

Fires at

> 120 characters

Fix estimate

10 min per page

Page answers an unexpected status

Check what this URL is meant to answer, because search engines treat anything outside 200 and 301 as a page they cannot keep.

Id

PAGE_UNEXPECTED_STATUS

Fix estimate

10 min per page

Pagination only links forwards

Add the matching rel=prev on the page being pointed at, so the sequence can be walked in both directions.

Id

PAGINATION_SEQUENCE_ONE_WAY

Fix estimate

15 min per page

robots.txt has lines nothing reads

Fix or remove the unrecognised lines: a misspelled directive is skipped in silence, so the rule you wrote never applied.

Id

ROBOTS_TXT_INVALID_LINE

Fix estimate

15 min per page

No robots.txt

Serve a robots.txt, even a permissive one: without it every crawler guesses, and the file is where the sitemap is announced.

Id

ROBOTS_TXT_MISSING

Fix estimate

15 min per page

robots.txt does not name the sitemap

Add a Sitemap line to robots.txt: it is the one place every crawler looks for it without being told.

Id

ROBOTS_TXT_NO_SITEMAP

Fix estimate

10 min per page

robots.txt is reached through redirects

Serve /robots.txt directly on this host; a chain of redirects, or a hop to another host, is a file some crawlers give up on.

Id

ROBOTS_TXT_REDIRECT_CHAIN

Fix estimate

15 min per page

robots.txt is larger than crawlers read

Google reads the first 500 KiB of robots.txt and ignores the rest; shorten the file so every rule is inside the part that is read.

Id

ROBOTS_TXT_TOO_LARGE

Fix estimate

15 min per page

robots.txt is behind 401 or 403

Let /robots.txt be read without credentials; crawlers treat a refused file as absent, and every rule in it goes unread.

Id

ROBOTS_TXT_UNAUTHORIZED

Fix estimate

15 min per page

Two sitemap generators answer at once

WordPress core still serves wp-sitemap.xml next to the SEO plugin's sitemap; switch the core one off (most plugins have the setting) so search engines read one map.

Id

SITEMAP_GENERATORS_COMPETING

Fix estimate

10 min per page

A sitemap index lists itself

A sitemap index points back at itself or at an index above it; remove the loop: an index may list only URL sets.

Id

SITEMAP_INDEX_CYCLE

Fix estimate

10 min per page

Sitemap dates are in the future

Some lastmod values are ahead of today; once one date is impossible Google stops trusting all of them, so fix the generator or drop lastmod.

Id

SITEMAP_LASTMOD_IN_FUTURE

Fix estimate

10 min per page

Page missing from sitemap

This page is not in the sitemap, though it is reachable and fit for the index. A map is not compulsory, Google finds pages through links too, but on large and deep sites it is the map that tells the crawler about new addresses fastest. A page outside the map waits until the crawler stumbles on it. Usually it is a content type the map generator ignores: custom post types, pages from a separate system, landing pages. Add them to the map, and if the generator cannot see them, check its settings.

Id

SITEMAP_MISSING_URL

Fix estimate

4 min per page

A sitemap is past the protocol limits

Split the sitemap: one file may hold 50,000 URLs and 50 MB uncompressed, and what is past the limit is ignored.

Id

SITEMAP_TOO_LARGE

Fix estimate

20 min per page

Double slash in the path

Collapse the repeated slash and redirect the old form: both addresses answer, so both get crawled.

Id

URL_DOUBLE_SLASH

Fix estimate

10 min per page

Same parameter twice in one address

Emit each parameter once: repeated keys make addresses that differ only by order of the same values.

Id

URL_REPEATED_PARAMETER

Fix estimate

10 min per page

Address is very long

Shorten the path to the words that identify the page: an address nobody can read is an address nobody links to.

Id

URL_TOO_LONG

Fix estimate

10 min per page

Address contains unsafe characters

Replace spaces and punctuation with hyphens: the address works in a browser and breaks everywhere it is pasted as text.

Id

URL_UNSAFE_CHARACTERS

Fix estimate

10 min per page

Training crawlers are blocked

Whether a model may learn from the site is your call: worth a look because this state is usually inherited from a template or a CDN preset, and rarely chosen on purpose.

The site blocks the crawlers that collect content for training models. That is a decision, and this reading does not judge it; it notes it, because it is worth knowing what follows from it. Blocking training crawlers does not affect Google results or live citation, as long as the answer crawlers may enter. If this was the intent, do nothing. If the block list grew by accident, check whether it also cut out crawlers you want.

Id

AI_CRAWLER_BLOCKED

Fix estimate

15 min per page