Log Intelligence: what crawlers actually do on your server

A reading can say a page is indexable, linked and in the sitemap. It cannot say whether Google ever came for it. Only the server knows that, and it writes it down in the access log with every request.

Log Intelligence reads those logs against your latest reading and answers six questions a site owner actually asks: how much of each template is being crawled, which pages have never been visited, which URLs bots fetch that you did not know existed, where the crawl budget goes to waste, who is really crawling, and how long a new page waits for its first visit.

There is no dashboard builder and no query language. The reports are the product, and each one is a screen you can read in a minute and export as CSV.

Search Console hides its most useful report under Settings: how many requests Googlebot makes, what it gets back and how long the server takes. Google publishes no API for it. Your access log records the same three facts for every verified Googlebot request, so the Bot activity screen shows them: the last seven days the logs cover against the seven before.

Two verdicts come out of it, and both reach the reading. If the server got slower and Googlebot came less often, Google is backing off a struggling host and new pages will wait longer for the index. If verified Googlebot is being answered 5xx, 429 or 403, something refuses it, and a 403 almost always points to a firewall rule, with the server itself in order.

Each screen is one question, answered per URL template, so a site of forty thousand URLs reads as a dozen rows.

• Coverage: how much of each template verified Googlebot reached, with the change week over week and a trend.
• Uncrawled: indexable URLs with no verified Googlebot hit in the period, filtered by template and click depth.
• Orphans: URLs bots fetched that your reading never found, with the top crawler and when it was last seen.
• Waste: bot hits that bought nothing, on non-indexable URLs, 4xx and 5xx, with a breakdown by query parameter.
• Bot activity: daily verified hits by crawler, with spoofed claims drawn separately.
• Time to first crawl: days from a URL first appearing in a reading to its first verified Googlebot hit, as a median and a p90 per template.

Anyone can put "Googlebot" in a request header. Every hit is therefore checked against the IP ranges Google, Bing, OpenAI and Perplexity publish. A request that claims a crawler from an address outside its published ranges is counted as spoofed: stored, shown separately, and kept out of the numbers you make decisions on.

The catalogue also names the AI fetchers that publish no ranges, ClaudeBot among them, so their traffic is visible even where it cannot be verified. Ingest refuses to run at all if the published ranges are more than two weeks stale, because a report built on old ranges would be confidently wrong.

The module is part of the Agency plan. Once the plan is in place, the Server logs section appears in the site navigation, and setup is one screen.

1. Open Server logs, then Log settings, and declare the log format once: nginx, Apache, IIS / W3C Extended or Cloudflare Logpush. Pick a preset or paste your log_format or LogFormat directive verbatim.
2. Press "Check against my file". The first hundred lines of your own file are parsed and shown as a table, so you confirm the format on real rows before anything is imported.
3. Upload the files: .log, .txt, .json or .ndjson, .gz and .zip, up to 5 GB each. Files uploaded before the format is confirmed wait, and are released when it is.
4. Choose the crawl data to join against: your latest SilentStork reading, or a CSV with the columns url, status, click_depth, inlinks, indexable, canonical and first_seen.
5. Optionally import templates from your routing file: a Laravel routes file, a Next.js page tree or a sitemap. Otherwise templates are generated from the URLs and you refine them with Rename, Merge and Split. There is no regular expression to write.

Human traffic is discarded while the file is being parsed. Only verified crawler activity is written, so there is no visitor IP address in our database to export, anonymise or delete, and no data-processing paperwork to sign before your first upload.

Log-management platforms cannot make that promise, because storing everything is their product. Here the logs are read once, for one purpose, and what remains is a table of what crawlers did.

Three alert rules, each with one threshold you set, send an email when something moves: coverage of any template drops week over week by more than a chosen number of points, verified Googlebot starts seeing server errors above a chosen share, or verified Googlebot volume falls off against its trailing seven-day median. A weekly digest carries the coverage table with its deltas.

Every open design decision in this module resolves toward the simplest thing to build and operate, even where that pushes a step onto you. So it is worth knowing the edges up front.

• No format auto-detection: you declare the format once, against your own file.
• No log pull from S3, GCS or FTP, and no real-time ingest: you upload files.
• No Slack or webhook alerts, no rule builder, no custom dashboards.
• An import with more than two percent unparseable lines is rolled back whole, so a partial file cannot pass for the full picture. A byte-identical file uploaded twice is recognised and skipped.

Indexable and crawled are two different facts. With logs uploaded, each reading compares its indexable pages with a month of verified Googlebot and Bingbot traffic and raises one card per crawler for the pages neither has fetched. Because the card exists only when you upload logs, it is one of the very few findings outside the core that may enter the weekly five, and it outranks any cosmetic issue on the same page.

The Bing card carries a one-line IndexNow request drafted for the affected addresses, with a placeholder for your key. It is a draft you run; nothing is sent on your behalf. Importing the site into Bing Webmaster Tools is a one-time setup step and then stays done. The card stays quiet without a week of crawler traffic in the logs, or when no crawler appears at all, because logs from the wrong host would mark every page unfetched.