Log Intelligence: what crawlers actually do on your server

A reading can say a page is indexable, linked and in the sitemap. It cannot say whether Google ever came for it. Only the server knows that, and it writes it down in the access log with every request.

Log Intelligence reads those logs against your latest reading and answers six questions a site owner actually asks: how much of each template is being crawled, which pages have never been visited, which URLs bots fetch that you did not know existed, where the crawl budget goes to waste, who is really crawling, and how long a new page waits for its first visit.

There is no dashboard builder and no query language. The reports are the product, and each one is a screen you can read in a minute and export as CSV.

Each screen is one question, answered per URL template rather than per page, so a site of forty thousand URLs reads as a dozen rows.

• Coverage: how much of each template verified Googlebot reached, with the change week over week and a trend.
• Uncrawled: indexable URLs with no verified Googlebot hit in the period, filtered by template and click depth.
• Orphans: URLs bots fetched that your reading never found, with the top crawler and when it was last seen.
• Waste: bot hits that bought nothing, on non-indexable URLs, 4xx and 5xx, with a breakdown by query parameter.
• Bot activity: daily verified hits by crawler, with spoofed claims drawn separately.
• Time to first crawl: days from a URL first appearing in a reading to its first verified Googlebot hit, as a median and a p90 per template.

Anyone can put "Googlebot" in a request header. Every hit is therefore checked against the IP ranges Google, Bing, OpenAI and Perplexity publish. A request that claims a crawler from an address outside its published ranges is counted as spoofed: stored, shown separately, and never mixed into the numbers you make decisions on.

The catalogue also names the AI fetchers that publish no ranges, ClaudeBot among them, so their traffic is visible even where it cannot be verified. Ingest refuses to run at all if the published ranges are more than two weeks stale, because a report built on old ranges would be confidently wrong.

The module is part of the Agency plan. Once the plan is in place, the Server logs section appears in the site navigation, and setup is one screen.

1. Open Server logs, then Log settings, and declare the log format once: nginx, Apache, IIS / W3C Extended or Cloudflare Logpush. Pick a preset or paste your log_format or LogFormat directive verbatim.
2. Press "Check against my file". The first hundred lines of your own file are parsed and shown as a table, so you confirm the format on real rows before anything is imported.
3. Upload the files: .log, .txt, .json or .ndjson, .gz and .zip, up to 5 GB each. Files uploaded before the format is confirmed wait, and are released when it is.
4. Choose the crawl data to join against: your latest SilentStork reading, or a CSV with the columns url, status, click_depth, inlinks, indexable, canonical and first_seen.
5. Optionally import templates from your routing file: a Laravel routes file, a Next.js page tree or a sitemap. Otherwise templates are generated from the URLs and you refine them with Rename, Merge and Split. You never write a regular expression.

Human traffic is discarded while the file is being parsed. Only verified crawler activity is written, so there is no visitor IP address in our database to export, anonymise or delete, and no data-processing paperwork to sign before your first upload.

Log-management platforms cannot make that promise, because storing everything is their product. Here the logs are read once, for one purpose, and what remains is a table of what crawlers did.

Three alert rules, each with one threshold you set, send an email when something moves: coverage of any template drops week over week by more than a chosen number of points, verified Googlebot starts seeing server errors above a chosen share, or verified Googlebot volume falls off against its trailing seven-day median. A weekly digest carries the coverage table with its deltas.

Every open design decision in this module resolves toward the simplest thing to build and operate, even where that pushes a step onto you. So it is worth knowing the edges up front.

• No format auto-detection: you declare the format once, against your own file.
• No log pull from S3, GCS or FTP, and no real-time ingest: you upload files.
• No Slack or webhook alerts, no rule builder, no custom dashboards.
• An import with more than two percent unparseable lines is rolled back rather than shown as a partial truth. A byte-identical file uploaded twice is recognised and skipped.