Log analysis produces a lot of interesting numbers and, in my experience, a small number of decisions. I would like to compare notes on which decisions.
Three have been worth the work for me. Finding a template soaking up crawl requests that has no business being crawled. Finding that an important set is fetched far less often than its update frequency needs. And proving that a directive took effect, rather than assuming it did from the configuration screen.
The things I have stopped extracting are crawl budget totals as a headline number, which nobody ever acted on, and bot traffic breakdowns beyond confirming which agents are real.
What makes this hard to justify is the setup. Getting the logs at all is a conversation with hosting, and with a CDN in front of the origin you often get a partial picture that hides exactly the requests you wanted to see.
For those of you doing this routinely, what is the finding that keeps earning its place?
Swiss Knife SEO ModeratorsModeration team