Forum category

Crawling and Indexing

Crawl behavior, log analysis, sitemaps, robots controls, coverage, and difficult indexing cases.

Discovery, crawl access, rendering, canonical choice, and quality all sit between a published URL and an indexed one. This category exists for the cases where that sequence breaks and the report gives you a label instead of a reason.

Bring the sequence, not the verdict. "Crawled, currently not indexed" describes a symptom with at least four common causes. Separating them takes log data, a live fetch, and a look at what else on the site says the same thing. Post what you have checked and the next step is usually obvious.

What belongs here

  • Coverage states, and what each one really means for a template
  • Log file analysis, crawl frequency, and wasted requests
  • Sitemap strategy, lastmod accuracy, and submission behavior
  • Robots directives, noindex timing, and the order controls take effect in
  • Soft 404 classification and thin page thresholds
  • Indexing delays after a launch, a migration, or a large publish run

Bring this with your question

  • The coverage state reported, and how many URLs share it
  • A live fetch of one affected URL, including headers
  • Whether the URL appears in the sitemap and in the internal link graph
  • How long the state has held, and what changed before it started
Swiss Knife SEO crawl report listing three ranked next moves, each with the number of pages affected.
A finished crawl ends with ranked next moves and the page count behind each one. That page count is the detail most indexing questions are missing. Swipe the panel to read it at full width.

Approved discussions

1 discussion