If Google cannot crawl a page, it cannot read or refresh its content, and a noindex tag on it will never be seen. B2B sites lose crawl coverage in quiet ways: a redesign, a new subdomain, a gated resources hub. Use this checklist each quarter and after every launch.

Test any page as you go with the free crawlability checker.

Access

  1. Robots.txt allows key sections. Test home, services, blog and resources in the robots.txt checker.
  2. No leftover noindex. Check meta robots and X-Robots-Tag headers on templates.
  3. Key pages return 200. No soft 404s, no 5xx. See HTTP status codes for SEO.
  4. Gated content has a crawlable landing page. Forms can gate the PDF, not the page that describes it.

One version of every URL

  1. One host. Pick www or non-www and 301 the other.
  2. One trailing slash rule. Pick one and redirect the other.
  3. No redirect chains. Every internal link points straight to the final URL.
  4. Self-referencing canonicals on the final host.

Discovery

  1. XML sitemap lists only 200, indexable, canonical URLs, and is in robots.txt.
  2. Every important page is 3 clicks or fewer from home.
  3. No orphan pages. Each service and industry page has internal links from related posts.
  4. Pagination and filters are crawlable links, not only JavaScript buttons.

Rendering and speed

  1. Main content is in the HTML or renders fast. Test with Search Console URL Inspection.
  2. CSS and JS are not blocked.
  3. Server responds fast. Slow servers lower how much Google crawls.
  • Structured data for Organization and key pages. Build it with the schema generator.
  • Allow the AI crawlers you want to be cited by.

A worked example

Here is a hypothetical example. Picture a SaaS site with 140 URLs in the sitemap but only 60 indexed. The crawl showed apex and www both live, trailing slash variants, and a noindex left on the blog template. Fixing hosts, redirects and the template tag made the full site eligible for indexing again.