# Crawlability Checklist for B2B Websites

> A 15-point crawlability checklist for B2B websites. Check robots.txt, status codes, canonicals, sitemaps, internal links and rendering, with free tools.

If Google cannot crawl a page, it cannot read or refresh its content, and a noindex tag on it will never be seen. B2B sites lose crawl coverage in quiet ways: a redesign, a new subdomain, a gated resources hub. Use this checklist each quarter and after every launch.

Test any page as you go with the free [crawlability checker](/tools/crawl-checker).

## Access

1. **Robots.txt allows key sections.** Test home, services, blog and resources in the [robots.txt checker](/tools/robots-txt-tester).
2. **No leftover noindex.** Check meta robots and X-Robots-Tag headers on templates.
3. **Key pages return 200.** No soft 404s, no 5xx. See [HTTP status codes for SEO](/blog/understanding-http-status-codes-seo-optimization).
4. **Gated content has a crawlable landing page.** Forms can gate the PDF, not the page that describes it.

## One version of every URL

5. **One host.** Pick www or non-www and 301 the other.
6. **One trailing slash rule.** Pick one and redirect the other.
7. **No redirect chains.** Every internal link points straight to the final URL.
8. **Self-referencing canonicals** on the final host.

## Discovery

9. **XML sitemap** lists only 200, indexable, canonical URLs, and is in robots.txt.
10. **Every important page is 3 clicks or fewer from home.**
11. **No orphan pages.** Each service and industry page has internal links from related posts.
12. **Pagination and filters** are crawlable links, not only JavaScript buttons.

## Rendering and speed

13. **Main content is in the HTML** or renders fast. Test with Search Console URL Inspection.
14. **CSS and JS are not blocked.**
15. **Server responds fast.** Slow servers lower how much Google crawls.

## Bonus for AI search

- Structured data for Organization and key pages. Build it with the [schema generator](/tools/schema-generator).
- Allow the AI crawlers you want to be cited by.

## A worked example

Here is a hypothetical example. Picture a SaaS site with 140 URLs in the sitemap but only 60 indexed. The crawl showed apex and www both live, trailing slash variants, and a noindex left on the blog template. Fixing hosts, redirects and the template tag made the full site eligible for indexing again.

<!-- TODO(Gabriel): replace with a real, approved client result before marking this PR ready. -->

## Frequently asked questions

- **How often should I check crawlability?** Quarterly, and after every launch or migration.
- **What is the difference between crawlability and indexability?** Crawlability is whether Google can reach the page. Indexability is whether it is allowed and chooses to store it.
- **Do small B2B sites need to worry about crawl budget?** Rarely. Fix access and duplicates first.

---

Source: https://www.axiolo.com/blog/crawlability-checklist-b2b-websites