A single line in robots.txt can hide your whole site from Google. Most errors are small, easy to miss and easy to fix once you know what to look for. This guide covers the errors we see most often on B2B sites and how to fix each one.

Before you start, paste your domain into the robots.txt checker so you can see your live file and test URLs against it.

1. Disallow: / left over from staging

What it looks like:

User-agent: *
Disallow: /

What it does: blocks every crawler from every page. This often ships when a staging site goes live. Fix: remove the line or change it to Disallow: with nothing after it. Then request indexing for key pages in Search Console.

2. Robots.txt returns a 404 or 5xx

A missing file (404) is fine. Google treats it as “crawl everything”. A server error (5xx) is not. Google may pause crawling until the file loads again. Fix: make sure /robots.txt returns 200 or 404, never 5xx. Check the status with the crawl checker. Read more on HTTP status codes for SEO.

3. Blocking CSS and JavaScript

Rules like Disallow: /assets/ stop Google from rendering your pages properly. Fix: allow the folders that hold CSS, JS and images.

4. Using robots.txt to remove pages from Google

Disallow stops crawling, not indexing. A blocked URL can still show in results with no description. Fix: allow crawling and add a noindex meta tag, or remove the page and return 404 or 410.

5. Wrong case or wrong path

Paths are case sensitive. Disallow: /Admin does not block /admin. Fix: match the exact path and test it in the checker.

6. Wildcards that block too much

Disallow: /*? blocks every URL with a parameter, including paginated or filtered pages you may want indexed. Fix: target the exact parameter, for example Disallow: /*?sessionid= and Disallow: /*&sessionid= so the parameter is blocked in any position.

7. Missing or wrong sitemap line

Fix: add Sitemap: https://www.yourdomain.com/sitemap-index.xml using your final www or non-www host.

8. Rules for the wrong host

Robots.txt applies per host. Rules on yourdomain.com do not apply to www.yourdomain.com or blog.yourdomain.com. Fix: serve the right file on each host, and redirect non-canonical hosts.

How to check your robots.txt in 2 minutes

  1. Open the robots.txt checker.
  2. Enter your domain and 3 key URLs (home, a service page, a blog post).
  3. Confirm each is allowed for Googlebot.
  4. Run the crawl checker on the same URLs to confirm status, noindex and canonical.