# Robots.txt Errors and How to Fix Them

> The most common robots.txt errors, what each one does to crawling, and the exact fix. Test your file free with the Axiolo robots.txt checker.

A single line in robots.txt can hide your whole site from Google. Most errors are small, easy to miss and easy to fix once you know what to look for. This guide covers the errors we see most often on B2B sites and how to fix each one.

Before you start, paste your domain into the [robots.txt checker](/tools/robots-txt-tester) so you can see your live file and test URLs against it.

## 1. Disallow: / left over from staging

What it looks like:

```
User-agent: *
Disallow: /
```

What it does: blocks every crawler from every page. This often ships when a staging site goes live.
Fix: remove the line or change it to `Disallow:` with nothing after it. Then request indexing for key pages in Search Console.

## 2. Robots.txt returns a 404 or 5xx

A missing file (404) is fine. Google treats it as "crawl everything". A server error (5xx) is not. Google may pause crawling until the file loads again.
Fix: make sure /robots.txt returns 200 or 404, never 5xx. Check the status with the [crawl checker](/tools/crawl-checker). Read more on [HTTP status codes for SEO](/blog/understanding-http-status-codes-seo-optimization).

## 3. Blocking CSS and JavaScript

Rules like `Disallow: /assets/` stop Google from rendering your pages properly.
Fix: allow the folders that hold CSS, JS and images.

## 4. Using robots.txt to remove pages from Google

Disallow stops crawling, not indexing. A blocked URL can still show in results with no description.
Fix: allow crawling and add a noindex meta tag, or remove the page and return 404 or 410.

## 5. Wrong case or wrong path

Paths are case sensitive. `Disallow: /Admin` does not block `/admin`.
Fix: match the exact path and test it in the checker.

## 6. Wildcards that block too much

`Disallow: /*?` blocks every URL with a parameter, including paginated or filtered pages you may want indexed.
Fix: target the exact parameter, for example `Disallow: /*?sessionid=` and `Disallow: /*&sessionid=` so the parameter is blocked in any position.

## 7. Missing or wrong sitemap line

Fix: add `Sitemap: https://www.yourdomain.com/sitemap-index.xml` using your final www or non-www host.

## 8. Rules for the wrong host

Robots.txt applies per host. Rules on yourdomain.com do not apply to www.yourdomain.com or blog.yourdomain.com.
Fix: serve the right file on each host, and redirect non-canonical hosts.

## How to check your robots.txt in 2 minutes

1. Open the [robots.txt checker](/tools/robots-txt-tester).
2. Enter your domain and 3 key URLs (home, a service page, a blog post).
3. Confirm each is allowed for Googlebot.
4. Run the [crawl checker](/tools/crawl-checker) on the same URLs to confirm status, noindex and canonical.

## Frequently asked questions

- **Does a robots.txt error hurt rankings?** Only if it blocks pages or resources Google needs. A blocked page cannot be crawled, so Google cannot read or refresh its content. It can still appear in results with no snippet. To keep a page out of Google, allow crawling and use a noindex tag.
- **How fast does Google pick up a fixed robots.txt?** Google usually caches robots.txt for up to 24 hours.
- **Should I block AI crawlers?** It depends on whether you want to appear in AI answers. For most B2B sites we recommend allowing them.

---

Source: https://www.axiolo.com/blog/robots-txt-errors-how-to-fix