Noindex vs Robots.txt Disallow: Which One to Use
Basics3 minInnovialabs editorial team
In short
Disallow in robots.txt stops crawlers from fetching a page, while noindex stops it from appearing in search; to remove a page from results, use noindex and keep the URL crawlable.
The short answer
- Disallow in robots.txt blocks crawling: the crawler does not fetch the page.
- noindex (a meta tag or HTTP header) blocks indexing: the crawler fetches the page but does not show it in search.
If you want a page out of search results, use noindex. If you want crawlers to stop spending resources on useless URLs, use Disallow.
Why Disallow does not remove a page from search
Disallow only says "do not enter". If other pages, on your site or elsewhere, link to a blocked URL, Google may still index that URL without knowing its content. It can show up in results as a bare address with no proper description.
That is why robots.txt is a crawl management tool, not a way to hide pages.
How noindex works
Noindex is set in HTML:
<meta name="robots" content="noindex">
Or as an HTTP header, which also works for PDFs, images and other non-HTML files:
X-Robots-Tag: noindex
For the directive to work, the crawler must fetch the page and see it. Google and Yandex both respect the noindex meta tag. A noindex rule inside robots.txt is not supported by Google.
Why Disallow + noindex backfires
This is the most common mistake: a page is blocked in robots.txt and given noindex "just to be safe". The result:
- Robots.txt forbids fetching the page.
- The crawler never fetches it, so it never sees noindex.
- If the URL has links pointing to it, it may stay in the index.
The correct order for removing an already indexed page:
- Add noindex and do not block the URL in robots.txt.
- Wait until the page drops out of the index (check in Search Console or Yandex Webmaster).
- Only then, if needed, block it with Disallow to save crawl resources.
What to use for typical pages
| Page type | What to use |
|---|---|
| Cart, checkout, user account | noindex or Disallow; these rarely get indexed on their own |
| Internal site search | noindex; Disallow if there are huge numbers of such URLs |
| Filters and sorting with parameters | canonical to the main page, Disallow for endless combinations |
| "Thank you" page after a form | noindex |
| PDFs and files you do not want in search | X-Robots-Tag: noindex |
| Duplicate pages | canonical, not Disallow |
| Staging site | password protection; robots.txt and noindex do not prevent access |
| Technical URLs, service scripts | Disallow |
Common mistakes
- Blocking CSS and JS in robots.txt. The crawler cannot render the page and may evaluate it incorrectly.
- Shipping
Disallow: /from staging to production. The whole site stops being crawled. Check robots.txt after every release. - Leaving noindex in a template after development. Pages quietly disappear from search.
- Using robots.txt for secret sections. The file is public, so anyone can read which paths you are hiding.