Three questions, three tools
- May you crawl this? Answered by robots.txt, a text file at the root of the site that tells crawlers which URLs they may fetch.
- May you show this in results? Answered by noindex, a rule in the page's HTML or HTTP headers that a crawler reads after fetching the page.
- Which of these copies should you show? Answered by canonicalization: a redirect, a rel="canonical" link, or a sitemap entry that names the preferred URL.
Crawling comes before indexing, and that order explains most of the surprises below.
Robots.txt: crawling, not hiding
Google's documentation says a robots.txt file is used primarily to manage crawler traffic to your site. That is its job: keep crawlers away from parts of the site that waste their requests, such as internal search results or endless filter combinations. What it does not do is remove pages from search. In Google's words, it is not a mechanism for keeping a web page out of Google, and Google might still find and index a disallowed URL if it is linked from other places on the web; the result then shows the URL without a description, because Google never read the page. To keep a page out of Google, the documentation says to block indexing with noindex or to password-protect the page.
Disallow: /search/
Disallow: /cart/
Sitemap: https://example.com/sitemap.xml
Two legitimate uses beyond traffic. Robots.txt can prevent image, video and audio files from appearing in Google Search results, since media files have no HTML to carry a noindex tag, though it won't prevent other pages or users from linking to the files. And it can block unimportant image, script or style files, but Google adds a condition: only if pages loaded without those resources won't be significantly affected, because Googlebot renders pages and needs the CSS and scripts that shape them. The robots.txt generator writes the file with the right syntax and a sitemap line; the XML sitemap generator builds the sitemap it points to.
Noindex: out of the results, after a crawl
Noindex is a rule set with either a meta tag or an HTTP response header. When Googlebot crawls the page and sees it, Google drops the page entirely from Search results, regardless of whether other sites link to it. That last clause is what robots.txt cannot offer.
The one rule that trips people up: for noindex to be effective, the page must not be blocked by robots.txt. If it is blocked, the crawler never sees the noindex rule, and the page can still appear in results if other pages link to it. So a page you want gone must stay crawlable. Timing is the other catch. Google has to recrawl the page to notice the tag, and because crawl frequency varies it may take months for Googlebot to revisit a page; the URL Inspection tool in Search Console lets you request a recrawl. The meta tag generator has a noindex checkbox that adds the tag to a page's head, and the HTTP header checker shows whether a URL is sending an X-Robots-Tag.
Canonical: one URL to show among copies
Most sites serve the same content at more than one address: with and without tracking parameters, http and https, with and without www, a print version, a mobile version. Google picks one of them as the canonical URL and consolidates the rest into it. Its documentation gives four reasons to tell it which one: to specify which URL you want people to see in search results, to consolidate signals for similar or duplicate pages, to simplify tracking metrics for a piece of content, and to avoid spending crawling time on duplicate pages.
Google lists the methods in order of how much they weigh:
- Redirects: a strong signal that the target of the redirect should become canonical. The redirect generator writes the server rules for a 301 or 308.
- rel="canonical" link annotations: a strong signal that the specified URL should become canonical.
- Sitemap inclusion: a weak signal that helps the URLs in a sitemap become canonical.
None of the methods are required, and all of them are signals rather than commands: Google weighs them against what it finds. Its "don't" list is short and worth keeping. Don't use the robots.txt file for canonicalization purposes. Don't use the URL removal tool for canonicalization. Don't specify different URLs as canonical for the same page using different techniques, for example a canonical link saying one thing and a sitemap saying another. Don't specify a URL fragment as canonical. And don't use noindex to prevent selection of a canonical page within a single site, because it will completely block the page from Search; a duplicate with noindex is not consolidated, it is deleted.
Which one, when
| What you want | Use | Not |
|---|---|---|
| Keep a page out of Google Search results | noindex (meta tag or X-Robots-Tag), and let the page be crawled | robots.txt: a blocked page can still be indexed if other sites link to it |
| Stop crawlers hammering a section that has no search value (internal search results, endless filters, cart) | robots.txt Disallow | noindex on thousands of URLs: each one still has to be crawled to be seen |
| Keep images, video or audio files out of Google results | robots.txt Disallow for the file paths | noindex in HTML, which media files do not have; X-Robots-Tag also works |
| The same content lives at several URLs (tracking parameters, http and https, www and bare, print versions) | rel="canonical" pointing to the one URL you want shown, or a redirect if the duplicates should go away | noindex on the duplicates, which blocks them entirely and passes nothing on; robots.txt, which Google says not to use for canonicalization |
| A page has moved permanently | a 301 or 308 redirect, the strongest canonical signal | a canonical link from the old URL while it still serves the old page |
| Private or sensitive content | password protection; Google lists it next to noindex as the way to keep a page out | robots.txt, which is itself public and lists the paths you want hidden |
The mistakes that follow from mixing them up
- Blocking a page in robots.txt to remove it. It stays in the index as a bare URL if anything links to it. Use noindex and keep it crawlable.
- Adding noindex and then blocking the page. The block hides the noindex. Order matters: noindex first, wait for the recrawl, block afterwards if you still want to.
- Noindex on duplicates. The copies vanish rather than consolidating. Point them at the canonical instead.
- Conflicting canonical signals. A canonical link to one URL and a sitemap listing another leave Google to choose. Make the sitemap list only canonical URLs; the sitemap generator's advice to list each page in the form you want shown is exactly this.
- Blocking CSS and JavaScript. Google renders pages; block the resources and it sees a broken page. Google allows blocking unimportant resource files only when the page is not significantly affected without them.
- Expecting instant results. Every one of these takes effect at the next crawl. Request one in Search Console rather than adding more rules.
Frequently asked questions
Does robots.txt stop a page from being indexed?
No. Google is explicit that robots.txt is not a mechanism for keeping a web page out of Google and that it might still find and index a disallowed URL if it is linked from other places on the web. Robots.txt controls crawling, mainly to manage crawler traffic. To keep a page out of search results, use noindex or password protection.
Can I use robots.txt and noindex together on the same page?
Not if you want the noindex to work. Google says that for the noindex rule to be effective the page must not be blocked by a robots.txt file, because a blocked crawler never sees the rule, and the page can still appear in results if other pages link to it. Let the page be crawled, serve the noindex, and only block it in robots.txt once it has dropped out, if at all.
What is the difference between noindex and canonical?
Noindex removes a page from Google Search entirely. A canonical says that this page is one of several copies and names the one that should be shown, so the copies consolidate into it instead of disappearing. Google recommends against using noindex to prevent selection of a canonical within a site, because it will completely block the page from Search.
Is rel="canonical" a directive?
Google describes redirects and rel="canonical" annotations as strong signals and sitemap inclusion as a weak signal that help a URL become canonical, and notes that none of them are required. Google can choose a different canonical if its signals disagree with yours, for example when the canonical you specify is itself a duplicate of something else or points at a URL fragment.
How long does noindex take to work?
Google has to recrawl the page to see the meta tag or header, and crawl frequency varies, so it can take months for Googlebot to revisit a page. You can request a recrawl with the URL Inspection tool in Search Console, and for urgent cases Google points to its removals tool.
Sources
- Google Search Central: Introduction to robots.txt (used primarily to manage crawler traffic; not a mechanism for keeping a web page out of Google; a disallowed URL can still be indexed if linked from elsewhere; use noindex or password protection; media and resource files)
- Google Search Central: Block Search indexing with noindex (meta tag and X-Robots-Tag syntax, the page must not be blocked by robots.txt, recrawl timing and the URL Inspection tool)
- Google Search Central: How to specify a canonical with rel="canonical" and other methods (reasons to specify a canonical, redirects and rel="canonical" as strong signals, sitemaps as a weak signal, do not use robots.txt or the removal tool for canonicalization, do not use noindex to prevent canonical selection)