Blog
Robots.txt: purpose, example and limits
Robots.txt is a public file that tells crawlers which paths on a site they may request. Its purpose is to manage crawling. Blocking a page from being fetched can still leave its address visible in Google when other pages link to it. To keep a public page out of Google, use an indexing instruction that Google can read. Protect private information with access controls.
A robots.txt example, line by line
Imagine a store with public product pages and an internal search feature that produces many combinations of results. This illustrative file asks compliant crawlers to avoid the search paths:
User-agent: *
Disallow: /search/
Sitemap: https://www.example.com/sitemap.xml
User-agent: * defines the general crawler group. Disallow: /search/ restricts that path prefix; it does not automatically block /products/. The last line points to the sitemap. The domain and paths are fictional. Before adapting a rule, check which URLs your site generates and what content depends on them.
Serve the file at the host root using the path /robots.txt. On the illustrative host https://www.example.com, that is the location to check. A store on another subdomain needs its own rules checked. Uploading a file to /blog/robots.txt does not set the host’s general crawl rules. Google’s robots.txt introduction explains the scope and limitations.
What Disallow: / changes
In the group above, Disallow: / asks crawlers to avoid every path on that host. That may be intentional in a working environment. Copying it into production would prevent compliant crawlers from reading public content. When an important page stops being crawled after a release, compare the current file with the previous version and identify the applicable crawler group.
Check the resources needed to understand the page, too. Blocking images, CSS or JavaScript indiscriminately can change what Google can render.
Choose the control that matches the task
| Task | What to examine |
|---|---|
| Avoid crawling low-value internal search combinations | Rules for those paths, after reviewing affected links and content |
| Keep a public page out of Google | A noindex directive the crawler can read |
| Restrict customer documents | Server-side authentication and authorization |
| Help crawlers discover public pages | Internal links and a sitemap |
Combining a crawl block with noindex on the same page can prevent Google from reading the instruction you need it to process. Its noindex documentation makes this access requirement explicit. Listing sensitive paths in robots.txt also exposes their names. Anyone can open the file, and bots may ignore it.
Check the published rule
Open the live file, identify the host and applicable group, and examine one URL that should remain accessible and one you intend to restrict. Check server responses, page directives and required resources. In Search Console, distinguish the last recorded crawl from a current test: a recent change may still await processing.
For a page missing from Google, continue with crawling versus indexing. To list the public pages you want discovered, read the separate explanation of XML sitemaps.
Plan your next post in HeyMark.
Keep the idea, review the draft with your team, and see how it performed in the accounts you connected.