HeyMark
Content

Blog

Robots meta tag: how noindex and nofollow work

3 min read

A robots meta tag communicates indexing and link-following instructions for an HTML page. noindex asks for the page to be excluded from the index. nofollow asks for its links not to be followed; Google treats it as a hint rather than a guarantee that destinations will remain undiscovered. Google needs access to the response to read these instructions: blocking crawling can prevent it from finding the tag.

Noindex for an HTML page

Suppose a public confirmation page contains no private information, but the business wants it excluded from search. This tag in the head expresses that choice:

<meta name="robots" content="noindex" />

There is no nofollow in the example: excluding the page does not require applying that instruction to all its links. If a CMS emits noindex, follow, the follow value states the default following behavior; it does not make the page indexable.

Google explains the scope of these rules in its robots meta tag and X-Robots-Tag documentation. Check every applicable instruction when they conflict. A permissive declaration does not automatically cancel a more restrictive one.

What nofollow changes

This version applies a link-following instruction across the page:

<meta name="robots" content="nofollow" />

It is distinct from noindex. To describe the relationship of a particular link, such as a paid placement, review nofollow, sponsored and ugc attributes. Using a page-wide value to address one link may affect more content than intended.

Nofollow also cannot hide destinations. Google may discover those URLs through other links or sources. Private documents require appropriate access controls.

PDFs and other resources: the HTTP header

A PDF has no HTML head for a meta tag. The server can return this header to communicate exclusion for that resource:

X-Robots-Tag: noindex

Inspect the exact file’s response. An overly broad server rule might also affect PDFs that should appear in search. With a CMS, identify the responsible setting before changing a whole folder without examining its contents.

The common robots.txt conflict

Blocking a URL in robots.txt may prevent Google from crawling it and reading noindex. A blocked address can still appear in results based on outside references, even without its content being shown. Exclusion requires an instruction the search engine can process.

Before changing a block, establish why it exists and whether the page contains anything that must remain restricted. Opening a private area solely to expose a tag to a crawler would be the wrong fix. Noindex governs search inclusion; it does not grant or restrict access.

Verify the actual response

Check the final URL, published HTML and headers. Look for directives from the template, SEO plugin and server. For a property you manage, URL Inspection helps compare indexed information with a live test.

After correcting an accidental directive, verify the new response and record the change date. That check does not confirm an index update or establish a removal or inclusion deadline. For a public class page that should not be excluded, the task should identify the exact rule to remove, where it is generated and how the corrected response will be verified.

Plan your next post in HeyMark.

Keep the idea, review the draft with your team, and see how it performed in the accounts you connected.