Not every page belongs in the Google index – and not every page that should be there ends up there automatically. Proper control of indexing determines which content is visible in search and which is not.
Incorrect settings can lead to important pages not being found or Google wasting unnecessary resources on irrelevant content. That's why it's essential to know exactly when index, noindex, follow or nofollow make sense.
Indexing as Part of Search Engine Processing
Google processes content in three steps:
- Crawling: Googlebot searches the web and retrieves pages.
- Indexing: Google decides whether a page should be included in the search index.
- Ranking: Google evaluates indexed pages and positions them in search results.
A page can be crawled but not indexed – and an indexed page can remain invisible if it doesn't rank well enough. That's why targeted indexing control is crucial.
How to Control Indexing Strategically?
SEO experts use various techniques to give Google clear signals:
- Meta-Robots-Tags (index, noindex, follow, nofollow)
- X-Robots-Tag in HTTP-Header (for non-HTML files like PDFs)
- robots.txt (regulates crawling, but not directly indexing)
- Canonical-Tags (avoiding duplicate content)
- Google Search Console (manual control of indexing)
Proper configuration is crucial to avoid pointless indexing and ensure that relevant content appears in search results.
Misconfigurations and Their Consequences
- noindex on important pages → traffic drop
- index on irrelevant pages → duplicate content issues, crawl budget waste
- nofollow on internal links → link juice loss, worse rankings
In the next sections, we'll look at when which setting makes sense and how typical mistakes can be avoided.
Index, Noindex, Follow, Nofollow – When to Use Which Setting?
The right indexing strategy determines whether a page is found, evaluated and ranked by Google. But not every page should be indexed and not every link should be followed by Google.
1. Index vs. Noindex – When Should a Page Be in the Index?
- index: The page is indexed by Google and can appear in search results.
- noindex: Google does not include the page in the index – it cannot be found via search.
When Should a Page Be Indexed?
-
Important content with search potential:
- Product pages, blog articles, category pages with added value
- High-quality landing pages intended to generate traffic
-
Pages with unique content and SEO relevance:
- Content optimized for search queries
- Topics that don't have strong internal or external competition on your own website
When Should a Page Be Set to Noindex?
-
Duplicate or automatically generated content:
- Paginated pages (if no canonical solution exists)
- Print versions, PDF duplicates
-
Irrelevant or technical pages:
- Internal search results pages, filter URLs
- Terms and conditions, privacy policies, admin panels
Example of a Noindex Tag:
<meta name="robots" content="noindex">
Advanced SEO Strategies: When to Combine Noindex, Nofollow, Index and Follow?
Not every page needs a simple index, follow setting. In certain cases, a targeted combination of noindex, nofollow, index and follow is required to control indexing, avoid duplicate content and efficiently structure internal link building.
Combinations and Their Effects
| Setting | What happens? | When to use? | Possible risks |
|---|---|---|---|
| noindex, follow | The page is not indexed, but Google follows the links on it. | Tag pages, internal search results pages, archive pages that should pass link juice. | May result in Google paying less attention to these links since the page itself is not indexed. |
| noindex, nofollow | The page is not indexed, and Google does not follow the links. | Login pages, private dashboards, test environments or other non-public content. | Google may eventually remove the page completely from crawling if no internal links point to it. |
| index, nofollow | The page is indexed, but all outgoing links are ignored. | Pages with user-generated content to block uncontrolled external links. | Weakens internal linking and can hinder the distribution of link juice. |
| robots.txt vs. Meta-Tags | Robots.txt blocks crawling, while noindex prevents indexing. | Avoiding indexing of technical pages or resources like images or PDFs. | robots.txt prevents Google from crawling the page – if the URL is already known, it can still be indexed. |
Best Practices for the Right Combination
- noindex, follow is useful for pages that pass link juice but shouldn't appear in search results.
- noindex, nofollow should only be used for security-critical or irrelevant pages that should neither be crawled nor indexed.
- index, nofollow is rarely useful and should only be used in special cases.
- Important pages should not be blocked via robots.txt, but via noindex, so Google understands the deindexing.
In the next section, we'll look at practical examples for different website types and which indexing strategy works best there.
Practical Examples: Optimal Indexing for Different Website Types
Depending on the website type, there are different requirements for indexing. While e-commerce sites often need to optimize category pages and product pages, blogs need to deal with tag pages and archives. Here are proven strategies for different scenarios.
1. E-Commerce Websites (Online Shops)
| Page Type | Recommended Setting | Justification |
|---|---|---|
| Product Pages | index, follow | Generate organic traffic and should appear in search results. |
| Category Pages | index, follow | Important SEO pages often found through generic search terms. |
| Filter and Search Pages | noindex, follow | Avoiding duplicate content, but passing link juice to relevant pages. |
| Checkout and Shopping Cart Pages | noindex, nofollow | Not intended for search engines, no SEO relevance. |
| Print Versions of Pages | noindex, follow | Should not enter the index but still support internal linking. |
2. Content Websites & Blogs
| Page Type | Recommended Setting | Justification |
|---|---|---|
| Blog Articles | index, follow | Main source of organic traffic, should be fully indexed. |
| Category Pages | index, follow or noindex, follow | Depending on strategy – if optimized, a category page can be indexed. Otherwise better noindex to avoid duplicate content. |
| Tag Pages | noindex, follow | Mostly automatically generated pages with little inherent value, but important for internal linking. |
| Author Archives | noindex, follow | Prevents duplicate content from multiply indexed articles on author and category pages. |
| Paginated Pages (Page 2, 3, …) | noindex, follow or index, follow with Canonical to page 1 | Depending on SEO strategy, to avoid duplicate content and use crawl budget efficiently. |
3. Corporate Websites & SaaS Platforms
| Page Type | Recommended Setting | Justification |
|---|---|---|
| Landing Pages for SEO | index, follow | Important pages for organic traffic and conversions. |
| Terms & Privacy | noindex, follow | Must be present for legal reasons but is not relevant for SEO. |
| Login and Dashboard Pages | noindex, nofollow | Must not appear in the index and should not enable link juice passing. |
| FAQ Pages | index, follow | Helpful for SEO, especially with structured data for rich snippets. |
4. News Portals & Publishing Sites
| Page Type | Recommended Setting | Justification |
|---|---|---|
| News Articles | index, follow | Important source of traffic, should be indexed. |
| Archive Pages | noindex, follow | Older content that is no longer relevant should not be indexed but still distribute link juice. |
| Special Pages for Campaigns | index, follow or noindex, follow | Depending on relevance for search engines – time-limited campaign pages often with noindex. |
| RSS Feeds | noindex, follow | Not needed for SEO but pass link juice to relevant pages. |
Tools for Controlling and Optimizing Indexing
Even with a well-thought-out indexing strategy, errors can occur that result in important pages not being included in the Google index or irrelevant content remaining indexed. To identify and fix such problems early, specialized SEO tools should be used regularly.
Google Search Console: The Most Important Tool for Indexing Control
The Google Search Console (GSC) provides detailed insights into a website's indexing. Particularly relevant reports are:
- Check indexing status: Under "Indexing – Pages" it shows which URLs are indexed and which are not.
- Crawled but not indexed pages: Pages that Google has visited but not included in the index – often an indication of quality issues or incorrect meta tags.
- Manual inspection of individual URLs: The "URL Inspection" tool can test whether a page is indexed or blocked.
- Removal requests from the index: If pages have been indexed accidentally, they can be temporarily removed from search results via the "Request Removal" function.
Regular checks in Search Console help to avoid indexing errors and ensure the visibility of important pages.
Screaming Frog SEO Spider: Technical SEO Analysis
With Screaming Frog, technical SEO problems that affect indexing can be identified. The tool can crawl a website and shows, among other things:
- Pages with noindex that should actually be indexed.
- Pages with nofollow that prevent internal link juice passing.
- Redirect chains or faulty canonical tags that affect indexing.
- Faulty status codes like 404 or 5xx pages that Google might exclude from indexing.
Screaming Frog is particularly helpful for quickly identifying and specifically fixing technical errors.
robots.txt Tester: Identifying Crawling Problems Early
The robots.txt file controls which pages Google is allowed to crawl and which not. An incorrect entry can result in entire directories or even the entire website being inaccessible to Google.
With the Google robots.txt tester you can check:
- Which areas of the website are blocked for Google by disallow rules.
- Whether important pages have been incorrectly excluded.
- Whether the file is correctly formatted and Google understands the instructions.
Other Useful Tools for Indexing Analysis
- Ahrefs & SEMrush: Check which pages are actually indexed in Google and which don't rank.
- Site query in Google: With site:yourwebsite.com you can see which pages have been indexed by Google.
- Logfile Analysis: For larger websites, analyzing server logfiles can reveal which pages Google actually crawls and how often.
Best Practices for an Optimal Indexing Strategy
Proper control of indexing is crucial for a successful SEO strategy. Those who specifically tell Google which pages should be indexed and which not ensure that valuable content remains visible, while irrelevant or duplicate pages don't unnecessarily consume resources.
Key Insights from This Post
- Indexing is not the same as crawling: Google can crawl pages without indexing them – and conversely, an already known URL can remain in the index despite robots.txt blocking.
- Targeted use of meta tags: index, noindex, follow and nofollow should be used consciously to send the right signals to search engines.
- Avoid technical errors: Pages that are important for SEO should not be accidentally blocked by noindex or robots.txt.
- Regular monitoring is essential: With Google Search Console, Screaming Frog and other SEO tools, indexing problems can be identified and fixed early.
- Not everything needs to be indexed: Filter pages, search results pages or archived content should be marked with noindex to avoid duplicate content and unnecessary crawling processes.
- Consider internal linking: nofollow should be used sparingly on internal links to avoid losing valuable link juice.
Recommended Approach for Clean Indexing
- Define all important page types – Which pages should be indexed, which not?
- Consistently implement meta tags – Every page should have a clear indexing strategy.
- Combine robots.txt and meta-robots sensibly – Don't unnecessarily block important pages.
- Use canonical tags to avoid duplicate content – Especially important for shops and blogs.
- Check Google Search Console regularly – Quickly identify and fix errors in indexing.
Summary
Those who control their indexing strategically avoid problems with duplicate content, inefficient crawling and wasted crawl budget. The combination of the right meta tags, a clear structure and regular analysis ensures that Google indexes the truly relevant pages and optimally positions them in search results.
This lays the foundation for a sustainable and effective SEO strategy.
