Indexability
Indexability determines whether search engines can include a page in their search index. A page may be crawlable but not indexable — or indexable but excluded from search results through directives.
After this lesson you can control exactly which pages appear in Google's index using noindex, canonical tags, robots meta directives, X-Robots-Tags, and duplicate URL handling.
This lesson covers the seven indexability areas (leaves 3.2.1–3.2.7): noindex management, index coverage analysis, index bloat prevention, canonical tag control, duplicate URL handling, robots meta directive management, and X-Robots-Tag directive management.
Why This Matters
- Indexing is the gate between being found and being invisible. Pages that are not indexed do not appear in search results.
- Controlling which pages are indexed (and which are not) ensures Google's index contains only your valuable content.
- Index bloat (too many low-value pages in the index) dilutes crawl budget and can cause important pages to receive less attention.
Noindex Management
The noindex meta tag or X-Robots-Tag tells search engines not to include a page in their index.
How to implement noindex:
| Method | Implementation | Scope |
|---|---|---|
| Meta robots tag | <meta name="robots" content="noindex"> in HTML <head> | Single HTML page |
| X-Robots-Tag HTTP header | X-Robots-Tag: noindex in server response | HTML pages and non-HTML files (PDFs, images) |
| Per-directory via .htaccess | <FilesMatch "\.pdf$"> Header set X-Robots-Tag "noindex"</FilesMatch> | All files matching pattern |
When to use noindex:
| Page Type | Why Noindex |
|---|---|
| Thin or low-value pages | Pages with minimal unique content |
| Duplicate or near-duplicate pages | Version pages, print versions, tag pages |
| Thank-you / confirmation pages | Post-conversion pages with no search value |
| Internal search results | Search result pages that are infinite and low-value |
| Staging or test pages | Pages not intended for public search |
When NOT to use noindex:
- As a substitute for access control (noindex does not prevent access — it just hides the page from search).
- On pages you want to rank (obviously).
- On pages that have external backlinks (noindex prevents link equity from passing).
Noindex pitfalls:
- Googlebot must crawl the page to see the noindex directive. A page blocked by robots.txt with a noindex tag will not be visible (Google cannot crawl to see the noindex).
- If noindex is added to a sitemap, Google will ignore the sitemap signal and respect the noindex.
- Noindex does not immediately remove the page from the index. It tells Google to exclude the page at the next crawl.
Index Coverage Analysis
Index coverage analysis tracks which pages are in Google's index, which are excluded, and why.
GSC Indexing report sections:
| Section | Meaning | Action |
|---|---|---|
| Indexed | Pages in Google's index | No action needed |
| Not indexed | Pages Google chose not to index | Review reason and fix if needed |
| Excluded | Pages with explicit exclusion directive | Verify exclusion is intentional |
| Errors | Pages Google cannot index | Fix immediately |
Common "not indexed" reasons by action:
| GSC Reason | Likely Cause | Fix |
|---|---|---|
| Crawled - currently not indexed | Page has low perceived value or similarity to other pages | Improve content uniqueness, reduce duplication |
| Discovered - currently not indexed | Google knows the URL but has not crawled it yet | Build internal links, ensure crawlability |
| Duplicate without user-selected canonical | Google chose a different canonical than intended | Add explicit canonical tag |
| Page with redirect | URL redirects to another URL | Update links to point directly |
| Soft 404 | Page looks like a 404 but returns 200 | Add meaningful content or return 404 |
| Blocked by robots.txt | URL blocked from crawling | Update robots.txt if page should be crawled |
Index Bloat Prevention
Index bloat occurs when a large number of low-value pages occupy the index, consuming crawl budget and potentially diluting the site's overall index quality.
Common sources of index bloat:
| Source | Example | Prevention |
|---|---|---|
| Filtered/faceted URLs | /products?color=red&size=m&sort=price | Use noindex or robots.txt for parameter URLs |
| Pagination pages | /category/page/2, /category/page/3 | Use canonical to page 1 or implement view-all |
| Tag and category archives | /tag/seo, /category/uncategorized | Noindex low-value archive pages |
| Session IDs and tracking parameters | /product?session_id=abc&ref=xyz | Remove parameters or canonical to clean URL |
| Sort and filter variations | /products?sort=price&order=asc | Use noindex or block in robots.txt |
| Duplicate content across domains | www.example.com and example.com | Canonicalize to primary domain |
| Thin affiliate pages | Affiliate pages with minimal unique content | Noindex or improve content |
Index bloat monitoring:
| Metric | Warning Signal | Action |
|---|---|---|
| Indexed pages vs sitemap pages | Indexed count significantly higher than sitemap count | Find and remove unintended indexed URLs |
| "Crawled - not indexed" count | High and growing | Reduce low-value content, improve internal linking to important pages |
| Average indexed page quality | Low engagement on indexed pages | Audit index and prune low-value content |
Index bloat cleanup workflow:
- Export all indexed URLs from GSC (or use crawl data).
- Identify low-value pages (filtered URLs, pagination, archives).
- Add noindex tags or update robots.txt for pages that should not be indexed.
- For pages that should be consolidated, set up 301 redirects.
- Monitor index coverage report for changes.
Canonical Tag Control
The canonical tag (rel="canonical") tells search engines which URL is the preferred version when multiple URLs contain the same or similar content.
Canonical tag syntax:
<link rel="canonical" href="https://example.com/preferred-url/" />
Canonical rules and best practices:
| Rule | Implementation |
|---|---|
| Self-referencing canonical | Every page should have a canonical pointing to itself (unless there is a specific reason not to) |
| One canonical per page | Multiple canonicals are ignored |
| Absolute URLs | Use full URLs including protocol and domain, not relative paths |
| Consistent domain | Use the preferred domain (www vs non-www) consistently |
| Lowercase URLs | /Page/ and /page/ are different URLs; use lowercase consistently |
| Trailing slash consistency | Choose a convention (with or without trailing slash) and stick to it |
Canonical mistakes to avoid:
- Using canonical on paginated pages: canonicalize paginated pages to themselves, not to page 1.
- Cross-domain canonical without a specific reason: only use cross-domain canonical when content is intentionally duplicated.
- Canonical that points to a redirect: point to the final destination URL, not an intermediate redirect.
- Missing canonical on syndicated content: if content is published on multiple sites, each secondary site should canonicalize to the original.
Duplicate URL Handling
Duplicate URLs occur when the same content is accessible through multiple URLs. Duplicate content can confuse search engines about which page to rank.
Common duplicate URL patterns:
| Pattern | Example | Solution |
|---|---|---|
| WWW vs non-www | www.example.com and example.com | 301 redirect to preferred domain |
| HTTP vs HTTPS | http://... and https://... | 301 redirect HTTP to HTTPS |
| Trailing slash inconsistency | /page and /page/ | Choose one and redirect the other |
| Case sensitivity | /Page and /page | Use lowercase URLs consistently |
| Parameter variations | /product?id=1 and /product?id=1&ref=abc | Remove unnecessary parameters |
| Pagination duplicates | /category/page/2 with rel=prev/next | Use canonical to self or view-all |
| Printer-friendly versions | /page and /page?print=1 | Noindex print versions |
Duplicate content detection:
- GSC Indexing report: "Duplicate without user-selected canonical" indicates Google found duplicates.
- Screaming Frog crawl: URLs with the same content hash (checksum) are duplicates.
- Manual review: search
site:domain.com/page-urlto see which URLs Google considers duplicates.
Duplicate content is not a penalty. Google simply picks one URL to show. The issue is that Google may not pick your preferred URL. Canonical tags and redirects ensure Google shows the right URL.
Robots Meta Directive Management
Robots meta directives (tags in HTML <head>) control how search engines interact with a specific page.
Available robots meta directives:
| Directive | Meaning | Usage |
|---|---|---|
index | Allow indexing (default) | Valuable pages |
noindex | Do not index this page | Thin pages, low-value pages |
follow | Follow links on this page (default) | All pages |
nofollow | Do not follow links on this page | Spammy content, login-walled pages |
noarchive | Do not show cached version | Subscription content, dynamic pages |
nosnippet | Do not show a text snippet or preview | Restricted content |
max-snippet:-1 | Allow unlimited snippet length | Content you want in featured snippets |
Implementation:
<!-- Do not index this page, follow links -->
<meta name="robots" content="noindex, follow">
Best practices:
- Use
noindex, followfor low-value pages that you still want to pass link equity. - Avoid
noindex, nofollowunless the page has no value (no index, no link equity). - Test directives after implementation using GSC URL Inspection tool.
X-Robots-Tag Directive Management
X-Robots-Tag is an HTTP header that provides more flexibility than HTML meta tags, especially for non-HTML resources.
X-Robots-Tag syntax (server response header):
X-Robots-Tag: noindex, nofollow
X-Robots-Tag: noindex, follow
X-Robots-Tag: none
Use cases for X-Robots-Tag:
| Resource Type | Why Use X-Robots-Tag |
|---|---|
| PDF files | Cannot add HTML <meta> tags to PDFs |
| Images | Control whether images appear in image search |
| JavaScript/CSS files | Ensure these are crawlable (via unavailable_after or allow) |
| Dynamic content | Apply based on request path (e.g., noindex on staging environment) |
Implementation examples:
Apache (.htaccess):
<FilesMatch "\.pdf$">
Header set X-Robots-Tag "noindex, follow"
</FilesMatch>
<If "%{HTTP_HOST} == 'staging.example.com'">
Header set X-Robots-Tag "noindex, nofollow"
</If>
Nginx:
location ~* \.pdf$ {
add_header X-Robots-Tag "noindex, follow";
}
X-Robots-Tag overrides HTML meta robots. If both are present, the X-Robots-Tag takes precedence.
Workflow
- Audit current index status: Export indexed pages from GSC and compare to your expected index.
- Review noindex implementations: Ensure noindex is applied correctly to low-value pages.
- Check canonical tags: Verify self-canonical, absolute URLs, correct domain.
- Identify duplicates: Find and resolve duplicate URL patterns.
- Check for index bloat: Identify low-value pages consuming index space.
- Implement X-Robots-Tag where needed: PDFs, staging environments, non-HTML resources.
- Monitor index coverage: Weekly check in GSC.
Common Mistakes
- Noindex on robots.txt-blocked pages: Google cannot crawl to see the noindex. Remove robots.txt blocking or remove noindex.
- Inconsistent canonical across page versions: The same page should have the same canonical regardless of how users access it.
- Canonical to page 1 from paginated pages: Use canonical to self or view-all, not to page 1.
- Noindex on pages with external backlinks: Noindex prevents link equity from passing. Use redirect or consolidate content instead.
- Not monitoring index bloat: Index bloat grows silently. Check the indexed page count trend monthly.
Checklist
- Noindex is correctly applied to all low-value page types.
- Noindex is NOT applied to pages that should rank (verify).
- GSC Indexing report is error-free (no critical errors).
- Self-canonical tags are present on every page.
- Canonical tags use absolute URLs with the preferred domain.
- Duplicate URL patterns are identified and consolidated.
- Index bloat sources (facets, tags, pagination, filters) are controlled.
- X-Robots-Tag is configured for non-HTML resources (PDFs) if needed.
- Staging environment is protected with noindex directives.