Data Quality & Page Quality Control
Data quality and page quality control are the most important aspects of programmatic SEO. One flawed template or incorrect data set can produce thousands of low-quality pages.
Learning Focus
After this lesson you can validate source data, set quality thresholds, implement automated QA checks, and prevent thin pages at scale.
This lesson covers the seven quality control areas (leaves 9.3.1–9.3.7): source data validation, missing data handling, duplicate page detection, thin page prevention, quality threshold rules, user value scoring, and automated QA checks.
Source Data Validation
Validate the accuracy and completeness of source data before generating pages.
Data validation checks:
| Check | Method |
|---|---|
| Completeness | Every required field has data for each record |
| Format consistency | Dates, numbers, text fields follow consistent format |
| Uniqueness | No duplicate records in the dataset |
| Accuracy | Data matches authoritative sources |
| Freshness | Data is within the acceptable age range (e.g., < 90 days) |
| Relevance | Data is appropriate for the intended page purpose |
Validation workflow:
- Define required fields for each record type.
- Run automated validation scripts before generation.
- Flag records with missing or invalid data for manual review.
- Reject records below quality thresholds.
Missing Data Handling
Define how to handle records with missing or incomplete data.
Missing data strategies:
| Strategy | Description | Risk |
|---|---|---|
| Skip record | Do not generate a page for incomplete records | Missed opportunity if data is mostly complete |
| Use default value | Substitute a placeholder for missing data | May result in generic, low-value pages |
| Dynamic fallback | Use a different template for incomplete records | Template complexity |
| Partial page | Show only available data, mark missing sections | User experience if page feels incomplete |
| Noindex incomplete | Generate but noindex pages below quality threshold | Crawl budget still consumed |
Recommended approach:
- Require minimum data completeness threshold (e.g., 70% of fields populated).
- Pages below threshold: do not generate or noindex.
- Pages above threshold: generate but flag missing fields for manual review.
Duplicate Page Detection
Detect and prevent duplicate or near-duplicate pages.
Duplicate detection methods:
| Method | Application |
|---|---|
| URL uniqueness check | No two pages should have the same URL |
| Content similarity scoring | Compare content across generated pages using similarity checksum |
| Title uniqueness | Every page must have a unique title tag |
| H1 uniqueness | Every page must have a unique H1 |
| Template variation check | Ensure template produces distinct output for each input |
Duplicate prevention:
- Design templates to produce unique content for each input.
- If two input combinations produce near-identical content, either consolidate or noindex one.
- Run duplicate detection on a sample before full rollout.
Thin Page Prevention
Prevent pages with insufficient unique content.
Thin page criteria:
| Criterion | Thin Page Signal |
|---|---|
| Word count | < 300 words of unique body content |
| Unique content ratio | Content should be substantially differentiated from other generated pages. While no fixed uniqueness threshold exists publicly, aim for clear differentiation. Use canonical tags for intentionally overlapping content. |
| Template repetition | Same content structure with only keyword substitution |
| User value | User can get the same information from a search results page |
| Metadata uniqueness | Title/description are identical to other pages |
Thin page prevention strategies:
- Set minimum word count thresholds per template type.
- Require unique content modules on every page.
- Run thin page detection on a sample before full rollout.
- Noindex or suppress pages below quality thresholds.
Quality Threshold Rules
Core Concept
Define the minimum quality threshold for a generated page to be indexed.
Quality threshold factors:
| Factor | Threshold |
|---|---|
| Word count | Minimum 300 words (adjust by page type) |
| Unique content | Content should be substantially differentiated from other generated pages. While no fixed uniqueness threshold exists publicly, aim for clear differentiation in angle, depth, or data. |
| Schema validity | Valid schema with required properties |
| Metadata completeness | Unique title, description, H1 present |
| Data completeness | Minimum 70% of data fields populated |
| Image presence (if required) | At least one image (for product/listing pages) |
| Internal links | At least 2 contextual internal links |
Threshold enforcement:
- Pages below threshold: noindex (do not remove — keep for data completeness but exclude from index).
- Pages meeting threshold: index, follow.
- Pages significantly above threshold: prioritize in sitemaps.
User Value Scoring
Score each generated page for user value.
User value scoring factors:
| Factor | Measurement |
|---|---|
| Search demand | Does the combination have search volume? |
| Answer uniqueness | Can the user get this information elsewhere more easily? |
| Actionability | Can the user take action from this page? |
| Information depth | Does the page provide sufficient detail? |
| Comparison value | Does the page help user compare options? |
Value scoring workflow:
- Score each page during generation (based on data completeness, search demand).
- For pages that score below the value threshold, suppress from search or noindex.
- Periodically review value scoring against actual user engagement data.
Automated QA Checks
Implement automated quality assurance checks for every generated page.
Automated QA checks:
| Check | Tool | Pass Criteria |
|---|---|---|
| HTTP status | Crawler | 200 (not 4xx, 5xx, or redirect) |
| Title tag | Crawler | Present, within length, unique |
| Meta description | Crawler | Present, within length, unique |
| H1 | Crawler | Present, unique |
| Schema validation | Rich Results Test API | No critical errors |
| Internal links | Crawler | All links return 200 |
| Page speed | Lighthouse API | LCP < 2.5s, CLS < 0.1 |
| Mobile rendering | Lighthouse (mobile audit) | Pass |
| Thin content | Word count, similarity check | Pass thresholds |
QA automation workflow:
- Generate pages in staging.
- Run automated QA checks on all pages.
- Flag pages that fail QA for review.
- Fix template issues (if systemic) before production.
- After production, run QA on a weekly sample.
Workflow
- Validate source data before generation: check completeness (all required fields populated), format consistency, uniqueness (no duplicate records), accuracy (matches authoritative sources), and freshness (within acceptable age range).
- Define missing data handling rules: require minimum data completeness threshold (e.g., 70% of fields populated). Pages below threshold: do not generate or noindex.
- Implement quality threshold rules: minimum 300 words unique content, valid schema, unique metadata, minimum data completeness, at least one image (for product/listing pages), at least 2 internal links.
- Run automated QA checks on all generated pages: HTTP status, metadata uniqueness, H1 uniqueness, schema validity, internal link health, page speed, mobile rendering, thin content detection.
- Score pages for user value: search demand, answer uniqueness, actionability, information depth, and comparison value. Noindex pages below value threshold.
Common Mistakes
warning
- Setting quality thresholds too low: A 100-word page with a title swap from a template is not "good enough." Set meaningful thresholds (300+ words, substantially unique content) and enforce them.
warning
- Relying on post-launch QA only: Finding thin or broken pages after they are live in the index means algorithmic exposure before you catch the problem. Run QA in staging before production.
- No duplicate detection: Near-identical pages from similar data inputs (e.g., "dentist in austin" vs "dentist in dallas") may look unique to a word count check but are functionally duplicate. Use content similarity scoring.
- Defaulting to index for all generated pages: New programmatic pages should default to noindex until they pass quality thresholds. Indexing everything and deduplicating later is harder.
- Neglecting data freshness: Data that was accurate at generation time becomes stale over time. If prices, availability, or dates are no longer correct, the page provides bad user experience. Automate data refresh.
Checklist
- Validate source data for completeness, accuracy, format consistency, and freshness
- Define minimum data completeness threshold (e.g., 70%) for page generation
- Set quality thresholds: word count, unique content, schema validity, metadata uniqueness
- Implement duplicate page detection (URL uniqueness, content similarity, title/H1 uniqueness)
- Run automated QA on all pages before production (HTTP, metadata, schema, speed, mobile)
- Default new pages to noindex; index only when quality thresholds are met
- Score pages for user value and noindex pages below value threshold
- Automate data refresh pipeline to prevent stale data