Module 15: Crawling & Indexing Patents

85+ patents across web crawling, document indexing, URL canonicalization, sitemap processing, duplicate detection, JavaScript rendering, real-time indexing, and mobile-first.

Overview

Before any ranking can happen, Google must crawl and index your content. These 85+ patents reveal how Google discovers pages, assigns crawl priority, processes JavaScript, handles duplicates, and determines what gets indexed vs. filtered out.

---

Web Crawling (12 Patents)

Crawl Budget Allocation

How Google Allocates Crawl Budget:

Crawl Priority Factors:

• PageRank / internal link equity
• Update frequency (pages that update often get crawled more)
• Server response time (fast = more pages crawled)
• Site quality score (low quality = reduced budget)
• sitemap submission and priority signals

---

Crawl Frequency Determination

Update Pattern Recognition:

• Google tracks how often content changes
• Pages that change frequently get more frequent crawls
• Pages with no changes get less frequent crawls
• Breaking news pages can be re-crawled within minutes

---

Document Indexing (15 Patents)

Caffeine Architecture (2010)

Near-Real-Time Indexing:

• Replaced old batch-update index with continuous updates
• New content can appear in results within minutes to hours
• Distributed crawling across thousands of servers
• Incremental updates without full index rebuilds

Before Caffeine: New content took weeks to index After Caffeine: Quality content can rank within hours

---

Index Partitioning

How the Index Is Structured:

• Documents sharded across thousands of servers
• Each query hits multiple shards simultaneously
• Results merged and ranked in real-time
• Different index tiers: fresh index, main index, supplemental

---

URL Canonicalization (8 Patents)

US9081861B2 - URL Canonicalization

Year: 2008-2015

URL Normalization Rules:

1. Lowercase all characters
2. Remove trailing slashes (or add them consistently)
3. Resolve relative URLs to absolute
4. Remove default ports (:80, :443)
5. Sort query parameters consistently
6. Remove tracking parameters (utm_*, fbclid, etc.)

---

Common Canonicalization Issues

Problem	Impact	Fix
www vs. non-www	Splits link equity	Redirect to canonical version
HTTP vs. HTTPS	Splits link equity	Redirect all HTTP to HTTPS
Trailing slash inconsistency	Duplicate pages	Canonicalize consistently
Parameter variations	Thousands of duplicate URLs	Block via robots.txt or canonical
Case sensitivity	/Page vs. /page = two URLs	Redirect all to lowercase

---

Parameter Handling

Google Parameter Types:
- Session IDs: ?sessionid=abc → FILTERED (not indexed)
- Tracking params: ?utm_source=email → FILTERED
- Content-changing params: ?color=red → INDEXED separately
- Sort/filter: ?sort=price → Often FILTERED (canonical to main)

Best Practice: Use Google Search Console parameter handling
to tell Google which parameters change content vs. don't.

---

Sitemap Processing (5 Patents)

XML Sitemap Interpretation

What Google Reads from Sitemaps:

• URL list (discovery signal, not ranking)
• <lastmod> — when content was last modified
• <changefreq> — hint for crawl frequency
• <priority> — relative priority (0.0-1.0)

Important: Sitemaps are discovery tools, not ranking tools. They do NOT directly influence ranking — only crawl efficiency.

---

Sitemap Best Practices (Patent-Grounded)

[ ] All important URLs included
[ ] No noindex URLs in sitemap
[ ] No redirect URLs (only canonical destinations)
[ ] lastmod dates accurate (not inflated)
[ ] Submitted and processed in Search Console
[ ] Large sites: sitemap index with multiple sitemaps
[ ] Image sitemap for important images
[ ] Video sitemap for indexable video content

---

Duplicate Detection (10 Patents)

US7734627B1 - Document Fingerprinting (Revisited)

In Crawl Context:

• Documents fingerprinted during crawl
• Near-duplicates identified immediately
• Only canonical version indexed fully
• Duplicate versions receive minimal signals

---

Cross-Domain Duplicate Handling

Syndication and Duplicate Content:

Scenario	Google's Action
Your content syndicated elsewhere	Original indexed; copies may be filtered
You scraped content	Your version filtered; original indexed
Press release republished	Multiple versions compete; original preferred
Canonical tag points to original	Your copy defers credit to canonical

Best Practice: Always add canonical tag pointing to original source when publishing syndicated content.

---

JavaScript Rendering (8 Patents)

Dynamic Rendering Detection

How Google Handles JavaScript:

Crawl Phase 1: HTML Download
- Googlebot downloads raw HTML
- Renders immediately from HTML only
- JavaScript assets in render queue

[TIME DELAY: Hours to Weeks]

Render Phase 2: JavaScript Execution
- Headless Chrome renders page
- Dynamic content extracted
- JavaScript-injected text indexed
- Rendered version compared to raw HTML

Critical Risk: Content only visible after JavaScript execution may not be indexed immediately, or at all for lower-priority pages.

---

JavaScript SEO Implications

Index via JavaScript (Risky):

• Dynamic content (React, Angular, Vue SPA apps)
• Lazy-loaded content
• AJAX-loaded content below fold

Solutions:

1. Server-Side Rendering (SSR) — Best solution. Content in initial HTML.
2. Static Site Generation (SSG) — Pre-rendered HTML for all pages.
3. Dynamic Rendering — Serve pre-rendered version to Googlebot.

---

Real-Time Indexing (6 Patents)

Breaking News Detection

Real-Time Indexing Triggers:

• Sudden spike in queries for related topics
• News publisher content flagged for fast indexing
• Social signal velocity detection
• Verified news sources prioritized

For Publishers: Use IndexNow (Bing, now Google) to push URL notifications immediately after publishing.

---

Rapid Re-Crawl Triggers

What Triggers Fast Re-Crawl:

• Significant page content change detected
• New internal links pointing to page
• External link acquisition
• Search Console "request indexing" submission
• sitemap <lastmod> update

---

Mobile-First Indexing (5 Patents)

Mobile Content as Primary Index (Since 2018)

Google's Primary Index Is Now Mobile:

• Mobile version of your page is what's indexed
• Desktop version is secondary
• If content is hidden on mobile (accordion, tabs), it may have lower weight

---

Mobile Usability Signals

Signal	Impact
Viewport meta tag	Required for mobile-friendly classification
Tap target size	Small targets = mobile usability failure
Font size	<12px = readability failure
Content width	Horizontal scroll = layout failure
Intrusive interstitials	Ranking penalty (2017 patent)

---

Responsive Design Detection

Google detects:
- viewport meta tag
- @media queries in CSS
- Same HTML served to all devices
- vs. separate m.site.com (less preferred)
- vs. dynamic serving (some risk)

Best Practice: Responsive design with same HTML for all devices. Google can verify the design adapts correctly.

---

Technical SEO Implications

Crawl Efficiency Checklist

[ ] All important pages crawlable (no robots.txt blocks)
[ ] No noindex on key pages
[ ] XML sitemap submitted and processed
[ ] Canonical tags on all duplicate/near-duplicate pages
[ ] Server-side render critical content (not just JS)
[ ] Mobile-friendly design verified
[ ] Crawl budget not wasted on thin/duplicate pages
[ ] URL structure clean (no excessive parameters)
[ ] hreflang for international content
[ ] Redirect chains max 1 hop (301 → final destination)

---

Key Patents Referenced

Patent	Title	Year
US9081861B2	URL Canonicalization	2008-2015
US7734627B1	Document Fingerprinting	2005-2010
Multiple	Caffeine Indexing Architecture	2010
Multiple	JavaScript Rendering (8 patents)	Ongoing
Multiple	Mobile-First Indexing (5 patents)	2018+

---

Next Steps

1. Structured Data Module — Schema markup
2. Domain & URL Module — Domain signals
3. Crawl Budget Optimizer Audit — Apply these techniques