Crawl Budget Optimizer

Patents: Crawl Rate Optimization + Web Decay Detection + URL Ownership & Redirects Source: Bill Slawski, SEO by the Sea

What Is Crawl Budget?

Crawl budget is the number of pages Googlebot will crawl on your site within a given timeframe. It's determined by two factors working together:

1. Crawl rate limit: How fast Google crawls without overwhelming your server
2. Crawl demand: How much Google actually wants to crawl your site (based on popularity, freshness requirements, and detected changes)

For small sites (under 1,000 pages), crawl budget is rarely a problem. For large sites (10,000+ pages), crawl budget is actively limiting which pages get indexed and how quickly new content appears in search results.

The waste problem: If 40% of your crawlable pages are thin, duplicate, or low-value, Googlebot is spending 40% of its crawl budget on pages that shouldn't be indexed. This delays indexing of your valuable content and signals poor site maintenance to Google.

9-Point Audit (100 points total)

Point 1: robots.txt Efficiency (0-15 points)

Goal: Block crawlers from low-value URLs to concentrate budget on important pages.

What to audit in robots.txt:

• Are admin pages blocked? (/wp-admin/, /admin/, /login)
• Are internal search result pages blocked? (/search?q=*)
• Are faceted navigation parameters blocked? (?sort=*, ?filter=*)
• Are print/preview URL patterns blocked?
• Are staging/development paths blocked?
• Are any IMPORTANT pages accidentally blocked?

Common over-blocking errors:

• Disallow: /images/ — blocks image indexing (may be intentional)
• Disallow: /css/ or Disallow: /js/ — blocks rendering resources (bad — prevents Googlebot from rendering pages)
• Disallow: / on staging but same robots.txt on production

robots.txt audit checklist:

1. Open robots.txt at yourdomain.com/robots.txt
2. Test all important pages: Google Search Console > URL Inspection > test robots.txt
3. Look for Disallow: / or overly broad rules
4. Check that all Googlebot-facing user-agents are addressed

Scoring:

• 15: robots.txt efficiently blocks all low-value URL patterns, no important pages blocked
• 10: Minor over-blocking or under-blocking, no critical issues
• 5: Some important pages blocked, or major low-value patterns not blocked
• 0: robots.txt absent, or Disallow: / blocking all crawl

---

Point 2: Sitemap Completeness (0-10 points)

Goal: Ensure Googlebot's crawl prioritization includes all important pages through an accurate, current sitemap.

Sitemap audit checks:

• Does an XML sitemap exist? (yourdomain.com/sitemap.xml)
• Is it submitted to Google Search Console?
• Does it include all important pages (not just the homepage)?
• Does it EXCLUDE low-value pages (no-index pages should not be in sitemap)
• Are lastmod dates accurate? (Don't update lastmod for CMS template changes — only for content changes)
• For large sites: are there sitemap index files organizing multiple sitemaps?
• Do all URLs in the sitemap return 200 status?

Critical error: Including noindex pages in your sitemap sends conflicting signals to Google. If you don't want the page indexed, don't include it in the sitemap.

Scoring:

• 10: Complete sitemap, all important pages included, no noindex pages, accurate lastmod
• 7: Sitemap present and submitted, minor completeness issues
• 5: Sitemap present but not submitted, or has significant gaps
• 2: Sitemap exists but is outdated or has major errors
• 0: No sitemap

---

Point 3: URL Structure Quality (0-10 points)

Goal: Ensure URLs are clean, descriptive, and don't create crawl confusion.

URL quality criteria:

• No URL parameters on important pages (use clean URLs, not ?id=42&cat=3)
• Descriptive slugs: /services/plumbing-repair not /services/service1
• Consistent subdomain handling (www.domain.com and domain.com canonicalized to one)
• Consistent trailing slash (always or never — pick one and enforce it)
• No URL encoding where plain ASCII would work
• Short slugs for important pages (depth signals importance)

Crawl-wasting URL patterns:

• domain.com/?p=142 (WordPress default permalinks)
• domain.com/category/uncategorized/tag/post-name (over-nested taxonomy)
• domain.com/page.php?id=42&lang=en&session=abc123
• Long URLs with repeated keyword stuffing (/best-seo-services-seo-agency-top-seo-company)

Scoring:

• 10: Clean, descriptive, consistent URLs throughout — no crawl-wasting parameter patterns
• 7: Mostly clean, minor inconsistencies
• 5: Some clean URLs, but parameter-based URLs for important pages
• 2: Predominantly parameter-based or poorly structured URLs
• 0: URL chaos — sessions in URLs, inconsistent patterns, no clean slugs

---

Point 4: Redirect Chain Analysis (0-10 points)

Goal: Eliminate redirect chains that waste crawl budget and bleed PageRank.

The redirect chain problem:

• Each redirect requires an additional HTTP request
• Google follows redirect chains up to ~5 hops before stopping
• PageRank is not passed at 100% through redirects — some value leaks at each hop
• Crawl budget is consumed by every hop in a redirect chain

Common redirect chains:

• HTTP → HTTP/www → HTTPS/www (3-hop chain where 1 hop is enough)
• Old URL → redirect hub page → current URL (2-hop chain)
• Expired campaign URL → category page → product page (3-hop chain)
• Migrated content with multiple redirect history layers

Audit method:

1. Crawl site with Screaming Frog
2. Check "Response Codes > 3xx" tab
3. Export all redirect chains > 1 hop
4. Fix: Direct every redirect to its final destination

Scoring:

• 10: No redirect chains longer than 1 hop; all redirects go directly to final destination
• 7: A few 2-hop chains, no 3+ hop chains
• 5: Multiple 2-hop chains, a few 3-hop chains
• 2: Widespread redirect chains, some 4+ hop chains
• 0: Redirect chain audit has never been done; chains everywhere

---

Point 5: Orphan Pages (0-10 points)

Goal: Ensure no important pages exist without internal links pointing to them.

Orphan pages are pages that exist in the site (via sitemap or direct URL) but receive zero internal links from other pages. Googlebot following links will never discover them. They can only be crawled via sitemap.

More importantly: Orphan pages receive zero PageRank from internal links, making them nearly impossible to rank regardless of content quality.

Finding orphan pages:

1. Export all URLs from your crawl tool (found via crawling)
2. Export all URLs from your XML sitemap
3. Find sitemap URLs not in the crawled list = orphan candidates
4. Verify by checking "Inlinks" count for each candidate — 0 inlinks = orphan

Action for each orphan:

• Important page: add internal links from 2+ relevant existing pages
• Low-value page: noindex and remove from sitemap, or 301 to nearest relevant page
• Historical content that's still valuable: add to a relevant category/topic hub page

Scoring:

• 10: Zero orphan pages among important content
• 7: A few orphans among lower-priority pages (not among core content)
• 5: Some important pages are orphans
• 2: Widespread orphan problem — many important pages not linked internally
• 0: Majority of site pages are orphans (no internal link structure)

---

Point 6: Thin/Duplicate Content Ratio (0-15 points)

Goal: Ensure crawl budget is not wasted on pages Google will de-index or devalue.

Every thin or duplicate page Googlebot crawls is a crawl budget unit wasted. On a site with 40% thin pages, 40% of crawl budget is wasted — and those thin pages are also dragging down site-level quality scores.

Types of crawl-wasting pages:

• Auto-generated category/tag archive pages with no unique content
• Thin product variant pages (color/size variants of the same product)
• Empty pagination pages (page 50 of a blog with no content)
• Auto-generated location pages using identical template
• Test pages, draft pages, or development pages left indexable
• Retired product/service pages returning 200 (should 301 to current)

Measurement:

• Crawl the site, export all URLs with word count
• Count pages with <300 words of main content (potential thin content)
• Calculate ratio: thin pages / total pages
• Anything above 20% thin pages = crawl budget problem

Scoring:

• 15: Under 5% thin/duplicate pages in crawlable site
• 10: 5-15% thin pages, actively being addressed
• 5: 15-30% thin pages
• 2: 30-50% thin pages
• 0: Over 50% thin/duplicate pages — site is predominantly low-quality to crawlers

---

Point 7: Server Response Time (0-10 points)

The Crawl Rate Limit connection: Googlebot adjusts its crawl rate based on your server's response time. A slow server triggers throttling — Googlebot crawls less frequently to avoid overwhelming it.

Check:

• Average server response time (TTFB — Time to First Byte)
• Response time under crawl load (different from typical user load)
• Server errors (5xx) rate — high error rate causes Googlebot to back off

Thresholds:

• TTFB under 200ms: Excellent — Googlebot crawls aggressively
• TTFB 200-500ms: Good
• TTFB 500ms-1s: Moderate — some crawl throttling likely
• TTFB over 1s: Poor — significant crawl rate reduction
• Intermittent 5xx errors: High risk — Googlebot may mark site as unreliable

Check in Google Search Console: Crawl Stats report → Average response time

Scoring:

• 10: TTFB consistently under 200ms, no 5xx errors
• 7: TTFB 200-500ms, rare 5xx
• 5: TTFB 500ms-1s, occasional 5xx
• 2: TTFB over 1s, frequent 5xx
• 0: Server regularly failing with 5xx, TTFB consistently over 2s

---

Point 8: Web Decay Detection (0-10 points)

Patent Concept: Web Decay Detection identifies sites that are not maintained. Signals: broken internal links, 404 pages with no redirects, outdated content, abandoned pages.

Google's interpretation: A site with high web decay is a less reliable information source. Crawl frequency is reduced for decaying sites.

Check:

• 404 error count (Google Search Console: Coverage report → Not Found)
• Broken internal links (Screaming Frog: Response Codes > 4xx client errors)
• Broken outbound links (external links returning 404)
• Pages with no recent updates on time-sensitive topics

Web decay remediation:

1. Fix or redirect all broken internal links
2. Fix or remove broken outbound links
3. Set up 301 redirects for any intentionally removed pages that had inbound links
4. Add "Last Updated" timestamps to content scheduled for regular review

Scoring:

• 10: Zero 404 errors in GSC Coverage report, no broken internal links
• 7: A few 404s for genuinely removed pages (with redirects in place for the important ones)
• 5: Some broken internal links, moderate 404 count
• 2: High 404 count (50+), broken internal links throughout
• 0: Site has significant web decay — hundreds of 404s, many broken links, no maintenance

---

Point 9: Overall Crawl Budget Health (0-10 points)

After completing points 1-8, this final dimension rates the overall crawl budget health holistically:

Scoring:

• 10: All 8 dimensions scoring 7+, Crawl Stats in GSC show healthy crawl volume with no response time issues
• 7: Most dimensions strong, 1-2 moderate issues being addressed
• 5: Mixed scores — some good, some problematic
• 2: Multiple dimensions scoring poorly, crawl budget is actively limiting indexing
• 0: Crawl budget is severely impacted — new content takes weeks to index, important pages not indexed

---

Scoring Summary

Point	Max	Score
robots.txt Efficiency	15	/15
Sitemap Completeness	10	/10
URL Structure Quality	10	/10
Redirect Chain Analysis	10	/10
Orphan Pages	10	/10
Thin/Duplicate Content Ratio	15	/15
Server Response Time	10	/10
Web Decay Detection	10	/10
Overall Health	10	/10
TOTAL	100	/100

Score interpretation:

• 80-100: Healthy crawl budget — new content indexing quickly, no wasted budget
• 60-79: Moderate issues — address low-scoring dimensions to improve indexing speed
• 40-59: Significant crawl budget waste — new content indexing slowly
• Below 40: Critical — major pages may not be indexed; crawl budget is severely misallocated

Priority Action Order

1. Fix robots.txt (fastest win — immediate effect on next Googlebot visit)
2. Fix redirect chains (every hop costs budget; direct to final destination)
3. Address orphan pages (add internal links to important pages)
4. Thin content audit (noindex or improve the worst offenders)
5. Fix broken links (web decay signal addressed)
6. Improve server response time (infrastructure — may require hosting upgrade)
7. Sitemap cleanup (remove noindex pages, fix lastmod)