Crawling & Indexing: How Google and AI Crawlers Find Your Site

Crawling is the process by which search engines and AI systems discover and download your pages; indexing is the process of storing and organising them so they can be ranked or cited. Crawling comes first and is non-negotiable: a page that is never crawled can never be indexed, ranked, or cited by an AI answer engine.

This guide is a supporting page to Unity SEO's Technical SEO pillar. It goes deep on the discoverability and indexability layers — the foundation that everything else in SEO sits on top of.

The Crawl → Index → Serve Pipeline

1. Crawl — A crawler (Googlebot, plus AI crawlers GPTBot, ClaudeBot, PerplexityBot) requests a URL and downloads its HTML.
2. Render — The crawler renders the page, executing JavaScript where needed, to see the content as a user would.
3. Index — The system processes and stores the content, resolving canonicals and duplicates.
4. Serve — Indexed content becomes eligible to rank in results or be cited inside an AI answer.

Each stage is a filter. Optimising stage 4 (content) is pointless if you fail stage 1 (crawl).

robots.txt — The Crawler Gatekeeper

robots.txt (at yourdomain.com/robots.txt) controls which paths crawlers may access.

• The most damaging error in SEO is a stray Disallow: / left over from a staging environment going live — it blocks the entire site from crawling.
• Disallow only what genuinely shouldn't be crawled (internal search results, infinite filter combinations, admin paths). Disallowing does not remove a page from the index — use noindex for that.
• Do not block AI crawlers if you want AI citation eligibility. Confirm GPTBot, ClaudeBot, PerplexityBot and Googlebot are allowed on your content paths.
• Link your sitemap from robots.txt: Sitemap: https://yourdomain.com/sitemap.xml.

XML Sitemaps — Your Crawl Priority List

A sitemap tells crawlers which URLs you consider important. Keep it disciplined:

• Include only indexable, canonical URLs — no 404s, no redirected URLs, no noindex pages, no parameter variants.
• Use accurate <lastmod> dates; Google reads sitemaps regularly and uses freshness signals (Google crawl budget docs).
• Submit and monitor in Google Search Console; watch the discovered-vs-indexed gap.
• For large sites, split into multiple sitemaps under a sitemap index (max 50,000 URLs / 50MB each).

Internal Linking — The Underused Crawl Signal

Internal links are how crawlers move through your site and how authority flows between pages.

• No orphan pages. A page with zero internal links is rarely discovered and rarely crawled.
• Logical hierarchy. Pillar pages link down to supporting pages; supporting pages link back up. This is the cluster structure that also builds topical authority.
• Descriptive anchor text. Tells crawlers (and AI systems) what the linked page is about — far better than "click here".

Crawl Budget — When It Actually Matters

Crawl budget is the number of URLs a crawler will fetch in a given period. It becomes a real constraint when a site exceeds ~10,000 URLs, publishes content at high frequency, or uses faceted navigation (w3era). For most Norfolk SMEs it is not the limiting factor — but for large e-commerce, listings, or directory sites it is decisive.

To preserve crawl budget (Google):

• Consolidate duplicate content so crawling focuses on unique pages.
• Eliminate soft 404s — they keep getting crawled and waste budget.
• Block parameterised URL variants and return 410 for permanently deleted pages.
• Avoid long redirect chains, which slow crawling.
• Improve server response time — faster servers let Googlebot raise the crawl rate.

Indexing Controls

Common mistake: combining robots.txt Disallow with a noindex tag — Google can't see the noindex if it's blocked from crawling, so the page may stay indexed via external links.

Diagnosing Crawl & Index Problems

Google Search Console is the ground truth:

• URL Inspection — see exactly how Google crawled, rendered and indexed a specific URL; spot "Hostload exceeded" (server capacity) issues.
• Page Indexing report — find pages excluded from the index and why (crawled-not-indexed, duplicate without canonical, soft 404, blocked by robots.txt).
• Crawl Stats — monitor crawl volume, response codes and average response time.

Part of Unity SEO's Technical SEO pillar. Sources: Google crawl budget documentation; w3era Crawl Budget Optimization Guide 2026.