Technical SEO: The Complete 2026 Guide for UK Businesses

Technical SEO is the practice of optimising a website's infrastructure — crawling, indexing, performance, architecture, and structured data — so that search engines and AI answer engines can discover, understand, trust, and serve its content. It is the foundation layer beneath content and authority: without it, the best content in Norfolk will never be found, indexed, or cited.

In 2026 technical SEO matters more than ever, not less. AI answer engines (Google AI Overviews, Perplexity, ChatGPT, Gemini) retrieve content through dedicated crawlers and can only cite what they can reach, parse, and verify. A technically broken site is invisible to AI search by definition — and as confirmed by Google, AI discovery remains "almost completely bound to HTML pages and normal web pages" (Search Engine Journal).

This pillar guide covers the full technical stack. Two areas — crawling & indexing and site migration — are deep enough to warrant their own supporting guides, linked below.

The Four Layers of Technical SEO

Unity SEO structures technical SEO as a Four-Layer SEO Framework, fixed bottom-up because each layer depends on the one beneath it:

1. Understandability
2. Performance
3. Indexability
4. Discoverability

1. Discoverability — Can search engines and AI crawlers find your URLs? (crawling, robots.txt, sitemaps, internal linking)
2. Indexability — Once found, can pages be indexed cleanly without duplication or canonical conflicts?
3. Performance & experience — Do pages load fast and respond instantly? (Core Web Vitals)
4. Understandability — Can machines parse the meaning and trust the entity? (structured data, semantic HTML, entity clarity)

Work always proceeds bottom-up: there is no value in optimising structured data on a page Googlebot cannot crawl.

Layer 1 — Crawling & Discoverability

Crawling is the very first stage of search and AI processing. If a page is not crawled, nothing downstream can happen.

• robots.txt — Controls crawler access. The single most damaging technical SEO error is accidentally disallowing important paths. Critically, do not block AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Googlebot) if you want to be eligible for AI citation.
• XML sitemaps — Keep them current; include only indexable, canonical URLs (no 404s, redirects, noindex, or parameter variants), and use accurate <lastmod> values. Google reads sitemaps regularly (Google crawl budget docs).
• Internal linking — Your most powerful, most underused crawl signal. A logical internal link structure tells crawlers what matters and distributes authority. Orphan pages (no internal links) are rarely crawled.
• Crawl budget — Only a material concern once a site exceeds ~10,000 URLs, publishes at high frequency, or uses faceted navigation. For most Norfolk SMEs it is not the bottleneck; for large e-commerce and listings sites it is decisive.

> Crawling and indexing are deep enough to need their own playbook. See the supporting guide: Crawling & Indexing: How Search Engines and AI Find Your Pages.

Layer 2 — Indexing & Indexability

Crawled is not the same as indexed. Indexing decides whether a page is eligible to rank or be cited at all.

• Canonical tags — Declare the single authoritative version of duplicate or near-duplicate pages. Self-canonicalise unique pages; never canonicalise to a redirected or noindexed URL.
• noindex vs robots disallow — A noindex tag keeps a page out of the index but still allows crawling (so the directive is seen); a robots.txt disallow blocks crawling so Google may never see the noindex. Use them deliberately, never together for the same intent.
• Duplicate content — Consolidate it. Duplicate content wastes crawl budget and dilutes ranking signals across competing URLs.
• Soft 404s — Pages that return 200 but show "not found" content keep getting crawled and waste budget. Return proper 404/410 status codes for genuinely missing pages.

Validate the live index state with Google Search Console's URL Inspection and Page Indexing reports — these are the ground truth, not third-party estimates.

Layer 3 — Performance & Core Web Vitals

Core Web Vitals are Google's user-experience signals and a confirmed (if modest) ranking factor. The 2026 thresholds:

Good≤ 2.5 s

Needs improvement2.5 – 4.0 s

Poor> 4.0 s

2.5 s

INP replaced First Input Delay in March 2024 and is now the most commonly failed Core Web Vital — roughly 43% of sites fail the 200ms INP threshold in 2026 (Search Engine Hub). It is therefore the highest-leverage performance fix for most sites.

Highest-impact fixes:

• LCP — optimise and preload the hero image, use modern formats (AVIF/WebP), eliminate render-blocking CSS/JS, ensure fast server response (TTFB).
• INP — break up long JavaScript tasks, defer non-critical scripts, minimise main-thread work; on Astro this means shipping less JS and using islands selectively.
• CLS — set explicit width/height on images and embeds, reserve space for ads/fonts, avoid inserting content above existing content.

Measure field data (real users) via the Chrome UX Report and Search Console's Core Web Vitals report; debug with Lighthouse and PageSpeed Insights. See also Unity SEO's dedicated Core Web Vitals guide.

Layer 4 — Understandability: Structured Data & Semantic HTML

This layer makes content machine-readable and the business a verifiable entity — the prerequisite for AI citation.

• Structured data (JSON-LD) — Organization/LocalBusiness for the entity; Article + Author for E-E-A-T; FAQPage for AI Overview eligibility; BreadcrumbList for navigation. Implementation detail lives in Unity SEO's Schema Markup guide.
• Semantic HTML — Correct heading hierarchy (one H1, logical H2/H3), descriptive link text, real lists and tables. Clean semantics improve both accessibility and AI extraction.
• HTTPS, mobile-first, hreflang — HTTPS is a baseline ranking factor; Google indexes the mobile version first; hreflang handles UK/international or multi-language variants without duplicate-content penalties.
• Entity consistency — NAP (Name, Address, Phone) identical across the site, GBP, Companies House and directories. Inconsistency actively harms both local ranking and AI trust.

Site Migrations: The Highest-Risk Technical Project

A botched migration can erase years of SEO equity overnight; a well-run one preserves rankings and often improves them. Because it is high-stakes and procedural, it has its own playbook.

> See the supporting guide: SEO Site Migration: The Complete Checklist (Move Without Losing Rankings).

Technical SEO and AI Search

Every AI answer engine begins with retrieval through a crawler. The technical requirements for AI citation are, in order:

1. Crawlable — AI crawlers not blocked; pages reachable via internal links.
2. Indexable & fast — clean canonicals, passing Core Web Vitals (AI retrieval favours responsive pages).
3. Parseable — semantic HTML and direct-answer structure so the answer is extractable.
4. Verifiable — schema + NAP consistency so the system can trust the entity.

This is Unity SEO's AI Trust Architecture expressed at the technical layer: a machine cannot cite what it cannot reach, parse, or trust.

1. 1Crawlable
3. 2Indexable and fast
5. 3Parseable
7. 4Verifiable

Technical SEO Audit Checklist

1. Confirm robots.txt allows important paths and AI crawlers
2. Validate XML sitemap (indexable canonical URLs only, accurate lastmod)
3. Check internal linking — no orphan pages; logical hierarchy
4. Review canonical tags and resolve duplicate content
5. Eliminate soft 404s; return correct status codes
6. Pass Core Web Vitals (INP ≤200ms, LCP ≤2.5s, CLS ≤0.1) on field data
7. Implement Organization/LocalBusiness + Article/Author + FAQPage + BreadcrumbList schema
8. Verify HTTPS, mobile-first rendering, and any hreflang
9. Confirm NAP consistency everywhere
10. Monitor GSC Page Indexing + Core Web Vitals reports monthly

Request a technical SEO audit from Unity SEO — we assess crawlability, indexing, Core Web Vitals and structured data, then prioritise fixes by impact.

Sources: Google crawl budget documentation; Search Engine Hub, Core Web Vitals 2026; Search Engine Journal, Google on llms.txt; Unity SEO Schema Markup, Core Web Vitals and AEO guides.