What Is Technical SEO? A Checklist for Crawling, Indexing and Site Infrastructure

What is technical SEO, and what does it cover?

Technical SEO isn’t concerned with the content itself but with how that content reaches search engines: it covers the infrastructure behind crawling, rendering, indexing and serving. However strong your content and links are, a page that can’t be crawled or indexed gets nowhere in search results.

We approach technical SEO through four questions and verify each answer with a report in Search Console. Tasks often labelled “advanced SEO techniques”, such as weeding out broken and spammy links or detecting duplicate content, also fall under one of these questions.

We explain how technical SEO differs from on-page and off-page SEO in our guide to what SEO is; this article focuses on infrastructure alone.

Can Google crawl your site? robots.txt, server responses and crawl budget

For Google to crawl a site, important URLs must not be blocked by robots.txt, the server must respond to those addresses reliably with the correct status code, and pages must be reachable through internal links that can be followed. Crawl budget, by contrast, is only a separate optimisation topic for very large sites or sites whose content changes very quickly.

robots.txt

robots.txt is a file that tells crawlers which URLs they can access, and its main purpose is to stop the server being overloaded. As Google states explicitly, it is not a way to keep a page out of Google; for that, you use noindex or password protection.

The mistakes we see most often in audits are a Disallow: / line left over from the test environment that shuts off the whole site, blocking the CSS and JavaScript files needed to render the page correctly, and blocking important category pages along with the filter URLs that were meant to be shut off.

robots.txt now covers AI crawlers too. According to Google, Google-Extended controls whether content is used in Gemini models and does not affect inclusion or ranking in Google Search; at OpenAI, GPTBot for training and OAI-SearchBot for ChatGPT search are managed separately. Bot blocking at CDN level can also stop these bots regardless of robots.txt. Since 2026, Search Console has offered a separate setting that lets a site stay in Google Search while opting out of AI features; Google also states that files such as llms.txt are not required for Google Search. More detail in our article on AI crawlers.

Server responses and file size

When a site slows down, or returns 5xx server errors or rate-limiting responses such as 429, Google lowers its crawl capacity and crawls less. Persistent server errors can also cause pages to drop out of the index, which is why we monitor server logs and the Crawl Stats report together.

For Google Search, Googlebot crawls the first 2 MB of a supported file and the first 64 MB of a PDF file; each CSS and JavaScript file the page uses is subject to the same limit separately. In bloated HTML, important content can end up beyond this limit. Server response time (TTFB) is not a Core Web Vitals metric, but it does affect LCP; web.dev recommends 0.8 seconds or less as the target for most sites.

Crawl budget

Crawl budget is the set of URLs Google can and wants to crawl on a site; it combines the crawl capacity limit, set so as not to overload the server, with the site’s crawl demand. According to Google, although there is no exact threshold, this mainly concerns sites with more than one million unique pages whose content changes weekly, sites with more than ten thousand pages whose content changes every day, and sites with a large number of URLs sitting in the “Discovered - currently not indexed” status.

This is what the familiar claim that “popular sites get indexed faster” really means: popularity is one of the factors that increase crawl demand, and URLs that attract more interest are crawled more often. What wastes the budget is duplicate URLs, soft 404s, long redirect chains and parameters that generate endless filter combinations.

Are your pages being indexed? noindex, canonical tags and duplicate versions

For a page to be indexed, it must be crawlable, must not carry noindex, and Google must choose it as the canonical version among its possible duplicates. The Page indexing report in Search Console, formerly the Coverage report, shows which URLs are not indexed and why.

Google says you shouldn’t expect every URL on a site to be indexed. The goal is for important, canonical pages to be in the index, while duplicates and pages deliberately left out stay outside it. That is the lens through which we read the statuses in the report.

For a noindex rule to work, the page must not be blocked by robots.txt; Google can never see the tag on a page it can’t crawl. A canonical tag, meanwhile, is a strong signal but not a definitive command: Google treats redirects and the rel=canonical tag as strong signals and inclusion in a sitemap as a weak one, and if the signals conflict, it makes its own choice. The mistakes Google asks you to avoid are using robots.txt for canonicalisation, using noindex to choose a canonical within the same site, and specifying different canonicals for the same page through different methods.

The most common sources of duplicate versions are http and https, www and non-www addresses, trailing slashes, and parameters such as UTM, sorting and filters; we consolidate these into a single preferred URL. At this stage we also detect near-identical or copied content and merge weak pages that cover the same topic.

How should you set up an XML sitemap and site architecture?

An XML sitemap should contain only URLs you want indexed, that return a 200 status code and are canonical, listed with their full addresses; site architecture should rest on a logical category and URL structure in which important pages can be reached from the homepage in a small number of internal links. A sitemap makes discovery easier, while internal links shape both discovery and how important each page is within the site.

• A single sitemap file can contain at most 50,000 URLs or 50 MB uncompressed; larger sites group several files in a sitemap index file.
• Google uses the lastmod value when it is consistently and verifiably accurate, and ignores the priority and changefreq values. lastmod should be updated for meaningful changes to the content, not for minor edits such as updating the copyright year.
• URLs that redirect, carry noindex, return an error or are not canonical do not belong in the sitemap.
• The sitemap is submitted through the Sitemaps report in Search Console or declared with a line added to robots.txt. According to Google, submitting a sitemap is only a hint; it doesn’t guarantee that the sitemap will be downloaded or used.

For architecture we look at two measures: orphan pages that receive no links from any other page, and how far important pages are from the homepage. The category hierarchy, readable URLs that separate words with hyphens and breadcrumb navigation are all parts of this structure.

Architecture is cheapest to get right at the design stage; changing the URL structure of a live site requires a migration project. That is why, on new site projects, we put technical SEO at the top of our web design SEO checklist, and in our web design projects we plan the URL structure and internal linking alongside the design.

How should you handle redirects, 404 pages and broken links?

Every permanently moved URL should be redirected server-side, in a single step, with a 301 or 308 to its closest equivalent; a page with no equivalent should return a real 404 or 410, and internal links should be updated to point straight to the final URL. Chains, loops and pages that say “page not found” but return 200 wear out users and crawlers alike.

The problem we find most often in audits is redirects that have piled up on top of each other over the years: chains running from http to https, from there to www, and from there to a new URL structure. We collapse these into a single step. What we don’t do is redirect old pages with no equivalent to the homepage in bulk; users wouldn’t find what they were looking for.

A 404 is not an error in itself but the correct response for removed content; what matters is that a custom 404 page that helps users find their way still returns a 404 code. If a removed URL had external links pointing to it, redirecting it to a page that covers the same topic preserves the value of those links.

We regularly pull out broken internal links with a crawling tool. Spam links left in comment and forum sections call for moderation and the use of rel=ugc or nofollow, while hidden links injected into hacked pages call for a security audit. We walk through the redirect plan for domain changes and site migrations step by step in our guide to site migration and 301 redirects.

How do HTTPS and site security affect technical SEO?

HTTPS is one of the basic questions Google asks when it comes to good page experience, and it protects visitors’ data; on the security side, hacked content or malware can lead to the site being shown with a warning in search results and to a loss of visibility. Neither is a set-and-forget job.

• All http URLs should redirect with a 301 to their https equivalents in a single step; canonical tags, internal links and the sitemap should point only to https URLs.
• No images, scripts or stylesheets on the page should still load over http (mixed content).
• Certificate renewal should be monitored automatically; an expired certificate greets visitors with a browser warning.
• The server should support HTTP/2 or HTTP/3. Google’s crawlers crawl over HTTP/1.1 and HTTP/2; according to Google, crawling over HTTP/2 brings no ranking benefit and only saves resources. The benefit of HTTP/3 lies in connection performance on the visitor’s side.

The Security issues report in Search Console shows hacked content, malware and social engineering issues that Google has detected; outdated plugins and themes are the most common entry point we see.

Third-party scripts are a security and quality issue too. In April 2026, Google defined back button hijacking, the practice of stopping users from returning to the previous page with the browser’s back button, as a clear violation of its malicious practices spam policy. That is why we also check whether ad and recommendation scripts interfere with navigation. On e-commerce sites that take payments, HTTPS is a precondition for customer trust before it is ever a ranking question.

Why are Core Web Vitals and mobile-first indexing part of technical SEO?

Core Web Vitals are measures of real-user experience that Google uses in its ranking systems; with mobile-first indexing, Google crawls and indexes the mobile version of a site. Both concern how content is delivered, and the problems are mostly solved at template, server and code level.

For a page to meet the thresholds, the 75th percentile value in real-user data must stay within the threshold, separately for mobile and desktop. INP replaced FID on March 12, 2024. Google also stresses that it aims to show the most relevant content even when page experience is poor; speed is no substitute for relevance. We explain how to measure and improve these metrics in our Core Web Vitals guide.

Google announced in October 2023 that mobile-first indexing was complete; the last sites still crawled with the desktop crawler were moved to the mobile crawler after July 5, 2024. Because only the content in the mobile version is used for indexing, text, links and structured data missing from the mobile HTML are not taken into account. Responsive design, which adapts to the screen on the same URL, is the lowest-risk setup; AMP, meanwhile, has not been required even for Top Stories since 2021. The mobile side is covered in detail in our article on mobile SEO.

How does Google handle JavaScript-generated content?

Google renders the page by running its JavaScript and indexes the rendered HTML; however, this happens in a separate stage after crawling, and content that waits for user interaction may not be seen. The safest approach is to have important text and links in the HTML sent by the server, or in the initial render.

Rendering is done with a Chromium-based browser on pages that return a 200 status code. Google still considers server-side rendering (SSR) or pre-rendering a good idea, because it makes the site faster for both users and crawlers; dynamic rendering, on the other hand, is not a recommended solution but a workaround. Moving a client-side application to server-side rendering is a job for software development, not a theme setting.

• Links should be built with <a> elements that carry an href attribute; Google cannot follow menu items that work only through a JavaScript event as links.
• Content that loads only after a user interaction (such as products that appear later in an endlessly loading list, or tabs that load when opened) may never reach Googlebot.
• In single-page applications, content that can’t be found should trigger a JavaScript redirect to a URL that returns a 404, or the error page should be given a noindex; otherwise a soft 404 occurs.
• JavaScript and CSS files needed to render the page should not be blocked by robots.txt.
• Don’t assume that every crawler other than Google’s, AI bots included, runs JavaScript.

We verify this with the live test in the URL Inspection tool: if the main text, headings, internal links and structured data appear in the rendered HTML, the page is readable as far as Google is concerned.

Why do structured data and hreflang tags matter for technical SEO?

Structured data labels the information on a page in a machine-readable format and determines eligibility for rich results; hreflang links the language and regional versions of the same content so that the right version is shown to the right user. Both can stop working without any warning when they are set up incorrectly, so they need regular validation. Seeing the missing fields in a page’s JSON-LD with our schema checker tool can be the first step in that validation.

Structured data

Google recommends the JSON-LD format and requires markup to match the content visible on the page; marking up content that readers can’t see can prevent rich results even when the syntax is correct. Correctly implemented markup doesn’t guarantee a rich result either.

The rich results landscape is changing as well: HowTo results were removed in 2023, and FAQ rich results, restricted that same year to government and health sites, have not been shown in Google Search since May 7, 2026. The simplification that began in June 2025 also removed other little-used features from search results. We implement schema for clarity rather than as a promise of how a page will appear, and monitor errors with the Rich Results Test and the Schema Markup Validator; more detail in our structured data guide.

hreflang

hreflang can be declared in three ways: a link element in the HTML, an HTTP header or the XML sitemap. Each version must point both to itself and to every other version; if two pages don’t point to each other, Google ignores the tags. Language codes are written in ISO 639-1 format and region codes in ISO 3166-1 Alpha 2 format (tr, tr-TR, en-GB); x-default can be used for users who don’t match any of the languages.

The mistakes we see most often are tags without return links, pointing the canonical of one language version to the page in another language, and invalid codes such as en-UK. Google determines a page’s language with its own algorithms rather than from the lang attribute, so each version must genuinely be written in that language.

In what order should you run a technical SEO audit?

We run technical SEO audits starting with the problems that could cut off visibility completely: first crawling and indexing, then architecture and redirects, then security, speed, mobile and JavaScript, and finally structured data and hreflang. No fix is signed off until it has been verified with the relevant report in Search Console.

1. Access: robots.txt, server responses and Crawl Stats; confirming that the site, or a section of it, hasn’t been shut off by mistake.
2. Indexing: The Page indexing report, noindex and canonical checks, duplicate URLs.
3. Architecture and redirects: Sitemap, orphan pages, redirect chains, broken links, 404s and soft 404s.
4. Security: HTTPS, mixed content, and the Security issues and Manual actions reports.
5. Experience: Core Web Vitals, mobile content parity and JavaScript rendering.
6. Clarity: Structured data, hreflang and the internal linking structure.

The picture becomes clear when a crawling tool, Search Console reports and server logs are read together. We rank findings by weighing impact against effort: a one-line robots.txt fix comes before a lengthy template change.

Because it takes Google time to recrawl and process a change, we repeat the verification as the reports update. We explain the steps of a comprehensive audit in our guide to how to do an SEO analysis, and how to use Search Console reports in our Google Search Console guide. Our free technical SEO analysis is a good starting point for seeing where your site stands on these checks; we bring crawling, indexing, structured data and speed issues together in a single report.