How Does a Page Get From Your Site to a Search Result?
A useful beginner model is five stages: discovery, crawling, rendering, indexing, and serving. This is a simplified SEORAF model. Google’s documentation describes a crawl, index, and serving pipeline and separately explains crawling, rendering, and indexing for JavaScript pages.
- Discovery: Google learns that a URL exists. A page can be discovered through links from known pages or through a sitemap.
- Crawling: a crawler such as Googlebot requests the URL from your server.
- Rendering: Google processes the page and, where necessary, executes JavaScript to understand the rendered content.
- Indexing: Google processes the content and may add the page to its index.
- Serving: when a user searches, Google’s systems retrieve eligible indexed content and determine which results to show and in what order.
A simplified five-stage learning model. The stages and signals shown here do not guarantee crawling, indexing, rankings, or rich results.
Crawling, Rendering, Indexing, Indexability, Canonicalization, and Serving: What Is the Difference?
Beginners often use these terms interchangeably, but each one answers a different question. Mixing them up can lead to fixing the wrong problem.
Six terms beginners often confuse
| Term | What it means | The question it answers | Typical controls or signals |
|---|---|---|---|
| Crawling | A crawler requests a URL from your server | Can Google fetch this page? | Robots.txt, server responses, internal links, sitemaps |
| Rendering | Google processes the page, including JavaScript where necessary | Can Google process the content it needs? | JavaScript-dependent content, blocked resources, rendered output |
| Indexing | Google processes a page’s content and may add it to its index | Can this page become part of Google’s index? | Content, duplicates, noindex, status codes |
| Indexability | An SEO term for whether a page is eligible to be indexed | Is anything preventing this page from being indexed? | noindex, access restrictions, error responses |
| Canonicalization | The process of selecting a representative URL for duplicate or near-duplicate content | Which URL should represent this content? | Redirects, rel="canonical", sitemap signals |
| Serving | Retrieving eligible indexed content for a search and ordering the results | Which results should appear, and in what order? | Relevance and many ranking signals |
Three points follow from the table. A crawled page is not automatically indexed. An indexed page is not automatically ranked well. And a canonical tag is a signal that Google evaluates, not a command it must follow. The crawling hub and the indexing hub go deeper into these areas.
Which Situation Are You In?
Start with the row that matches your site. These are practical starting points, with Google’s own documentation linked where it applies.
Where to start, by situation
| Your situation | Start here |
|---|---|
| Launching a new site | Decide your page groups and URL pattern first. Check that your platform provides a sitemap, HTTPS, and mobile-ready pages. Submit the sitemap in Search Console, keep staging copies protected, and confirm the live site has no leftover staging block. |
| Existing site, never reviewed | Check key pages with the site: operator and URL Inspection, then look for accidental robots.txt blocks or noindex. Work through the technical SEO checklist. |
| Redesign, migration, or URL change | Follow Google’s site move guidance. Redirect old URLs to their closest relevant new destinations with permanent redirects where appropriate. Redirects are strong signals, but they do not guarantee that rankings or traffic will carry over. See the technical SEO audit guide. |
| Traffic dropped | Follow Google’s guide to debugging traffic drops. Check for accidental noindex, robots.txt blocks, server errors, indexing changes, or major site changes around the date of the drop. To test changes, see SEO experiments. |
Crawlability: Can Search Engines Reach Your Pages?
Crawlability is a search engine’s ability to request a URL from your server. Three important areas are robots.txt, sitemaps, and internal links.
Robots.txt
Robots.txt is a plain-text file that tells crawlers which URLs they may request. Google’s introduction to robots.txt says it is mainly used to manage crawler access and is not a mechanism for keeping a page out of Google.
A blocked URL can still appear in search results, potentially without a normal description. Google also advises against blocking important scripts, styles, or images when doing so would prevent Google from understanding the page.
This example asks crawlers to avoid the /private/ path and points them to a sitemap.
XML sitemaps
An XML sitemap lists URLs that you want search engines to discover. Google’s sitemap guide sets a limit of 50 MB uncompressed or 50,000 URLs per sitemap file. Larger sets can use multiple sitemap files. Submitting a sitemap is a hint, not a guarantee of crawling or indexing.
Internal links
Internal links are an important way Google discovers URLs. Google’s JavaScript SEO basics explains that Googlebot can discover URLs from links in the href attribute of HTML links.
Crawl budget
Google’s crawl budget guide is mainly relevant to very large or frequently changing sites. Most beginners do not need to spend much time optimizing crawl budget.
How to check: open yoursite.com/robots.txt in a browser and review each Disallow rule. Then open the Sitemaps report in Search Console to check whether Google has processed your sitemap and reported errors.
Indexability: Is Each Page Eligible to Be Indexed?
Indexability is a term SEO tools and practitioners use for whether a crawled page is eligible to be stored in a search index. Being crawlable does not make a page indexable, and being indexable does not guarantee that Google will index it.
Noindex
To keep a page out of Google Search, use noindex or access restrictions such as password protection, as Google’s guide to blocking indexing explains.
<meta name="robots" content="noindex">A practical launch habit is to keep staging or test copies protected rather than relying on robots.txt alone, then confirm the live site does not retain an accidental noindex or robots.txt block.
Duplicate Content and Canonicalization
Canonicalization is the process of selecting a representative URL when several URLs contain duplicate or substantially similar content. Google explains that duplicate pages may not all be indexed separately.
You can indicate a preferred URL with a canonical tag. Google’s canonicalization documentation explains that redirects and rel="canonical" are strong signals, while sitemap inclusion is a weaker signal.
<link rel="canonical" href="https://example.com/dresses/green-dresses">Google evaluates these signals and may choose a different canonical URL. Keep the canonical consistent with other signals, and make sure JavaScript does not unintentionally replace the canonical with a different URL.
Not Every URL Needs to Be Indexed
Google does not expect every URL on a site to be indexed. Focus on the pages that should genuinely appear in Search. The indexability guide covers troubleshooting in more detail.
How to check: inspect the page source for noindex, then use the URL Inspection tool. It can show Google’s indexed information and test the live URL. Google’s selected canonical can differ from the canonical you declare.
Status Codes and Redirects: What Is Your Server Telling Crawlers?
Every URL returns an HTTP response, and Google handles different response classes differently. This table follows Google’s HTTP status code documentation.
| Code | General handling |
|---|---|
| 200 | Google can process the content for indexing; indexing is not guaranteed. |
| 301 / 308 | Google follows the redirect and treats it as a strong signal toward the destination. |
| 302 / 307 | Google follows the redirect, but temporary redirects generally send a weaker canonicalization signal. |
| 404 / 410 and other 4xx responses | These generally indicate that the requested resource is unavailable and are not indexed as normal content. Persistent indexed URLs can be removed. |
| 429 and 5xx | These can cause crawling to slow down; persistent server errors can eventually lead to URLs being removed from the index. |
A soft 404 is a page that looks like an error page but returns a successful response such as 200. Google may treat it as an error despite the HTTP status.
Redirect chains should generally be avoided when you can link directly to the final URL. When changing URLs, use the appropriate permanent redirect and keep internal links, canonicals, and sitemaps aligned with the final destination.
Removed Pages
If you permanently delete a page and have no relevant replacement, allowing it to return 404 or 410 can be appropriate. If a closely related replacement exists, redirect the old URL to that relevant destination instead of automatically sending every removed URL to the homepage.
How to check: review the Page indexing report in Search Console for 404, soft 404, redirect, and server-error patterns.
Site Structure and URLs: Does Your Layout Make Sense?
A sensible structure groups related pages under clear hubs and makes important pages easy to reach through internal links. Google does not prescribe one universal site architecture. The site architecture hub explains the approach SEORAF recommends.
Google’s URL structure guidance recommends crawlable, understandable URLs and explains how URL parameters and fragments work. For people, readable URLs with words separated by hyphens are generally easier to understand and share.
How to check: pick your five most important pages and trace the internal links needed to reach each one from the homepage. Look for important pages that are unnecessarily isolated or buried.
Rendering and JavaScript: Can Search Engines See Your Content?
Google processes JavaScript pages through crawling, rendering, and indexing. Google’s JavaScript SEO basics explains that Google Search uses an evergreen Chromium for rendering and that blocked resources can affect what Google can process.
JavaScript itself is not automatically a technical SEO problem. The practical issue is whether important content and links remain accessible and understandable when Google processes the page.
Where practical, keep critical content and important links available in the initial HTML so they do not unnecessarily depend on client-side rendering.
How to check: compare the browser’s “view page source” with the rendered page. The URL Inspection tool can also help you understand how Google processes a URL and its resources.
Page Experience: Is Your Site Fast, Mobile-Ready, and Secure?
Google says there is no single page-experience signal. Its page experience guidance says Core Web Vitals are used by its ranking systems, while other page-experience aspects can improve usability without being individual ranking signals.
Good Core Web Vitals scores do not guarantee top rankings. Google also says that chasing a perfect score purely for SEO may not be the best use of time.
Speed: Core Web Vitals
Core Web Vitals currently include three metrics. The good thresholds are evaluated at the 75th percentile:
- Largest Contentful Paint (LCP): 2.5 seconds or less
- Interaction to Next Paint (INP): 200 milliseconds or less
- Cumulative Layout Shift (CLS): 0.1 or less
Lab tests and real-user data can differ, so use both appropriately when diagnosing performance.
Mobile
Google uses the mobile version of a site’s content for indexing and ranking under mobile-first indexing. Keeping the important content and structured information consistent between mobile and desktop helps avoid mismatches.
HTTPS
Google’s page experience guidance includes secure page delivery as part of its overall page-experience assessment. Search Console also provides an HTTPS report.
Check HTTPS across the site, including subdomains, certificates, images, scripts, internal links, canonicals, and sitemaps.
How to check: run representative pages through PageSpeed Insights and review the Core Web Vitals report in Search Console.
Structured Data: Have You Labeled Your Content?
Structured data is machine-readable markup that helps describe the content of a page. Google’s general structured data guidelines recommend JSON-LD as a format and require structured data to represent visible page content and remain accessible to Googlebot.
Structured data can make pages eligible for certain supported search features, but Google does not guarantee that a rich result or other enhanced appearance will be shown.
Your CMS or SEO plugin may already generate common structured data. Avoid adding duplicate or conflicting markup without first checking what your site already outputs.
How to check: test eligible markup with Google’s Rich Results Test and review Search Console reports where available.
Monitoring: How Will You Know Something Broke?
Use Google Search Console as your first monitoring tool for Google Search. Google’s SEO Starter Guide suggests using the site: search operator as a quick indication that pages from your site are present in Google’s index. For individual URLs, URL Inspection provides more specific information.
For smaller sites, you may not need to spend much time on advanced crawl-budget analysis. For crawler-based tools and workflows, see the SEO crawler guide.
What Should You Fix First, and What Can Wait?
Fix problems that prevent important pages from being accessed or indexed before polishing secondary improvements. This is a practical SEORAF prioritization, not a Google ranking formula.
- Fix first: important pages accidentally blocked from crawling or indexing, persistent server errors, HTTPS problems, and broken redirects after a site move.
- Fix next: duplicate URL problems, unnecessary redirect chains, inappropriate redirects for removed pages, and missing or inconsistent sitemap information.
- Improve over time: Core Web Vitals, structured data, and internal-link cleanup. Google notes that chasing perfect Core Web Vitals scores purely for SEO may not be the best use of time.
Does Technical SEO Help With AI Overviews and AI Search?
Technical SEO helps keep pages accessible to Google Search, but Google says there is no special technical optimization required specifically for AI Overviews or AI Mode. Google’s AI features documentation says pages need to be indexed and eligible to appear with a snippet to be used as supporting links in these Search features, and there are no additional technical requirements.
Google’s guide to generative AI features also says standard Search best practices remain relevant. Google clarified in June 2026 that llms.txt is not needed for Google Search and does not affect visibility or rankings in Google Search.
Other AI systems can have their own crawlers, controls, and documentation. Treat those systems separately from Google Search.
Outdated Technical SEO Advice You Will Still See
Some common advice has been retired or is no longer needed for Google Search. Check the publication date and current documentation before following older technical SEO instructions.
Outdated advice and current guidance
| Outdated advice | What changed | Current guidance |
|---|---|---|
| Optimizing for First Input Delay (FID) | INP replaced FID as a Core Web Vital in March 2024. | Measure responsiveness with INP. A good score is 200 milliseconds or less. |
| Relying on the Mobile Usability report or Mobile-Friendly Test | Google retired these older tools and reports. | Use current Google documentation and tools such as Lighthouse when evaluating mobile experience. |
| Adding llms.txt to improve Google Search visibility | Google’s current AI guidance says Google Search does not use llms.txt for visibility or ranking. | Focus on crawlable, indexable, useful content and standard Search best practices. |
Do I Need a Developer?
Often not for the basics. Many CMS platforms expose common SEO settings through their own interfaces. Google notes that some platforms manage areas such as robots.txt or sitemap generation for you.
Developer support becomes more useful when the issue involves server responses, complex JavaScript rendering, large migrations, custom robots rules, application routing, or other platform-level behavior.
Key Takeaways
- Crawling is about fetching URLs, rendering is about processing the page, and indexing is about processing content for possible inclusion in Google’s index.
- Robots.txt manages crawler access. It is not the primary mechanism for keeping a page out of Google Search;
noindexor access restrictions serve that purpose. - Redirects and canonical tags are signals Google evaluates when selecting representative URLs. Google can choose a different canonical.
- Core Web Vitals are used by Google’s ranking systems, but good scores do not guarantee top rankings.
- Search Console is a useful first place to investigate crawling, indexing, performance, and Search visibility issues.
Frequently Asked Questions
Keep Learning
- Technical SEO Guide: the broader technical SEO framework
- Technical SEO audit guide: how to review a site systematically
- Technical SEO checklist: a step-by-step technical checklist
- SEO crawler guide: how crawler-based tools work
- SEO experiments: how to test and measure SEO changes
- Google’s hreflang guide: for sites serving localized versions
Written by Mousume Akter for SEORAF, an independent SEO knowledge resource. Last fact-checked October 1, 2026 against Google Search Central, Search Console Help, and web.dev. Google updates its documentation regularly, so check the linked source before relying on changing thresholds, limits, or product behavior. Nothing on this page guarantees rankings, indexing, or rich results.