Crawl, Index, Rank: The Three Things Google Needs From Your Website

“Why isn’t my new page showing up on Google?” It’s one of the most common questions we get. Often the answer has nothing to do with competition or keywords. Google simply hasn’t got to the page yet, or decided not to keep it.

Understanding the three stages takes the mystery out of it.

Stage 1: What is crawling?

Crawling is discovery. Google’s crawler, Googlebot, constantly visits web pages, follows the links on them, and finds new or updated pages.

Googlebot finds your pages through:

• Links from other pages on your site and from other websites
• XML sitemaps, a file listing your important URLs
• Manual requests through Google Search Console’s URL Inspection tool

What stops pages being crawled?

No links pointing to them. “Orphan” pages that nothing links to are hard to find.

Blocked by robots.txt. This file tells crawlers which areas to avoid. A misplaced line can block your whole site. It happens more often than you’d think after a redesign.

Server errors or very slow responses. If your server struggles, Google may crawl less. Our article on hosting and speed covers this.

Content only visible after interaction. Links hidden behind JavaScript actions may not be followed reliably.

Stage 2: What is indexing?

Indexing is Google reading the page, understanding what it’s about, and deciding whether to store it. Being crawled doesn’t guarantee being indexed. Google doesn’t index everything it finds.

What stops pages being indexed?

A “noindex” tag. This tells Google not to index the page. Useful for thank-you pages, a disaster if it’s accidentally left on your whole site from development. In WordPress, check Settings → Reading → “Discourage search engines from indexing this site” is unticked.

Duplicate content. If several URLs show near-identical content, Google picks one to index (the “canonical”) and may ignore the rest.

Thin or low-value content. Pages with very little useful content may be crawled but not indexed. In Search Console this often shows as “Crawled – currently not indexed”.

Quality concerns. Google is selective. Pages that add little beyond what already exists may simply not make the cut.

Stage 3: What is ranking?

When someone searches, Google looks through its index and orders results by what it judges most helpful for that query, in that location, on that device.

Google uses many signals, but the big ones are easy to understand:

Signal	In plain English
Relevance	Does the page actually answer this search?
Quality and helpfulness	Is it useful, accurate, and written with real expertise?
Authority and trust	Do other reputable sites link to or mention you?
Location	For local searches, how close and relevant is the business?
Page experience	Is it fast, mobile-friendly and secure?

Notice the order. Speed and technical health are the foundation that lets you compete; relevance and quality decide who wins.

How to check where your pages stand

Google Search Console is free and shows you all three stages:

• The Pages report shows which pages are indexed and why others aren’t.
• The URL Inspection tool shows whether a specific URL is indexed, when it was last crawled, and lets you request indexing.
• The Performance report shows which searches your indexed pages appear for.

If you haven’t set it up, our Search Console beginner’s guide walks you through it.

You can also do a quick check by searching site:yourdomain.co.za on Google. It’s not perfectly precise, but it gives a rough idea of what’s indexed.

A typical scenario

Imagine a physiotherapy practice in Rondebosch launches a new page about sports injury rehab. Three weeks later, it’s nowhere on Google.

In Search Console, the URL Inspection tool says “URL is not on Google: Discovered – currently not indexed”. Google knows about it but hasn’t crawled it yet. The page isn’t linked from the menu or any other page; it’s only in the sitemap.

The fix: link to it from the services page and a couple of related blog posts, then request indexing. That signals the page matters and gives Googlebot clear paths to it.

How to help Google at every stage

• Link important pages from your navigation and from related content.
• Submit an XML sitemap in Search Console (most SEO plugins generate one).
• Check robots.txt and noindex settings after any redesign.
• Keep your site fast and reliable. Check with our foundation checker.
• Publish pages that are genuinely useful and distinct, not thin variations of each other.
• Add structured data so Google understands what your pages represent.

Frequently asked questions

What is the difference between crawling and indexing?

Crawling is Google discovering and fetching a page. Indexing is Google processing it and deciding to store it so it can appear in search results.

How long does it take Google to index a new page?

Anywhere from a few days to a few weeks, depending on your site’s authority, internal linking and how often Google crawls it.

Why is my page crawled but not indexed?

Google may see it as duplicate, thin or not valuable enough compared with existing pages. Improving the content and internal links usually helps.

Can I force Google to index my page?

You can request indexing in Search Console, but Google decides whether to index it.