How long should indexing take?

Anywhere from hours to several weeks. Established sites that publish regularly get crawled faster; a new site with few inbound links can wait a long time. Requesting indexing in Search Console can help but doesn't guarantee anything.

Will blocking a page in robots.txt remove it from Google?

No, and it can make removal impossible. The page may stay listed from inbound links, and blocking the crawl prevents Google from ever seeing a noindex tag. Allow crawling and use noindex instead.

Do meta keywords help with indexing?

No. Google stopped using them in 2009 and every other major engine followed. They can be removed entirely with no effect.

Does submitting a sitemap guarantee my pages get indexed?

No. It helps with discovery, which is a prerequisite, but indexing is a separate judgement about whether the content is worth storing.

Why did my page get indexed and then drop out?

Usually a re-evaluation — the page was indexed provisionally and later judged duplicate or thin, or a stronger page took its place. Sudden sitewide drops point instead at a technical change: a bad robots.txt deploy, a canonical error, or an accidental noindex.

What actually decides whether your pages get indexed

Web9 min readPublished 17 September 2026

A page that isn't indexed cannot rank, however good it is. Before any question about rankings is worth asking, the more basic one needs answering: does the search engine know this page exists, is it allowed to fetch it, and did it decide the page was worth storing?

Those are three separate gates — discovery, crawling and indexing — and most 'my page isn't ranking' problems are actually a failure at one of them. They also have different fixes, and applying the wrong one is how sites end up worse off.

Discovery, crawling and indexing are not the same

Discovery is the search engine learning a URL exists, usually from a link, a sitemap or a redirect. Crawling is it fetching that URL. Indexing is it deciding the content is worth storing and serving.

A page can be discovered but not crawled (blocked in robots.txt), crawled but not indexed (judged thin, duplicate or low value), or indexed but not ranking (a genuinely different problem). Search Console reports on each stage separately, and reading which stage failed is the whole diagnosis.

Search Console says	Stage that failed	Usually means
Discovered – currently not indexed	Crawling	Known but not fetched yet; often a crawl-budget or quality signal
Crawled – currently not indexed	Indexing	Fetched and judged not worth storing
Blocked by robots.txt	Crawling	Your own robots file is preventing the fetch
Duplicate, Google chose different canonical	Indexing	Another URL was picked as the primary version
Excluded by noindex tag	Indexing	Working as instructed — intentional or not
Soft 404	Indexing	Page returns 200 but looks empty or like an error

The robots.txt mistake that keeps pages in Google

robots.txt controls crawling. It does not control indexing, and treating it as a removal tool produces exactly the opposite of what people intend.

A page blocked in robots.txt can still appear in search results, listed from inbound links alone, with no description because the crawler was never allowed to read it. Worse: if you add both a Disallow in robots.txt and a noindex tag to the page, the page stays indexed permanently. The crawler can't fetch the page, so it never sees the noindex telling it to leave.

The correct approach is the opposite of the instinct. To remove a page from search results, allow crawling and serve a noindex meta tag or X-Robots-Tag header. Use robots.txt only to stop crawlers wasting effort on URL spaces you don't care about at all — faceted filter combinations, session-ID URLs, internal search results.

What a sitemap is and isn't for

A sitemap helps discovery. It is not a ranking signal, it does not guarantee indexing, and for a small well-linked site it changes very little — crawlers follow links perfectly well.

It earns its place on large sites, new sites with no inbound links, and sites where pages aren't reachable through internal navigation. Even then, the most valuable thing it provides isn't the crawling: it's the Search Console coverage report, which tells you which submitted URLs were indexed and why the rest weren't. That feedback is hard to obtain any other way.

Include only canonical URLs that return 200 and are indexable. A sitemap full of redirects and 404s reduces trust in the whole file.
Google ignores the changefreq and priority fields entirely. lastmod is used, but only from sites that set it honestly — stamping everything with today's date gets it discounted.
Maximum 50,000 URLs and 50 MB uncompressed per file; split larger sets across files listed in a sitemap index.
Reference it from robots.txt and submit it in Search Console — do both.

Why a crawled page gets rejected

'Crawled – currently not indexed' is the frustrating one, because nothing is technically broken. The search engine fetched the page, evaluated it, and decided against storing it. There is no error to fix, only a judgement to change.

In practice that judgement usually comes down to one of a few things: the page duplicates another page on the site or elsewhere, it's very thin relative to what it's competing against, it's one of hundreds of near-identical templated pages, or the site as a whole hasn't established enough credibility for its marginal pages to be worth storing.

Templated pages that differ only in a variable — a city name, a product number — are the most common casualty. If the only unique thing on the page is the title, that's the problem.
Thin pages need substance, not more keywords. The question a reviewer or an algorithm is answering is whether the page is worth returning to a searcher instead of the existing results.
Internal linking matters more than people expect. A page linked only from a sitemap looks unimportant; a page linked from relevant articles looks like part of the site.
Fixing one page rarely works in isolation. Sitewide quality affects how generously marginal pages are treated.

A checklist before blaming the algorithm

Search for your exact URL in Google. If it appears, it's indexed and the problem is ranking, not indexing.
Run the URL through Search Console's inspection tool and read which stage it stopped at.
Check the page returns HTTP 200, not a redirect or a soft 404.
View the rendered HTML, not the source, and confirm the content is actually present — client-side-rendered content that fails to render is invisible.
Check for a stray noindex tag or X-Robots-Tag header, especially after a deployment or a platform migration.
Check robots.txt for a Disallow left over from staging. This is the single most expensive typo in the field.
Confirm the canonical tag points at the page itself, not at another URL.
Make sure the page is linked from somewhere on your site that is itself indexed.

Frequently asked questions

How long should indexing take?: Anywhere from hours to several weeks. Established sites that publish regularly get crawled faster; a new site with few inbound links can wait a long time. Requesting indexing in Search Console can help but doesn't guarantee anything.
Will blocking a page in robots.txt remove it from Google?: No, and it can make removal impossible. The page may stay listed from inbound links, and blocking the crawl prevents Google from ever seeing a noindex tag. Allow crawling and use noindex instead.
Do meta keywords help with indexing?: No. Google stopped using them in 2009 and every other major engine followed. They can be removed entirely with no effect.
Does submitting a sitemap guarantee my pages get indexed?: No. It helps with discovery, which is a prerequisite, but indexing is a separate judgement about whether the content is worth storing.
Why did my page get indexed and then drop out?: Usually a re-evaluation — the page was indexed provisionally and later judged duplicate or thin, or a stronger page took its place. Sudden sitewide drops point instead at a technical change: a bad robots.txt deploy, a canonical error, or an accidental noindex.