How 70,000 filter URLs get into an index, and how to get them out
Faceted navigation multiplies. Colour times capacity times finish times price band, and suddenly Google has found sixty thousand pages you never meant to publish. There are four ways to contain it, and three of them are wrong for your site.
How the URLs get created
Every filter on a product listing page is a dimension. Six colours, four capacities, three finishes and five price bands is not eighteen combinations, it is 360. Add sort order and pagination and you are into the thousands from a single category.
Most platforms make each combination a real URL with a real 200 response and a real template. If any of those URLs are linked from anywhere Google can reach, they get crawled. If they get crawled and look distinct enough, they get indexed.
Run a site: query against your domain and compare the number to your actual product count. If the index is more than about twice your catalogue, you have this problem. If it is more than ten times, it is your biggest SEO problem and everything else can wait.
Why it costs you rankings
Three ways, in order of how much they hurt.
- Crawl budget. Googlebot has a finite appetite for your site. Every request spent on a filter combination is a request not spent on your new products.
- Diluted signals. Internal links and any external links get spread across near-identical pages rather than concentrating on the category you actually want ranking.
- Quality perception. Tens of thousands of thin, near-duplicate pages is a signal in itself, and not a flattering one.
FIGURE 1
Search Console coverage report: indexed versus discovered, before and after containment
Replace with a real screenshot, 1600 x 900
The four containment options
| Option | Crawled | Indexed | Passes equity | Best for |
|---|---|---|---|---|
| robots.txt disallow | No | Sometimes, badly | No | Parameters with zero search demand, ever |
| noindex, follow | Yes | No | Yes | Facets with some internal linking value |
| Canonical to parent | Yes | Usually not | Yes | Sort and pagination variants of one page |
| Never generate the link | No | No | N/A | Combinations nobody searches for |
Why robots.txt is usually the wrong first answer
It is the one people reach for because it is fast. The problem is that a disallowed URL can still be indexed if something links to it, and because Google cannot crawl it, it cannot see your noindex tag either. You end up with URLs in the index that you have made it impossible to remove cleanly.
Why noindex is usually right
It keeps the page crawlable, so link equity still flows through it, and it removes the page from the index reliably. The cost is crawl budget, since Google keeps visiting. On a large site you eventually pair it with link-level containment.
How to choose
- Export every indexed filter URL from Search Console with its impressions and clicks.
- Anything with real impression volume stays crawlable and indexable. Those are pages, not accidents, and some of them are earning.
- Anything with impressions but no clicks gets noindex, follow.
- Anything with neither, and no internal links pointing at it, stops being generated as a link at all.
- Sort order and pagination variants get canonicalised to the base page, never noindexed.
Evidence decides, not opinion. A filter combination that earns impressions is a landing page whether you designed it to be one or not. Killing it because it looks untidy is how people lose traffic during cleanups.
Cleaning up what is already indexed
Containment stops new URLs. It does not remove the ones already there, and that part is slower than most people expect.
- Return 410 Gone rather than 404 for combinations you are permanently killing. Google drops 410s noticeably faster.
- Keep the noindex pages crawlable until they actually leave the index. Blocking them in robots.txt at this stage freezes them there.
- Expect months, not weeks, on a large index. Track the coverage report weekly and resist the urge to add another fix on top.
- Do not submit a removal request as the fix. It is a 6 month hide, not a deletion.
I did exactly this across 70,637 indexed URLs before a replatform, and the write-up is here: auditing 70,637 indexed URLs before a migration.
Index bigger than your catalogue?
Send me the site and I will tell you on a twenty minute call whether index bloat is your real problem or a symptom of something else.
Typical reply time: within one working day