Google’s Gary Illyes, an Analyst on the Google Search team, explained why URL parameters can cause crawler inefficiencies by creating infinite URLs for a single web page.
Gary also advises on how to fix the problem and mentions Google is looking for solutions.
URL parameter recap
URL parameters (also known as query strings) are strings of text that can modify a web page’s URL and track visitor numbers and actions.
Parameters help simplify a site’s navigation by providing additional information about a webpage without changing its content. Enabling visitors to find what they are looking for quickly.
Parameters are identified at the end of a URL after a ? symbol, and you can include multiple parameters by separating them with the & symbol.
However, as Gary explains, URL parameters can create infinite URLs and that can be a problem for SEO.
The multiple URL conundrum
In the latest Google Search Off The Record podcast, Gary Illyes, Lizzie Sassman, and John Mueller (Google Search team members) discussed how parameters can create infinite URLs for a single web page and cause crawl inefficiencies.
Mueller asked Gary, “What optimizations do you see happening regarding crawling?”
Gary led with “Maybe better URL handling” saying:
- “Technically, you can add that in one almost infinite–well, de facto infinite–number of parameters to any URL, and the server will just ignore those that don’t alter the response. Basically, it will just discard them.”
Gary also said:
- “For every single URL that’s on the internet, you have an infinite number of versions because you can just add URL parameters to it.”
And explained why this can give search engine crawlers a problem because:
- “The server is instructed to ignore them; it will not alter the content it returns.”
Lizzie asked Gary why URL parameters are a problem for crawling efficiency:
- “It sounds like it’s because we’re maybe wasting time looking at parameter versions of the links when it could be the same thing, but sometimes it is different,” Lizzie asked.
Gary replied:
- “Sometimes it is different, and that’s the problem.”
Duplicate content
A typical SEO issue URL parameters cause is generating duplicate content.
If a site has multiple URLs using different parameter values that lead to the same or similar content, crawlers can view it as duplicate content, negatively affecting its SEO and visibility in search.
As Gary explained, the problem is that crawlers cannot tell if a URL leads to the same or different content without visiting each URL, which can cause indexing issues because of the inefficient use of crawl resources.
Crawl budget waste
Crawl budget waste occurs when URL parameters create multiple variations of a single page, wasting search engine bots’ crawling resources.
Too many URL variations can divert crawl resources away from your site’s most important pages, causing incomplete indexing and reducing your site’s visibility.
Illyes pointed out:
- “Because you can just add URL parameters to it, it also means that when you are crawling, and crawling in the proper sense like ‘following links,’ then everything, everything becomes much more complicated.”
Crawling smarter, not harder
Illyes and the team also discussed potential solutions, including:
- How Google might develop algorithms to handle URL parameters.
- Implementing a system where website owners can notify Google about their URL parameters, Illyes said, “We could just tell them, ‘Okay, use this method to block that URL space.’
When asked what methods, Illyes mentioned robots., saying:
- “Robot.cxd could be used to instruct crawlers not to look at specific URL combinations.”
- With Dro cxd, it’s surprisingly flexible with what you can do with it.
Gary finished by saying Google just has to think of what to say.
