Google Analyst Gary Illyes says centralizing robots.txt files at the root domain isn`t rigid and shares some alternative methods that comply with Google standards but buck conventional wisdom.  

Robot.txt recap

Robot.txt is a set of commands you can set on a server or add to your website to give Google certain instructions, such as: 

• Webpage: To direct Google to not access and index certain URLs on your site. 
• Media: To stop audio, video, and image files from showing in Google Search results.
• Resource file: To block certain resource files, such as style sheets, images, or irrelevant scripts.

The primary purpose of using robots.txt files is to avoid overloading your website with requests.

Robots.txt files are flexible

Google Analyst Gary Illyes’s recent LinkedIn posts challenge the long-standing belief that robots.txt files must be in the root domain (e.g., conventionalwisdom.com/robot.txt.) 

Illyes`s alternative revelation that the root domain placement isn`t a requirement contradicts conventional wisdom.

According to Gary, having two domains hosting two separate robots.txt files, one for your website and the other for your content delivery network (CDN) is allowable.

Centralizing on CDNs is a good idea

Gary explained that site owners could have just one robots.txt file containing all the rules on a CDN, allowing you to control crawling from your main domain. 

Illiyes gave this example:

• “Sites can have two robots.txt files: one at https://cdn.example.com/robots.txt and another at https://www.example.com/robots.txt.”

Gary`s alternative approach would enable site owners to track rules they need to manage from a central robots.txt. 

• “(example.com/robots.txt).”

Gary`s post on LinkedIn:

Key points:

• “The robots.txt file doesn’t have to be located at the root domain.”
• “Robots.txt files can be centralized on CDNs, not just root domains.”
• “Websites can redirect robots.txt from the main domain to CDN.”

Gary followed this up later on LinkedIn, saying as long as robots.txt files aren`t in the middle, they`ll work just fine:

How a central rotors.txt file could help you

Applying Gary`s centralized management idea of consolidating your robots.txt file rules in a single location could help you steamline how you manage robots.txt files, improving your site SEO and management endevours. 

His single source approach for robots.txt rules could also reduce the risk of conflicting directives between your CDN and main domain. Check out the Google Search Central page to learn all about the robots.txt file and how to create or update one.

• Author
• Recent Posts

Terry O'Toole

Terry is a seasoned content marketing specialist with over six years of experience writing content that helps small businesses navigate where small businesses meet marketing - SEO, Social Media Marketing, etc. Terry has a proven track record of creating top-performing content in search results.
When he is not writing content, Terry can be found on his boat in Italy or chilling in his villa in Spain.

Latest posts by Terry O'Toole (see all)

• Google Goes Nuclear To Power AI Data Centers - October 15, 2024
• SEO Weekly News Roundup [October 7 to 11, 2024] - October 14, 2024
• Google Local Service Ads Update On The Way - October 11, 2024