Google’s Martin Splitt, a Googlebot expert on Google’s Search Relations team, explained how site owners and SEOs can tackle malicious bots and improve their site’s performance.
Malicious bot recap
Malicious bots are self-propagating automated malware programs that infect their host systems, steal data, or carry out other fraudulent activities. They then send the data back to their central server, which acts as a command-and-control center.
Malicious bots are a problem for all websites
Malicious bots attack every website, and besides the security risks, which can include unauthorized access to sensitive data and disrupting your operations, the biggest problem for site owners is when they attack your website with spam and advertising.
Sites that receive excessive spam and advertising can experience excessive crawling, which can cause a 500 server error response code (an inability to serve web pages), impeding Google’s ability to crawl your site.
How to fight the bots
In the August Google SEO Office Hours podcast, Martin Split answered a question from a site owner whose site was experiencing disruptions due to targeted scraping by malicious bots.
The site owner wanted to know how to stop the scraper bots from negatively impacting their server’s performance.
Anon, the site owner, asked the following question:
- “Our website is experiencing significant disruptions due to targeted scraping by automated software, leading to performance issues, increased server load, and potential data security concerns. Despite IP blocking and other preventive measures, the problem persists. What can we do?”
Splitt’s answer included identifying the bot owner and sending an abuse notification. Splitt also recommends using a CDN (Content Delivery Network) because of its firewall capabilities.
Martin Splitt’s answer:
- “This sounds a lot like a distributed denial-of-service issue if the crawling is so aggressive that it causes performance degradation.
You can try identifying the owner of the network where the traffic is coming from, like “their hoster,” for instance, and send an abuse notification. You can use WHOIS information for that, usually. If that doesn’t work, then maybe a traceroute helps.
Alternatively, you might want to use a CDN. CDNs often have features to detect this type of bot traffic and block it. By definition, they usually take the traffic away from your server, and also they distribute it nicely, so it’s a win-win.
Most CDNs do recognize legitimate search engine bots and won’t block them, but if that’s a major concern for you, consider asking their customer support before you choose a specific CDN to go with.”
Use a CDN to block malicious bot traffic
Splitt advises using a CDN (content delivery network) to block bots because they have a WAF (Web Application Firewall) that automatically blocks malicious bots.
The other advantage of using a CDN is it speeds up your site’s performance and reduces server resources by sending crawlers and browsers the requested web page from servers closest to their location.
Finally, keep your system up to date and use Web Application Penetration Testing to identify exploitable application-layer weaknesses that patching and bot filtering alone may not uncover.
You can listen to the conversation on the Google SEO Office Hours podcast:
