Crawl Error on Cloudflare cdn-cgi Files
If you are getting a crawl error in Google Webmaster Tools for a cdn-cgi file, please disallow Google from crawling that with a rule in robots.txt for:
Disallow: /cdn-cgi/
An additional option using WordPress:
Solution provided by Ben
You can make modifications in your /wp-includes/functions.php file. to create an additional Disallow rule. A sample of what it would look like is below:
function do_robots() {
header( ‘Content-Type: text/plain; charset=utf-8’ );
do_action( ‘do_robotstxt’ );
$output = “User-agent: *\n”;
$public = get_option( ‘blog_public’ );
if ( ‘0’ == $public ) {
$output .= “Disallow: /\n”;
} else {
$site_url = parse_url( site_url() );
$path = ( !empty( $site_url[‘path’] ) ) ? $site_url[‘path’] : ”;
$output .= “Disallow: $path/wp-admin/\n”;
$output .= “Disallow: $path/wp-includes/\n”;
$output .= “Disallow: $path/cdn-cgi/\n”;
}
Note: This won’t impact the actual crawl for your site at all. The error for /cdn-cgi/ is also not going to impact your rankings at all. Google is attempting to crawl a Cloudflare resource. One that doesn’t need to be crawl, and they possibly can’t read the DOM or JavaScript in that resource.