Allow Google to crawl CSS and JS

A small mistake to avoid when dealing with the googlebot

A couple of years ago a SEO spoke to me and mentioned, that you should prevent the googlebot from crawling css and js. You know for "search engine optimisation"-reasons. Sadly I followed that recommendation.

So recently I noticed, that this is a mistake, because if you're doing this:

User-agent: *
Disallow: /assets/css/
Disallow: /assets/fonts/
Disallow: /assets/js/

Now, the googlebot will no longer crawl your files in these directories. If you're working with the search-console, you'll likely get message about links being to close to each other and to small to use on mobile devices.

So make sure, to redact these entries from you robots.txt so the googlebot is able to crawl those directories. Then that problem is gone and Google will hopefully notice, that your site is optimised for mobile devices.