» URLs Restricted by Robots.txt in Google Webmaster Tools

The other night I popped into the Wordpress Support Forum to see if I could help solve anyone’s issues. One thread I responded to had to deal with someones site not being indexed properly by Google, plus Webmaster Tools was stating the site urls are restricted by robots.txt… but the site didn’t have a robots.txt.

I had seen this very issue with a couple of other websites / blogs that I run. In almost every case I had initially blocked search engines via the settings > privacy > I would like to block search engines, but allow normal visitors because the site(s) were in development. Once the sites were ready I would switch the privacy setting to allow search engines but for some reason Goolge would still think that robots were blocked.

If this happens to you there are two ways to go about things…

The passive approach is this.. you can do nothing. If you post or publish often (a couple of times a week) and you have the Google XML Sitemap Plugin installed on your Wordpress site then it might take a few weeks for Google to hash things out… but things should work out on there own.

The proactive approach is go back to your privacy settings toggle it block engines again, update.. settings then switch it back to allow engines and update settings. Next, in a text editor like Notepad create a new file called robots.txt and save to your website root folder (same directory as wp-config.php). In the file paste the following….

User-agent: *
Disallow:

Upload the file to the server and re-submit your XML sitemap via Google Webmaster Tools Interface. What we just did was override the robots exclude / restriction and tell all the engines that the entire site is open to be crawled.

Hope this is useful to