ignati [dot] com

Google Webmaster Tools has introduced a new official tool for site owners, Robots.txt Generator. This tool will help your website googlebot friendly by giving an easy wizard to make robots.txt.

With Google Robot.txt Generator you can create a robots.txt file to indicate which robots you don’t want crawling your site, and which files or directories you don’t want crawled. You can even choose which specific robot you want to have access to your site’s index and restrict other robots from doing the same thing.

Anyway some robots may ignore instruction in the robots.txt and continue to craw your site including those files or folders that have restrictions. So for highly sensitive files or documents, i would recommend you to use password protection for more privacy.

If you like this post then please consider to subscribe my feed. You can also have new post directly to your inbox if you prefer subscribe by email.