Robots.txt file is used to tell search engines what they are allowed to access on your website. The purpose of this is to stop search engines listing private and sensitive data such as admin login areas.

If you are happy to let search engines visit all files and folders that exist with your website then you do not need to use a robots.txt file. If you do not need to block any content then do not have a robots.txt file on your website.

If the file is incorrectly coded then you run the risk of blocking important content that you may want search engines to list.

If you want a robots.txt file to allow all the content and you really want to have one on your website then create a txt file called robots.txt and enter the following:

User-agent: *

Disallow:

For more information about robots.txt see here!

Tags: robots, robots.txt

This entry was posted on Thursday, February 4th, 2010 at 11:20 am and is filed under Blog Hosting, Cheap Web Hosting, MySQL Web Hosting, Shared Web Hosting, UK Web Hosting. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.