Robots.txt

File used to direct or to tell web bots what pages and directories to index or not index. This file must be placed in the root directory on the server hosting your pages. The file should be named robots.txt and should have read permissions. Below is an example of a robots.txt file:

#robots.txt file for http://www.computerhope.com
User-agent: *
Disallow: /directory1/
Disallow: /directory2/

In the above example of the robots.txt file, the first line is a remarked statement that describes the file. Next, the User-agent line tells what robots or bots this file applies to. In the above example, the * represents all bots. Finally, the next two lines are the two directories that you wish the bots not to index.

If you wish to allow a bot to query all URLs in your website, enter the name of the bot followed by disallow: with a blank value, as seen in the below example:

User-agent: Somebot*
Disallow:

• A listing of known robots and a short description can be found at: http://www.robotstxt.org 
• Computer Hope's robots.txt file can be found at http://www.computerhope.com/robots.txt
• Additional information about preventing bots from indexing images using the robots.txt file can also be found on document CH000977.

Also see: Meta tags, Robot