• Wednesday January 7, 2009 - 6:08:32 pm GMT
• registered : 1156343
• online : 77163
• questions/day : 5632
• Answer rate : 76.5%

Presentation of the robots.txt File

robots.txt is a text file that contains commands for search engine indexing robots that specify the pages that can and cannot be indexed. When a search engine explores a website, it starts by looking for the robots.txt file at the root of the site.

robots.txt File Format

The robots.txt file is an ASCII file found at the root of the site. It can contain the following commands:

• User Agent: used to specify the robot that is subject to the following orders. The value * means "all search engines"
• Disallow: used to identify the pages to be excluded during indexing. Each page or path that is to be excluded must be on a separate line and must start with / The value / alone means "all of the website's pages".

The robots.txt file should not contain any empty lines!

Here are some examples of robots.txt files:

• All pages are excluded:

  User Agent: *
  Disallow: /

• No pages are excluded (equivalent to having no robots.txt file, meaning that all the pages are visited):

  User Agent: *
  Disallow: 

• Only one robot is authorized:

  User Agent: RobotName
  Disallow:
  User Agent: *
  Disallow: /

• One robot is excluded:

  User Agent: RobotName
  Disallow: /
  User Agent: *
  Disallow:

• One page is excluded:

  User Agent: *
  Disallow: /directory/path/page.html

• All pages from a directory and its subfolders are excluded:

  User Agent: *
  Disallow: /directory/

Examples of User Agents

Here are a few examples of User Agents for the most popular search engines:

For More Information

The web robots page

Last update on Thursday October 16, 2008 02:43:14 PM.