Penn Computing
Computing Menu Computing A-Z
Computing Home Information Systems & Computing Penn

Robots.txt Format Example

A robots.txt file will prevent search engines from indexing Web documents. The file consists of one or more records separated by one or more blank lines. Each record identifies the name of the robot and the access policy for that robot.

Note the following when creating a robots.txt file:

  • You can insert comments in your file by using the pound (#) sign. The robot will ignore text after this sign.
  • You can type information in the file in uppercase or lowercase letters. It doesn't matter.
Example
The asterik (*) means that any robot visiting the site will be prevented from indexing any Web documents starting with the URL books/sample/author/.

User-agent: *

Disallow: books/sample/author/ #You can use the pound sign to enter comments.

See A Standard for Robot Exclusion for more information about the format of robots.txt files.

top

Information Systems and Computing
University of Pennsylvania
Comments & Questions


University of Pennsylvania Penn Computing University of Pennsylvania Information Systems & Computing (ISC)
Information Systems and Computing, University of Pennsylvania