Robots.txt

The robots.txt file is used to specify which web pages can be indexed by search engines. This file should be placed in the root directory of the website, and each subdomain should have its own robots.txt file.

Instructions in robots.txt follow a particular standard (see examples below). However, these instructions can't be enforced; compliance depends on search engines respecting the file's directives. Some malicious bots may even use the robots.txt file to find sensitive webpages and go directly to them.

Examples

Forbid indexation of the website to all robots.

User-agent: *
Disallow: 

Allow any robots to access all the website.

User-agent: *
Allow: /