Verify the Robots.txt File

Verifying that the robots.txt file is correctly placed for a robots.txt file:

  1. In your browser, enter the hostname of the instance, a slash, and then robots.txt.

    For example:

    • http://www.yourshophere.com/robots.txt
    • http://staging.realm.company.demandware.net/robots.txt
  2. You see the contents of your robots.txt file appear. If you get a blank page, the file isn't being picked up.
Note: You might need to make sure that the static content page cache was invalidated before you can see a new version of the robots.txt file.

To Verify That the Robots.txt File Is Behaving As Expected:

To check that your robots.txt file is behaving as expected, use the robots.txt analyzer tool in Google Webmaster Tools. You must have created a Google account to use this tool.