Verify the Robots.txt File

Verifying that the robots.txt file is correctly placed for a robots.txt file:

  1. In your browser, enter the hostname of the instance, a slash, and then robots.txt.

    For example:

    • http://www.yourshophere.com/robots.txt
    • http://staging.realm.company.demandware.net/robots.txt
  2. You see the contents of your robots.txt file appear. If you get a blank page, the file isn't being picked up.
Note: You might need to make sure that the static content page cache was invalidated before you can see a new version of the robots.txt file.

To Verify That the Robots.txt File Is Behaving As Expected:

To check that your robots.txt file is behaving as expected, use the robots.txt analyzer tool in Google Webmaster Tools. You must have created a Google account to use this tool.

© Copyright 2000-2020, salesforce.com inc. All rights reserved. Various trademarks held by their respective owners.

 Show URL Submit FeedbackPrivacy Policy
X Privacy Update: We use cookies to make interactions with our websites and services easy and meaningful, to better understand how they are used. By continuing to use this site you are giving us your consent to do this. Privacy Policy.