When a robot visits a web site, such as
http://www.yourshophere.com/, it first checks for the robots.txt file and
analyzes it to determine if it's allowed to index the site. For the robots.txt file to be used
by Salesforce B2C Commerce, you must set up your hostname alias.
Here is a sample robots.txt file that prevents all robots from visiting the entire site:
User-agent: *# applies to all
robots
Disallow: /# disallow indexing of all
pages
The robot simply looks for a /robots.txt URI on your
site, where a site is defined as an HTTP server running on a particular host and port
number.