A/B Test: Comparing Product Templates

A merchant feels that product reviews are important and wants the reviews to have a more dominant placement on product detail pages. The merchant asks a developer for help with understanding better what to do.

The developer does the following:

  1. In Studio: Copies the product detail page type template (pt_productdetails.isml). The copy is changed to locally include the template with either the new or the old layout, using the scripting API to determine the active test group for the customer (ABTestMgr.isParticipant). Because the new layout differs significantly from the original layout, the developer only wants to test it on a few selected products.
    • Changes the product detail page pipeline (Product-Show). The pipeline calls product.isml, which calls pt_productdetails.isml to dynamically select the copied product detail page template for products of brand MyBrand and the old one otherwise.
    • Uses remote includes to dynamically include either the new or old template for each customer.
  2. Selects site > Merchant Tools > Online Marketing > A/B Tests, and configures a test called Product-reviews.
    • Enables the test and enters an optional description that is for the developer's own usage.
    • Includes an email address to be notified when there are statistically significant results for the key metric.
    • Selects Adds to Baskets as the key metric, because of expecting the product review placement to result in more adds to the basket.
    • Sets the test to Never expire for the life of the test. (A slot on the product detail page always appears regardless of the test group the customer is assigned to.
    • Sets the test to run for 1 week.
    • Makes the test available to the Everyone group, so any customer that views the product detail page can be a participant.
    • Creates two segments, each allocated at 50%, but with no test segment experiences.
    • Clicks Add Experience button and sets the segment to Enable Custom Scripting.

The developer stages the test to production in concert with the cartridge changes containing the templates and pipeline. After 5 days, the developer checks the results. The old template seems to result in more adds to baskets than the new template, not the results expected. So the developer extends the test on production for an extra week, and after five more days, checks again. The developer now knows with certainty that the old template results in more Add to Baskets.

Note: Salesforce B2C Commerce doesn't compute statistical significance when the control group outperforms the test segments.
On production, the developer changes the end date of the test to the previous day. The changed templates and pipeline remain in place. However, all traffic now sees the old template because customers are no longer considered participants in the test group that shows the new template. The developer also coordinates a rollback of the template and pipeline changes so that the old template can be cached as it was before.

By sharing the results with the merchant, the merchant determines that the more prominent location of the product reviews pushed the Add to Cart button too far down the page. The merchant asks the developer to create a test that accounts for locations of both of these elements.

Note: Use this as an example. Your customized implementation could use different templates.
Related tasks
Create an A/B Test

© Copyright 2000-2023, Salesforce, Inc. All rights reserved. Various trademarks held by their respective owners.

 Show URL Submit FeedbackPrivacy Policy
X Privacy Update: We use cookies to make interactions with our websites and services easy and meaningful, to better understand how they are used. By continuing to use this site you are giving us your consent to do this. Privacy Policy.