A/B Test: Comparing Product Templates

A merchant feels that product reviews are important and should have a more dominant placement on product detail pages. The merchant asks a developer to help him understand better what to do.

The developer does the following:

  1. In Studio: copies the product detail page type template (pt_productdetails.isml), then changes the copy to locally include either the template with the new layout or the template with the old layout, by using the scripting API to determine what test group is active for the customer (ABTestMgr.isParticipant). Because the new layout differs significantly from the original layout, the developer only wants to test it on a few selected products.
    • Changes the product detail page pipeline (Product-Show), which calls product.isml, which calls pt_productdetails.isml to dynamically select the copied product detail page template for products of brand MyBrand and the old one otherwise.
    • Uses remote includes to dynamically include either the new or old template for each customer.
  2. Selects site > Merchant Tools > Online Marketing > A/B Tests, and configures a test called Product-reviews.
    • Enables the test and enters an optional description that is for the developer's own usage.
    • Includes an email address to be notified when there are statistically significant results for the key metric.
    • Selects Adds to Baskets as the key metric, because of expecting the product review placement to result in more adds to the basket.
    • Sets the test to Never expire for the life of the test. (A slot on the product detail page always appears regardless of the test group the customer is assigned to.
    • Sets the test to run for 1 week.
    • Makes the test available to the Everyone group, so any customer that views the product detail page can be a participant.
    • Creates two segments, each allocated at 50%, but with no test segment experiences.
    • Clicks Add Experience button and sets the segment to Enable Custom Scripting.

The developer stages the test to production in concert with the cartridge changes containing the templates and pipeline. After 5 days the developer checks the results. The old template seems to result in more adds to baskets than the new template, not the results expected. So the developer extends the test on production for an extra week, and after five more days, checks again. The developer now knows for sure that the old template results in more Add to Baskets.

Note: Salesforce B2C Commerce doesn't compute statistical significance when the control group outperforms the test segments.
On production, the developer changes the end date of the test to the previous day. The changed templates and pipeline remain in place, but all traffic now sees the old template because customers are no longer considered participants in the test group that shows the new template. The developer also coordinates a rollback of the template and pipeline changes so that the old template can be cached as it was before.

In sharing the results with the merchant, they determine that the more prominent location of the product reviews, resulting from the test, pushed the Add to cart button too far down on the page. The merchant asks the developer to create a new test that accounts for locations of both of these elements.

Note: Use this as an example. Your customized implementation might use different templates.

See Also:

  1. ISML
  2. APIs for A/B Testing.
  3. A/B Testing Examples
  4. Creating an A/B Test