A/B Test: Getting Complicated

A merchant wants to simultaneously test changes to the order of four recommended products and the background image for a home page slot. There is a limited time to perform the test and can't run the tests one after the other.

The merchant does the following on the Staging instance:

  1. Selects site > Merchant Tools > Online Marketing > A/B Tests, and creates an A/B test called recommended-products.
  2. Enables the test and enters an optional description that is for usage in the testing.
  3. Includes an email address to be notified when there are statistically significant results for the key metric.
  4. Selects Adds to Baskets as the key metric.
  5. Configures the test to:
    1. Expire along with sessions.
    2. Run for 1 week.
    3. Be available for the Everyone customer group.
  6. Creates four segments plus a control group and allocates 20% of the participants to each.
  7. Selects site > Merchant Tools > Online Marketing > Content Slots and opens the global content slot, home-right. They then create four slot configurations, one for each order of recommended products and background image.
    ID Content Type Content Template
    bgImage-1 Content Asset Image-1 slots/product/verticalcarousel.isml
    bgImage-2 Content Asset Image-1 slots/product/verticalcarousel.isml
    Product-1 Product product1, product2, product3, product4 slots/product/verticalcarousel.isml
    Product-2 Product product1, product2, product3, product4 slots/product/verticalcarousel.isml
  8. Returns to the test and assigns the corresponding slot configurations to the test segments. Each segment has the image applying first and the product order second, as follows:
    Segment ID Content Asset Slot Product Slot
    recommended-products-1 bgImage-1 Product-1
    recommended-products-2 bgImage-1 Product-2
    recommended-products-3 bgImage-2 Product-1
    recommended-products-4 bgImage-2 Product-2

The test is replicated to production and runs for the week. These are the results:

The combination of the better product order and background image shows a statistically significant difference in Adds to Baskets over the other combinations. Thus, the winning combination is used for an upcoming change to the homepage slot content.

X Privacy Update: We use cookies to make interactions with our websites and services easy and meaningful, to better understand how they are used. By continuing to use this site you are giving us your consent to do this. Privacy Policy.