A/B Test: Getting Complicated

A merchant wants to simultaneously test changes to the order of four recommended products and the background image for a home page slot. There is a limited time to perform the test and can't run the tests one after the other.

The merchant does the following on the Staging instance:

  1. Selects site > Merchant Tools > Online Marketing > A/B Tests, and creates an A/B test called recommended-products.
  2. Enables the test and enters an optional description that is for usage in the testing.
  3. Includes an email address to be notified when there are statistically significant results for the key metric.
  4. Selects Adds to Baskets as the key metric.
  5. Configures the test to:
    1. Expire along with sessions.
    2. Run for 1 week.
    3. Be available for the Everyone customer group.
  6. Creates four segments plus a control group and allocates 20% of the participants to each.
  7. Selects site > Merchant Tools > Online Marketing > Content Slots, opens the global content slot, home-right, and creates four slot configurations, one for each order of recommended products and background image.
    ID Content Type Content Template
    bgImage-1 Content Asset Image-1 slots/product/verticalcarousel.isml
    bgImage-2 Content Asset Image-1 slots/product/verticalcarousel.isml
    Product-1 Product product1, product2, product3, product4 slots/product/verticalcarousel.isml
    Product-2 Product product1, product2, product3, product4 slots/product/verticalcarousel.isml
  8. Returns to the test and assigns the corresponding slot configurations to the test segments, each with the image applying first and the product order second, as follows:
    Segment ID Content Asset Slot Product Slot
    recommended-products-1 bgImage-1 Product-1
    recommended-products-2 bgImage-1 Product-2
    recommended-products-3 bgImage-2 Product-1
    recommended-products-4 bgImage-2 Product-2

The test is replicated to production and runs for the week. These are the results:

This has led the combination of the better product order and better background image to show a statistically significant difference in Adds to Baskets over the other combinations. Thus, the winning combination is used for an upcoming change to the homepage slot content.

Related Links

A/B Testing Examples

Creating an A/B Test