A/B Test: Getting Complicated

A merchant wants to simultaneously test changes to the order of four recommended products and the background image for a home page slot. There is a limited time to perform the test and can't run the tests one after the other.

The merchant does the following on the Staging instance:

1. Selects site > Merchant Tools > Online Marketing > A/B Tests, and creates an A/B test called recommended-products.
2. Enables the test and enters an optional description that is for usage in the testing.
3. Includes an email address to be notified when there are statistically significant results for the key metric.
4. Selects Adds to Baskets as the key metric.
5. Configures the test to:
   1. Expire along with sessions.
   2. Run for 1 week.
   3. Be available for the Everyone customer group.
6. Creates four segments plus a control group and allocates 20% of the participants to each.
7. Selects site > Merchant Tools > Online Marketing > Content Slots and opens the global content slot, home-right. They then create four slot configurations, one for each order of recommended products and background image.

   ID	Content Type	Content	Template
   bgImage-1	Content Asset	Image-1	slots/product/verticalcarousel.isml
   bgImage-2	Content Asset	Image-1	slots/product/verticalcarousel.isml
   Product-1	Product	product1, product2, product3, product4	slots/product/verticalcarousel.isml
   Product-2	Product	product1, product2, product3, product4	slots/product/verticalcarousel.isml

8. Returns to the test and assigns the corresponding slot configurations to the test segments. Each segment has the image applying first and the product order second, as follows:

   Segment ID	Content Asset Slot	Product Slot
   recommended-products-1	bgImage-1	Product-1
   recommended-products-2	bgImage-1	Product-2
   recommended-products-3	bgImage-2	Product-1
   recommended-products-4	bgImage-2	Product-2

The test is replicated to production and runs for the week. These are the results:

• One recommended order of products leads to more Adds to Baskets than the other.
• One background image leads to more Adds to Baskets than the other.

The combination of the better product order and background image shows a statistically significant difference in Adds to Baskets over the other combinations. Thus, the winning combination is used for an upcoming change to the homepage slot content.