A/B Test: Comparing Promotions

As part of a new campaign, the Marketing Manager wants to know which of the following promotions perform the best against the control group:

The Marketing Manager does the following:

  1. Selects site > Online Marketing > Promotions, checks that the three promotions exist and adds any that are missing.
    • Free Shipping over 100
    • 10% off 100
    • $25 off 200
  2. Selects site > Merchant Tools > Online Marketing > A/B Tests, and creates a new test with the ID Content Promotion Test.
  3. Enables the test and enters an optional description that is for usage in the testing.
  4. Includes an email address to be notified when there are statistically significant results for the key metric.
  5. Selects Orders as the key metric, but also considers the other metrics when looking at the results. (The test is looking for the promotion that leads to the highest Conversion Rate. The test is also looking for the one that maximizes Average Order Value and minimizes the Average Discount Amount.
  6. Selects that the participation Never expires. The Marketing Manager wants to offer customers the same promotion if they return to the site in a repeat visit during the test.
  7. Configures the test to include participants from the Everyone group, so that all customers can participate.
  8. Sets up three test segments:
    • Free Shipping over 100 allocated 30%
    • 10% off 100 allocated 30%
    • $25 off 200 allocated 30%
      Note: Business Manager automatically allocates the remaining 10% to the control group.
  9. Selects site > Merchant Tools > Online Marketing > Content Slots, identifies a slot named Header Banner on the site to show a banner for each promotion included for the test, and creates three slot configurations for Header Banner, each with the content for one of the three promotions
  10. Returns to the test and includes the corresponding slot configuration as an experience for each test group besides the control.
  11. Sets the test start date to the coming midnight and the end date to 9PM Saturday. (The Marketing Manager would like an answer before the summer clearance begins next Monday.)

After creating the test on staging, the Marketing Manager replicates it to production. The test goes into effect at midnight. In the morning there are no results, but the following day, results start to come in. On Saturday morning, an email alert indicates that there were results, which can be reviewed on the results page, as follows:

The Marketing Manager concludes that the free shipping over $100 and $25 off $200 promotions both perform better than the control group, and selects them to be used on the site.

Related Links

A/B Testing Examples

Creating an A/B Test