A/B Test: Comparing Promotions

You can compare promotions with A/B tests.

As part of a new campaign, the Marketing Manager wants to know which of the following promotions perform the best against the control group:

• Free Shipping on orders over $100
• 10% of orders over $100
• $25 off orders over $200

The Marketing Manager does the following:

1. Selects site > Online Marketing > Promotions, checks that the three promotions exist and adds any that are missing.
   • Free Shipping over 100
   • 10% off 100
   • $25 off 200
2. Selects site > Merchant Tools > Online Marketing > A/B Tests, and creates a test with the ID Content Promotion Test.
3. Enables the test and enters an optional description that is for usage in the testing.
4. Includes an email address to be notified when there are statistically significant results for the key metric.
5. Selects Orders as the key metric, but also considers the other metrics when looking at the results. (The test is looking for the promotion that leads to the highest Conversion Rate. The test is also looking for the one that maximizes Average Order Value and minimizes the Average Discount Amount.
6. Selects that the participation Never expires. The Marketing Manager wants to offer customers the same promotion if they return to the site in a repeat visit during the test.
7. Configures the test to include participants from the Everyone group, so that all customers can participate.
8. Sets up three test segments:
   • Free Shipping over 100 allocated 30%
   • 10% off 100 allocated 30%
   • $25 off 200 allocated 30%
     Note: Business Manager automatically allocates the remaining 10% to the control group.
9. Selects site > Merchant Tools > Online Marketing > Content Slots and identifies a slot named Header Banner on the site to show a banner for each included promotion. They then create three slot configurations for Header Banner, each with the content for one of the three promotions
10. Returns to the test and includes the corresponding slot configuration as an experience for each test group besides the control.
11. Sets the test start date to the coming midnight and the end date to 9PM Saturday. (The Marketing Manager would like an answer before the summer clearance begins next Monday.)

After creating the test on staging, the Marketing Manager replicates it to production. The test goes into effect at midnight. In the morning, there are no results, but the following day, results start to come in. On Saturday morning, an email alert indicates that there were results, which can be reviewed on the results page, as follows:

• Orders: The Free Shipping over $100 segment shows a statistically significant difference in Order Rate at 5%. The other segments show a roughly equal Order Rate of 4%.
• Average Order Value: The $25 off $200 segment has a statistically significant difference at $59.50. The other segments show a roughly equal Average Order Value around $47.
• Average Revenue per Visit: The Free Shipping over 100 segment is higher, but the difference isn't statistically significant.

The Marketing Manager concludes that the free shipping over $100 and $25 off $200 promotions both perform better than the control group. They use the two promotions on the site.