A/B Test: Comparing Search Sorting

A search expert wants to test the difference the search order has on category revenue. The expert used Active Data to define two new sorting rules: Top Sellers and Best Margin. The test uses rules to see which one produces the highest revenue for the category.

The search expert does the following:

1. Selects site > Merchant Tools > Online Marketing > A/B Tests, and configures an A/B Test compare-search-sorting.
2. Enables the test and enters an optional description that is for usage in the testing.
3. Includes an email address to be notified when there are statistically significant results for the key metric.
4. Selects Average Order Value as the key metric.
5. Configures participation to expire per Session.
6. Sets the test to run from August 2 through August 31.
7. Makes the test available to all customers via the Everyone group.
8. Creates two test segments:
   • Sets a sorting rule test segment to Top Sellers as the default for a search refined by that category, allocated to 50% of the participants.
   • Sets a sorting rule test segment to Best Margin as the default for a search refined by that category, allocated to 45% of the participants
   • The control group gets a different rule, Fastest Sellers, because it's configured site-wide as the default for that category. This is allocated to 5%.

The test is replicated sometime in July and kicks in on August 2. The search expert checks the results now and then but there is never a conclusion. The test ends on August 31, and the search expert checks the results a final time. There were minimal differences among the groups for some metrics, including Average Order Value; but even with 2,500 orders, none of these differences were statistically significant.

The search expert redesigns the Top Sellers and Best Margin rules to use Sales Velocity for the last 7 days, instead of the last 365 days. The expert is hoping that the sort order is more responsive to current trends. The expert copies the test and runs it again for the 30 days following Labor Day. The changes to the sorting rules themselves are enough to affect the test; it's not necessary to change the test groups. The changed sorting rules and the copied A/B test are replicated to production and start running.

Two weeks later, an email indicates that the Top Sellers rule has shown a statistically significant higher Average Order Value. The suspicion that it was the timeliness of the active data that mattered in this case was correct. The search expert selects the Top Sellers rule as the default.