A/B Testing

Using Business Manager, you can assign promotions or slot content to an A/B test with common qualifiers such as source codes or customer groups. A/B test statistics use a combination of two-sided t-tests and chi-square tests to weigh test segments against each other and compute the statistical significance.

You can create and manage A/B tests, view test results, and change your site based on those results. You can configure changes to your sites that affect only a subset of visitors so that you can determine from collected data the experience that produces the best results.

You create test segment, where each segment configures a unique experience for a set of participants, against which you can test other segments and a control segment. You can define triggers that a customer must access to be considered for participation in a particular test.

When creating A/B tests, understand that, relative to other promotions available in Salesforce B2C Commerce, A/B tests are considered to have high precedence.

You can also customize A/B tests using the B2C Commerce API, and use the A/B test import/export functionality to manage A/B tests across your company. You can extract A/B test results in .csv output files, and use those files in your favorite spreadsheet application for further analysis.

A/B Testing vs. Personalization

Although the effect is similar, A/B testing segmentation is different from personalization segmentation. The regular personalization of a site, for example, done using customer groups, is segmented by criteria that you can define. Meanwhile, A/B test segmentation is executed randomly. With both, the user is presented with an experience that is potentially different than other users. A/B testing, however, goes beyond personalization. A/B testing collects metrics to help determine which experience, among two or more, performs the best according to one or more segmentation criteria.

A/B Testing Eligibility

A shopper’s eligibility for A/B tests is determined by the Tracking site preference. The Tracking site preference is session-specific and only shoppers for whom tracking is allowed are eligible for A/B tests.

Note: If your site doesn’t have the Tracking site preference enabled, A/B test eligibility is determined by your site's default Privacy setting.

To set this preference select Merchant Tools > Site Preferences > Privacy.

If a shopper opts in or out of tracking mid session, the A/B Test Metrics show a slight inconsistency. Changing tracking mid session can also cause a shopper's page to appear as though it's still tracking. Refreshing the page fixes this issue.
The consistency of an A/B test experience varies slightly for eligible registered and unregistered shoppers.
  • Eligible Registered–The mapping between the A/B test group assigned to the shopper's customer ID are stored in the database. As a result, the shopper's A/B test assignments are recalled, and therefore the shopping experience is consistent regardless of which device the shopper uses.
  • Eligible Guest–The A/B test group that the shopper is assigned to is written to a cookie on the shopper's browser. As a result, the shopper's A/B test assignments are recalled. The shopping experience is consistent only when the shopper is on the same device and using the same browser.

Eligibility and A/B Test Sample Size

The consent requirement for A/B test eligibility and the possibility of a varied A/B test experience for unregistered shoppers can impact your test sample size. Consider planning longer test durations or altering your test criteria to capture statistically significant A/B test date.

A/B Test Data Deletion

Shopper data deletion requests cascade to A/B test participant records. When a shopper requests to have their records deleted from the system, all the shopper's A/B test participant records are deleted.
X Privacy Update: We use cookies to make interactions with our websites and services easy and meaningful, to better understand how they are used. By continuing to use this site you are giving us your consent to do this. Privacy Policy.