A/B Testing

Using Business Manager, you can assign promotions or slot content to an A/B test with common qualifiers such as source codes or customer groups. A/B test statistics use a combination of two sided t-tests and chi-square tests to weigh test segments against each other and compute the statistical significance. You can create and manage A/B tests, view test results, and make changes to your site based on those results. You can configure changes to your sites that affect only a subset of visitors so that you can determine from collected data the experience that produces the best results.

You create test segments, where each segment configures a unique experience for a set of participants, against which you can test other segments and a control segment. You can define triggers that a customer must access in order to be considered for participation in a particular test.

When creating A/B tests, understand that relative to other promotions available in Salesforce B2C Commerce, A/B tests are considered to have higher precedence.

You can also customize A/B tests using the B2C Commerce API, and use the A/B test import/export functionality to manage A/B tests across your comapany. You can extract A/B test results in .csv output files, and use those files in your favorite spreadsheet application for further analysis.

A/B Testing vs. Personalization

Although the effect is similar, A/B testing segmentation is different from personalization segmentation. The regular personalization of a site, done using customer groups, for example, is segmented by criteria that you can define, while A/B test segmentation is done randomly. With both, the user is presented with an experience that is potentially different than that of other users. A/B testing, however, goes beyond personalization to collect metrics to help determine which experience, among two or more, performs the best according to one or more segmentation criteria.

A/B Testing Eligibility

A shopper’s eligibility for A/B tests is determined by the Tracking site preference. See Set Privacy Preferences. The Tracking site preference is session-specific and only shoppers for whom tracking is allowed are eligible for A/B tests.

Note: If your site does not set the Tracking site preference value for a shopper through a consent management system, A/B test eligibility is determined by your site's default Privacy setting.

To set this preference select Merchant Tools > Site Preferences > Privacy Settings.

If a shopper opts in or out of tracking mid session, the A/B Test Metrics show a slight inconsistency. Changing tracking mid session can also cause a shopper's page to appear as though it's still tracking. Refreshing the page fixes this.
The consistency of an A/B test experience varies slightly for eligible registered and unregistered shoppers.
  • Eligible Registered–The mapping between the A/B test group assigned to the shopper's customer ID are stored in the database. As a result, the shopper's A/B test assignments are recalled, and therefore the shopping experience is consistent regardless of which device the shopper uses.
  • Eligible Guest–The A/B test group that the shopper is assigned to is written to a cookie on the shopper's browser. As a result, the shopper's A/B test assignments are recalled and the shopping experience is consistent only when the shopper is on the same device and using the same browser.

Eligibility and A/B Test Sample Size

The consent requirement for A/B test eligibility and the possibility of a varied A/B test experience for unregistered shoppers can impact your test sample size. You might need to plan longer test durations or alter your test criteria to capture statistically significant A/B test date.

A/B Test Data Deletion

Shopper data deletion requests cascade to A/B test participant records. When a shopper requests to have their records deleted from the system, all of the shopper's A/B test participant records are deleted.

Related Links

Set Privacy Preference

Browser-Based Local Data Storage

Consent Management:Track Shopper Consent

Campaigns and Promotions

A/B Test Segments

A/B Test Participants

A/B Test Participant Triggers

A/B Test Rank Precedence

Managing A/B Tests

A/B Testing Examples

A/B Test Results

A/B Testing for Developers

A/B Test Object Import/Export