A/B Testing Known Issues

The following issues are known to Salesforce B2C Commerce:

"Visitor" Metric Incremented Twice Upon Logout

For an A/B test with Everyone as the assigned customer group, if a registered customer logs out after placing an order, and then places another order as a guest, the visitor metric is counted twice even though it's the same customer using the same PC in the same session. When a customer performs a guest checkout only, the customer is counted only once, no matter how many orders the customer makes. This is because when a registered customer logs out, B2C Commerce increments the visits and visitors metrics as if a new (anonymous) customer has come to the site. This anonymous customer is considered another visitor and is again evaluated for A/B test participation. If the customer qualifies for an A/B test, B2C Commerce generates relevant data as part of the customer's participation.

This could cause a problem with the metrics because the registered customer is counted twice and a guest checkout is counted once. B2C Commerce has no way of knowing when collecting A/B testing metrics, how many customers are registered users or how many of those might result in a guest checkout.

Download of A/B Test Results Displays Security Warning in IE8

If you try to download A/B tests into a .csv file for the first time using an IE8 browser, a security message appears advising that the download has been blocked. After you confirm the message, the A/B test list page appears and the file isn't downloaded. If you try a second time, the download begins. If you leave the A/B test module and return, the security warning reappears.

This only occurs with Microsoft® IE8.

A/B Test Limitations Enforced by Quotas

The following limitations are enforced by quotas:

Anonymous Customer Participates in A/B Test for Registered Customers

If a registered customer participates in an A/B test that has expiration set to Never and is configured for registered customers, an anonymous customer can participate in this test if the registered customer logs out or leaves the site and the personalization cookie still exists in the respective system.

This is to help produce a consistent storefront experience for the A/B test at the expense of potential false positives. When the cookie was set in a previous visit for someone else using the same browser, we must assume it was also set for the current browser user.

Participation in A/B Test for Registered Customers Might Happen Rarely

Because B2C Commerce uses a round-robin assignment of participants to tests, if an A/B test is configured for Everyone, a second A/B test configured for Registered Customers only have participants if those customers are remembered and the round-robin mechanism has chosen that test.

For example:

A/B test Triggered Customer group
A Immediately Everyone
B Immediately Registered

These are the potential scenarios:

We recommend that you run one test at a time for most participants, which might shorten the time to statistical significance; and no overlap, to prevent skewing the results.

Cannot Add a Campaign or A/B Test Schedule to a Content Slot

Users occasionally get confused about why they can't configure a campaign within the Business Manager Content Slot module. This is by design because users who have permission to access the Content Slot module might not have permission to access the Campaigns module. You can only create campaigns and A/B tests if you have permission to access the Business Manager Campaigns module.

Related Links

A/B Testing Suggestions

A/B Testing for Developers

A/B Testing FAQ

X Privacy Update: We use cookies to make interactions with our websites and services easy and meaningful, to better understand how they are used. By continuing to use this site you are giving us your consent to do this. Privacy Policy.