We've compiled a list of known issues and troubleshooting to help you configure your A/B tests as desired.
"Visitor" Metric Incremented Twice Upon Logout
Sometimes a visitor metric can be counted twice. For example, an A/B test has Everyone as the assigned customer group. If a registered customer logs out after placing an order, and then places another order as a guest, the visitor metric is counted twice. This customer is counted twice even though it's the same customer using the same PC in the same session. When a customer performs a guest checkout only, the customer is counted once, no matter how many orders the customer makes. They’re only counted once because when a registered customer logs out, B2C Commerce increments the visits and visitors metrics as if a new, anonymous customer has come to the site. This anonymous customer is considered another visitor and is again evaluated for A/B test participation. If the customer qualifies for an A/B test, B2C Commerce generates relevant data as part of the customer's participation.
This could cause a problem with the metrics because the registered customer is counted twice and a guest checkout is counted once. When collecting A/B testing metrics, B2C Commerce doesn’t know how many customers are registered or how many of users could result in a guest checkout.
Download of A/B Test Results Displays Security Warning in IE8
If you download A/B tests into a .csv file for the first time using an IE8 browser, a security message appears. The message informs you that the download has been blocked. After you confirm the message, the A/B test list page appears and the file isn't downloaded. If you try a second time, the download begins. If you leave the A/B test module and return, the security warning reappears.
This error only occurs with Microsoft® IE8.
A/B Test Limitations Enforced by Quotas
Quotas enforce the following limitations:
Anonymous Customer Participates in A/B Test for Registered Customers
An anonymous customer can participate in an A/B test configured for registered customers only in the following scenario. A registered customer participates in an A/B test that has expiration set to Never and is configured for registered customers. The registered customer leaves or logs out of the site and the personalization cookie still exists in the respective system. Now, an anonymous customer can participate in the test.
This is to help produce a consistent storefront experience for the A/B test at the expense of potential false positives. When the cookie was set in a previous visit for someone using the same browser, we assume it was also set for the current browser user.
Participation in A/B Test for Registered Customers Might Happen Rarely
B2C Commerce uses a round-robin assignment of participants to tests. For example, an A/B test is configured for Everyone. A second A/B test configured for Registered Customers only has participants if those customers are remembered and the round-robin mechanism has chosen that test.
For example:
| A/B test | Triggered | Customer group |
|---|---|---|
| A | Immediately | Everyone |
| B | Immediately | Registered |
Potential scenarios:
We recommend that you run one test at a time for most participants. Running one test at a time can shorten the time to statistical significance; and no overlap, to prevent skewing the results.
Cannot Add a Campaign or A/B Test Schedule to a Content Slot
Users occasionally get confused about why they can't configure a campaign within the Business Manager Content Slot module. This is by design because users who have permission to access the Content Slot module don’t always have permission to access the Campaigns module. You can only create campaigns and A/B tests if you have permission to access the Business Manager Campaigns module.