A/B Testing FAQ

These are some common A/B testing questions:

Q: How is A/B testing sorting rule inheritance related to category sorting rule inheritance?

A: They are directly related: you must configure your A/B tests to take advantage of that inheritance. When configuring A/B tests, you must only select the top level categories and rely on inheritance. This means that you shouldn't not select all subcategories in the category tree (explicitly assigning a sorting rule to each category). See A/B Testing Rank Precedence.

Q: Is it true that you must add <varyby="price_promotion"> to each template that uses iscache for A/B testing to work properly?

For example, the SiteGenesis product-related templates already include this, but the slot content templates don't. To use a slot content as an A/B test experience, do I need to add this parameter to the slot content-related templates?

A: Caching is a complex topic, and there is no general rule for all templates. If you have a template that shows static content, you should cache that template for a reasonably long period of time, such as 60 minutes. If you have a template that is specific to each storefront customer, such as the cart contents, you should never cache that template. If you have a template that is personalized by price, promotion, search results order, or a custom A/B test experience, you should cache that template using the <varyby="price_promotion"> setting.

If you have an unpersonalized home page, for example, it should be cached for 60 minutes. A/B tests still work for the site, and storefront customers are placed in A/B test segments as expected despite this cached page, whether it's the first page they view or if they never view it.

If you have a search that uses a sorting rule, your search results page should account for the possibility of personalized sorting rules by caching it with the <varyby="price_promotion"> setting. Even if you have no A/B tests or no A/B tests that have sorting rule experiences, you should do this because there could be campaigns that personalize the sorting rule for a search because of customer group, source code, or coupon qualifiers.

Using <varyby="price_promotion"> to cache pages isn't really about A/B tests, but about personalization. A/B tests are one way to personalize. Slot caching is a completely different way. Slot content personalization depends only on the customer groups and A/B test segments that apply for a storefront customer, not the promotions, price books, or the like.

You typically cache slot content for 60 minutes (for example) even if that content only appears to a specific customer group or a specific A/B test segment. This is because Salesforce B2C Commerce only renders that content specifically for that group or segment, and everyone in that group or segment sees the same content. If for example, only that specific customer group or A/B test segment sees the slot content, and that content includes the storefront customer's name, then it could not be cached for 60 minutes because everyone would see the same name. There are no personalized slot content templates in SiteGenesis, which is why <varyby="price_promotion"> doesn't appear there.

Q: We are experiencing some skewed visitor metrics when using the A/B testing feature. Do you know what might be causing this?

A: Possibly. If you are using Facebook social plug-ins, you might have issues with your analytics data. Depending on the plug-in, you might receive an artificial inflation of Facebook referring traffic and an increase in pageviews. This is because some Facebook social plug-ins redirect when a user interacts with the plug-in, and even when the plug-in merely appears on the page. The redirect goes to Facebook and quickly returns the user to the storefront site. The redirect doesn't pass the referrer through, so the traffic sources report show that the visit originated from Facebook and overwrites the correct campaign or traffic source that brought the user to the site. As a result, your traffic sources reporting, for example, visitor metrics, show more visits than actual customer visits.

These plug-ins might be affected:

In the case of the SiteGenesis or Mobile Web Storefront applications, the Like button is supposed to be an inactive display element until a user clicks it, but it actually makes a call to Facebook that issues another call back to the referenced product detail page. When a customer views a product, the Like button causes a second visit to the product detail page and is counted as another A/B test visitor. Because this is a consistent factor across all segments, it does not change the ratio for the test results.

Q: Are any A/B testing parameters enforced by quotas?

A: One A/B testing limitation is enforced by quotas: Maximum of five test segments within a test, including the control group segment, so four user-defined segments total.

Note: Best practice is to edit and configure A/B tests on staging. Then, only after they have been tested, reviewed and approved, should they be replicated to production.

Q: You can create or import an unlimited number of active A/B tests, yet only a limited number of them can be truly active. How does B2C Commerce select the active tests?

A: The Business Manager selects the active tests by:

This determination is dynamic. If a test is enabled or newly created, it could immediately take precedence over one of the others. B2C Commerce allows a maximum of 12 A/B test segments for all active A/B tests, while a single A/B test can have up to four segments plus the control group.

Q: Is Active Data tagging a pre-requisite to using A/B testing?

A: No. Active data isn't a requirement, but it's recommended. For example, active data can be used for sorting rules.

Q: A/B testing set up is done on staging and then replicated to production. Do I also need to re-index?

A: No. You don't need to re-index.

Q: Is it true that the more tests you run, the more diluted your participant pool gets?

A: Running tests one after the other has no effect on the participant pool. Running tests simultaneously results in the tests sharing the possible participants, because each customer can participate in only one test at a time. Each test has fewer participants than if a test was the only one running on the site. Fewer participants means a smaller sample size, which on average requires that tests run longer to reach a conclusion. However, there can be only three simultaneous A/B tests in B2C Commerce.

Q: What experiences does B2C Commerce support?

A: B2C Commerce supports experiences with promotions, content slots and sorting rules. In addition, we support script customization where you can define other, custom experiences. See A/B Testing for Developers.

Q: Do you recommend running one test for a campaign with multiple experiences per segment (for example, one slot/one sorting rule), as opposed to running two separate tests (one for the slots, the other for the sorting rules)?

A: The answer depends on what you want to test. If you want to test independently the effect of slots versus the effect of sorting rules, you would run separate tests. If you want to test the combined effect of a strategy that includes both slots and sorting rules, you would run a single test. If you run a single test, you can't tell if the results are due to the slots, the sorting rules, or a combination of the two. Run individual tests to better understand this level of detail.

Q: Do parallel running tests influence each other?

A: Yes and no. Yes, if experience overlap, because users don't participate in other tests, which might skew the results. No, if experiences don't overlap. However, if you measure against the same metric, for example, conversion rate, it's difficult to argue that an increase is caused by one or both experiences. Also, it takes longer until the tests are conclusive because there are more test segments into which the traffic is distributed. In general, we recommend that you run one test at a time, to eliminate all doubt of other influences.

Q: If I have an A/B test for slots on the homepage, how do I determine the percentage to allocate for segments versus the control group?

A: The reasons to select one percentage over another depends on the reason for the test and its configuration. For example, to test content slots on the homepage, comparing conservative content with something risky or provocative, a merchant might choose a low allocation for the group seeing the risky content. Alternatively, to test two versions of the same slot with minor differences, such as with color theme, messaging and font, a merchant would probably select equal percentage (50/50 or 33/33/33) allocations. Using a lower allocation results in a smaller sample size of participants in that segment, which on average requires that you run the test for a longer time period. You could run the risky content with a 50/50 allocation for a shorter test; but a with a smaller percentage allocation, you can pull the test faster if it generates a negative reaction, with fewer users having seen the risky content.

Q: What events cause an email notification to be sent, and how frequently are they sent?

A: B2C Commerce sends email notifications when:

Email notifications are sent hourly.

Related Links

A/B Testing Suggestions

A/B Test Rank Precedence

A/B Testing for Developers