Start Here - A/B Testing

In this tutorial, we'll cover the basics regarding how to deploy and measure two different kinds of tests: 50/50 split & hold out. 

Test Types

1
A/B Split Tests 

50/50 split tests are most commonly used to determine which of two campaigns will drive a stronger response rate for a particular aspect (opens, clicks, conversions, etc) of a campaign. For the specific area where you're trying to improve response, think "Campaign A" vs. "Campaign B". After choosing a sample size and activating a new test, Rejoiner will randomly "split" your segment into two equal parts and deliver a control (your original creative) against a variant (your challenger creative). For a split test, you can choose to compare open rate, click-through rate, conversion rate, or revenue per email to determine a test winner. Test reporting provides aggregated statistics for the entire campaign, as well as email vs email comparison for campaigns that include multiple emails.

2
Hold Out Tests

The underlying goal of a hold out test is different than a split test. Hold out tests are structured so that one campaign is delivered to 90% of your defined segment, and a 10% control group receives nothing, or is "held out". The test and control groups are then compared based on revenue generated per customer. This gives us insight into the revenue being attributed by the customers who received the campaign vs. the customers who didn't. The question a hold out test seeks to answer is, "Am I creating a more valuable customer by sending this campaign or would these customers have purchased anyway?"  Hold out tests become particularly useful when measuring the profitability of campaigns that employ discounts or promotions. 

Launch a 50/50 Split Test

  1. From your Email Campaign manager, click the A/B test icon on the campaign you wish to test. Your original campaign will become the control for the test.
  2. From the test creation modal, select 50/50 split test, define a sample size (minimum 1,000 customers), choose a winner criteria (Open Rate, Click-through Rate, Conversion Rate, or Revenue per Email) and provide a short description of the test being run.

  3. Once you're happy with your test settings, click Create Test
  4. Upon test creation, your control campaign will be cloned as the test variant. Note: The test is not live at this point.
  5. From your Email Campaign manager, click Edit Variant and make the changes you'd like to test within your variant creative.
  6. When you're finished setting up your variant creative, check your control creative and make sure everything looks good. Note: Once you launch the test, both the control and variant creative will be locked and you will not be able to edit either.
  7. To launch the test, click Launch Test from your Email Campaigns Manager and make sure that your control campaign is also set to an active state. In your account, the a campaign with an active 50/50 split test should look similar to: 
      
  8. To check the results for any test, click the See Test Data from your Email Campaign Manager.

Launch a Hold Out Test

  1. From your Email Campaign manager, click the A/B test icon on the campaign you wish to test. Your original campaign will become the control for the test.
  2. From the test creation modal, select Hold Out, define a sample size (minimum 1,000 customers), and provide a short description of the test being run.
  3. Hold Out tests have only one one winner criteria: Revenue per customer
  4. Once you're happy with your test settings, click Create Test. Since the hold out group isn't receiving any emails, there is no need to configure the variant.
  5. To launch the test, click Launch Test from your Email Campaigns Manager and make sure that your control campaign is also set to an active state. In your account, the a campaign with an active Hold out test should look similar to: 

Pause a Test

  1. After a test has been activated, you may pause a test by switching the campaign toggle to the "Off" position
  2. Note: Pausing an active test will delete any scheduled recovery efforts, which may skew your test results

Choose a Winner

  1. Winning variations can be declared from the Test Results page on your dashboard.
  2. Select your test from the campaign drop down.
  3. Click Choose Winner and select which variation you would like to deploy as the the new control variation.
  4. Note: Choosing a winner will end the test, archive the losing variation, and create a new single campaign with your winner as the control.

Clone an Archived Test

  1. From your Email Campaigns Manager, click Archive in the upper right hand corner
  2. Scroll down to the campaign/test that you'd like to clone and click the icon below:
  3. You'll be asked to make a choice between cloning only the control, or control + variant

Still need help? Contact Us Contact Us