Hold Out Test Reports

In this article, we'll learn more about hold out testing and how test results are calculated.


Hold out testing is the practice of measuring the purchase behavior/value of a control group who does not receive marketing against a group of customers who do receive marketing. The goal of the test is to compare the two based on the value of an individual customer within each group. These tests answer the question, 'Am I creating a more valuable customer by marketing vs. not marketing?'

Hold out tests are comprised of three parts:

A user-defined sample size
A randomly selected 10% holdout group (who will not receive marketing)
A 90% active group (who will receive marketing)

Hold Out Test Report

Interpreting Hold Test Results

Important: Hold out tests compare the value of a single customer in the Active group against the value of a single customer in the Holdout group. Unlike our normal influence based attribution model, revenue tracking in the Active group does not distinguish between customers who convert outside of Rejoiner campaigns vs. customers who open or click Rejoiner emails. All revenue generated by customers in the Active group is tallied, regardless of their engagement with emails.

Sends: The total of number of emails sent to the Active group. The Holdout group will always receive 0 emails.

Customers: The total number of customers included in each group

Revenue: The total revenue generated by each group, regardless of open or click behavior

Revenue per customer: The total revenue generated by each group during the test divided by the number of customers in each group

Lift: The percentage improvement / lack of improvement of the Holdout group compared to the Active group

Still need help? Contact Us Contact Us