Holdout Group: How to Measure the True Lift of Your Campaigns

It’s the Monday after Diwali. The CRM team’s deck says the festive email series drove ₹42 lakh in sales, and everyone’s pleased. Then the CFO asks one awkward question: how many of those people would have bought anyway? Nobody in the room knows. A holdout group is how you answer that question, and it’s simpler to run than you’d think.

Attribution tells you which message a buyer clicked last. It can’t tell you whether that message changed anything. That gap matters more every year, as budgets get tighter and customers get more messages than they read. Customer data platforms like NVECTA make holdouts a routine part of every campaign rather than a one-off experiment.

This guide covers-

• What a holdout group is, and how it differs from an A/B test
• Why attribution alone overstates what your campaigns achieve
• How to calculate lift, with a worked rupee example
• A step-by-step way to set one up, size it and read the results
• How NVECTA helps you run holdout groups on every campaign

What Is a Holdout Group?

Put simply, a holdout group is a random slice of your audience that deliberately doesn’t receive a campaign. Everyone else gets it as normal. Afterwards, you compare the two. Did the people who got the message buy more? By how much? That difference is the real effect of the campaign.

Think of it like a clinical trial. Doctors don’t judge a new medicine by counting patients who recovered; they compare against a group that got a placebo, because some people get better without any treatment at all and the trial needs to know how many. Marketing works the same way, because plenty of customers recover, or buy, on their own.

We’d argue it’s the most honest number in CRM. Last click? Irrelevant. It only asks whether sending the message moved sales, whether that’s a ₹499 kurta or a ₹5,000 SIP top-up.

Holdout Group vs Control Group

Any difference? Not really. Most marketers use the two terms for the same thing: the people who were held back. “Control group” comes from science and A/B testing, while “holdout” is more common in CRM and advertising. You’ll see both in tools and reports, so don’t let the naming confuse you.

Holdout Group vs A/B Test

People mix these up all the time, but they don’t answer the same question. An A/B test compares two versions of a message. A holdout compares sending something with sending nothing.

Why Do Holdout Groups Matter?

You can run a CRM programme for years without one, and plenty of brands in Mumbai and Bengaluru do exactly that, reporting open rates and attributed revenue every month without ever asking what would’ve happened if they’d stayed quiet. But without a holdout, every report you share is quietly answering the wrong question.

Attribution Takes Credit It Didn’t Earn

Say Meera orders atta and ghee from your grocery app every Saturday morning, like clockwork, and has done for the past eight months. You send a reminder on Friday evening, they buy on Saturday, and the reminder gets credit. Would they have bought without it? Almost certainly.

This isn’t only a hunch. A study of 15 large advertising experiments at Facebook found that methods relying on observational data often failed to match what the randomised tests showed. Our guide to marketing attribution models covers how credit gets assigned; a holdout is the check on whether that credit is deserved.

Some Campaigns Cost More Than They Make

Discount campaigns are the usual suspects. A 15% off code sent to people who were already going to check out doesn’t create sales. It gives away margin, and on a ₹2,000 basket that’s ₹300 handed to someone who’d have paid full price on Thursday anyway. Without a holdout, that campaign looks like a winner in every report.

You Can Defend Your Budget With Evidence

Finance teams have learnt to discount marketing’s numbers, and frankly they’re often right to. “This journey added 900 orders that wouldn’t have happened otherwise” is a sentence a CFO can’t easily wave away. It also fits neatly into any framework for measuring marketing ROI.

It Tells You When to Send Less

Here’s a twist you might not expect. Sometimes the holdout buys just as much as the group you messaged, and unsubscribes far less. That’s a clear sign you’re over-messaging, which our piece on channel fatigue explores in more depth.

How to Calculate Lift From a Holdout Group

The maths is simpler than it sounds. You’ll need the conversion rate for each group, then three quick calculations.

1. Lift. Subtract the holdout’s conversion rate from the messaged group’s rate, then divide by the holdout’s rate.
2. Incremental orders. Multiply that difference in conversion rate by the number of people you messaged.
3. Incremental revenue. Multiply the incremental orders by your average order value.

Let’s run the Diwali example. You send an offer email to 90,000 customers and keep 10,000 back. In the messaged group, 3,600 people buy, which is 4.0%. In the holdout, 300 buy anyway, which is 3.0%.

So the lift is 1 point on a base of 3, or 33%. The incremental orders are 1% of 90,000, which is 900. At an average order value of ₹1,800, that’s ₹16.2 lakh of revenue the email really created. Your attribution report would have credited it with all 3,600 orders, or ₹64.8 lakh.

Neither number is wrong, exactly. They just answer different questions. But only one of them should decide next year’s budget.

How to Set Up a Holdout Group

You don’t need a data science team for this. Honest answer? A clear question, a random split and some patience will get you most of the way.

Step 1: Decide What You’re Trying to Prove

“Does our abandoned-cart journey earn its keep?” is a good question. “Is marketing working?” is too vague to answer. Write the question down before you touch any settings, along with the number you’ll judge it by, such as orders within 7 days.

Step 2: Choose a Campaign or a Global Holdout

Campaign holdouts test one send or one journey. A global holdout keeps a small group out of all promotional messages for a few months, so you can measure your whole CRM programme at once. Platforms like NVECTA let you run both kinds side by side. We’d start with campaign holdouts on your two or three biggest journeys, then add a global one once the team trusts the method.

Step 3: Size It Properly

Holding back 5% to 10% of the audience is the usual starting point. The smaller your list, the larger the share you’ll need. Why? Because individual buying behaviour is noisy. Economists Randall Lewis and Justin Rao found in 25 large retail field experiments that measuring advertising returns precisely can need more than 10 million person-weeks of data. Your email programme won’t need that much, but it’s a useful warning against judging a 2,000-person test after three days.

Step 4: Randomise, Then Lock It

Pick the held-back customers at random, never by hand. If someone quietly moves “high-value customers” out of the holdout because they don’t want to lose sales, the comparison is broken. Once the group is set, keep the same people in it for the whole test.

Step 5: Keep Essential Messages Flowing

Your holdout should only miss promotional messages; it shouldn’t miss anything a customer actually needs. Order confirmations, OTPs, payment reminders and KYC alerts must still reach everyone. Nobody should miss a delivery update because they landed in a test.

Step 6: Run It Long Enough, Then Read the Result

Give it time. At least one full buying cycle, which for a fashion brand might be two weeks, and for a mutual fund SIP nudge could easily stretch to a month or more depending on salary dates. Then compare the groups on the number you chose in Step 1, and check the gap is large enough that it isn’t chance.

Common Holdout Group Mistakes

These are the ones that quietly ruin results:

• Picking the holdout by hand, or excluding your best customers from it.
• Calling the result after a few days because the early numbers look exciting.
• Letting the holdout receive the same offer through another channel, like a WhatsApp blast sent by a different team.
• Measuring only clicks. The holdout can’t click a message it never got, so compare purchases or revenue instead.
• Running the test once and never again. Customer behaviour shifts with seasons, so repeat it each quarter.

What Metrics Should You Track With a Holdout Group?

Conversion rate alone can mislead you. Pair a few measures so you’re seeing the full picture:

• Incremental conversion rate. The messaged group’s rate minus the holdout’s. We’d put this at the top of every campaign report.
• Incremental revenue per user. Extra revenue divided by people messaged. It’s the fairest way to compare a big campaign with a small one.
• Lift percentage. Useful for comparing campaigns, but always show the absolute numbers next to it.
• Unsubscribe and complaint rate. A campaign can add sales while burning your list. Our guide to success and counter metrics explains why you need this pairing.
• Repeat purchase rate over 60 to 90 days. Some messages pull a sale forward without adding one. A longer window catches that.

How NVECTA Helps You Run Holdout Groups

NVECTA is an AI-powered customer data platform, and holdouts are built into how it sends cross-channel campaigns and automated journeys. In NVECTA, this feature is called Control Group. A Control Group is a randomly chosen share of your audience that NVECTA deliberately leaves out of a message, so you can see what those customers do on their own and compare it with everyone who received it.

Hold Back One Campaign at a Time

Planning a Diwali offer and want proof it’s worth the discount? You can give that one campaign its own Control Group, say 10% of the audience, and send to the other 90% as usual. It’s a quick way to settle the “would they have bought anyway?” debate from the start of this post, campaign by campaign.

Keep a Standing Holdout for Your Whole Programme

NVECTA also supports one Control Group across your messaging, so a small, fixed share of customers sits out of promotions for a period. That’s how you measure the whole CRM programme rather than one send at a time. It answers the CFO’s bigger question: what is all this messaging adding up to?

Test Your Automated Journeys Too

Abandoned-cart reminders, welcome series and win-back flows run every day, which makes them easy to forget about. Control Groups work inside Customer Journey Orchestration as well, so a cart-recovery journey can be held back for a share of users. Then you’ll know whether the reminder brings people back, or whether they’d have returned on their own.

Report the Lift in Rupees

Clicks can’t tell you anything here, since the held-back customers never got a message to click. NVECTA keeps a record of everyone held out and tracks purchases for both groups. That means you can compare orders and revenue side by side, and report the lift the way finance wants to see it.

Use the Same Approach on Every Channel

A customer who skips your email might still be nudged by a push notification or an SMS. NVECTA applies Control Groups across channels such as email, SMS and push, so your holdout stays clean. You aren’t stuck measuring one channel while another quietly reaches the same people.

Conclusion

So should every campaign have a holdout? Not every single one, but every campaign that costs real money or margin should. We’d start this week with your biggest discount campaign and your abandoned-cart journey, hold back 10% of each, and read the results after a full buying cycle.

The first result is often humbling. Still, it’s the most useful number your CRM team will ever put in front of finance. NVECTA’s Control Group makes running that test a setting you switch on, not a project you have to plan.

Want to see how it works on your own customer data? Book a demo with the NVECTA team.

Frequently Asked Questions

What Size Should a Holdout Group Be?

Usually 5% to 10% of the audience. With a list of 2 lakh customers, 10% gives you 20,000 people, which is plenty for most campaigns. With only 5,000 customers you’d need a bigger share, maybe 20%, or a longer test, before the difference means anything.

Is a Holdout Group the Same as a Control Group?

Yes, in practice. Both mean the customers who are deliberately left out so you can compare. Some tools say holdout, others say control group. In NVECTA, it’s called Control Group, and it works at campaign level or across your whole programme.

How Long Should a Holdout Test Run?

At least one full buying cycle. A food delivery app might get a clear answer in a week. An insurance renewal campaign could need six weeks, because people take longer to decide. Ending early is the most common reason holdout results can’t be trusted.

Doesn’t a Holdout Group Lose Sales?

A little, for a short while. If the campaign works, the held-back customers buy slightly less than they would have. That’s the price of knowing. Compare it with what you’d lose by spending a whole year’s budget on campaigns that turn out to add nothing.