I ran an A/B test campaign (think I chose open rate as the metric, 20% to each option) with these numbers:
Losing email:
- 94 contacts
- 56.38% open rate (53)
-13.83% click rate (13)
Winning email:
- 337 contacts
- 36.34% open rate (137)
- 4.77% click rate (18)
How does AC determine the winner? Does it the full 20% for each test before sending the additional 60%? I’m just having a hard time trying to figure out how it chose the crappier one as the winner.
If it does send the the full 20% first, then I suppose the “winning” email could have had a better open rate when it was at 94 sends. But then it’s open rate would have had to have dropped off significantly.
Either way, I think the feature is really important and could be improved a lot.
It’d be better if we could choose a % threshold, say 90%, so the campaign will send them out evenly until it hits a 90% significance and then send the rest as the winner.
Also, the a/b testing reporting interface sucks / is non-existent. It’d be cool to see the results side by side rather than having to go back and forth between the split results. It would also be cool to compare the numbers at the decision point.
This is kind of a critical post, but I love you guys in general.
I just tried this again this morning, and then time I know for a fact it picked the email with the LOWER open rate.
Before it sent out the last 60%, one of the emails has like a 5% open rate and the other was closer to 8%, and it sent the rest to the email with the 5% open rate.
That doesn’t seem right?
Hi there, Amy from ActiveCampaign’s Customer Experience Team. We’re just refreshing our forum page and I wanted to add some help here for anyone who comes across this page/question/post.
Per this help article, you have the power to determine (and send using) the winner. This option sends your campaign variations to different portions of your list. After a set timeframe, the best variation will be sent to the remaining contacts on your list.
There are three steps you need to take to set this up:
- Choose how the winning message should be determined.
You can choose from the best click rate or the best open rate.
- Determine when the winning message should be sent.
The default time period is set to two days. Most email engagement happens during this time frame.
- Set the email ratios.
You can determine what percentage of your list should receive each email variation. In most cases, you’ll want these numbers to be equal. The last percentage, “Winning email,” shows you the percentage of remaining contacts who will receive the email variation that performed the best.
As this inquiry was posted in 2016, we’re unable to investigate why the system chose the email it did as the winner, though for anyone reading this now, struggling with the same issue, kindly contact our team via help@activecampaign.com. Kindly provide us with a description of the issue and attach the direct link to the campaign you are inquiring about. The direct link will look something like https://example.activehosted.com/campaign/id/overview > sample link, please don’t click.
Any questions or additional ideas, thread them below!