I’m wondering if anyone else is having trouble with the split test division in an ABCD test scenario. I’ve been doing these tests for a while and it’s becoming more regular where, in the test period, ABC all deliver to the same number of contacts while D somehow gets shorted, yet that shortage isn’t identified until after the send takes place. Yesterday, for example, I checked the report progress twice during the test window. The first time, ABCD had all been sent to the same number of contacts. The second time, perhaps 45 minutes through the hour, D had dropped by 400 sends. This skews my results because we determine the winner by 1-hour open rate, and D will always have a higher open rate because it didn’t send to as many contacts. I have asked support many times and can not get a straight answer as to why…if I’m sending to an entire distribution of 15,000 but testing each version to 10%, why ABC send to 1,500 contacts but D does not. Somewhere there is a glitch, either in the sending or the reporting.
I don’t see this issue when doing simpler AB tests and haven’t noticed it on ABC tests.