Design your video email A/B splits the right way: HTML5 video’s dirty little secret
Recently, I’ve gone on the record during a series of webinars about a new trend many email marketers are seeing when “A/B testing” clickthrough rates for video email campaigns featuring in-email video. More particularly, I’m referring to when marketers try to measure clickthrough rates on a list has been split 50/50, with 1/2 of the list receiving “the in-email video version,” and the other 1/2 receiving a static image thumbnail, where both are intended to drive traffic to video on a landing page. The trend, it seems, is that more and more email marketers are seeing clickthrough rates decline on the segment that includes video in the email.
What could be happening?
First, it’s important to note that when trying to send video IN the email, there are only two ways to do it: use an animated .GIF video (which plays silently in the email body), or use HTML5 video, which allows the entire video, with audio, to play within the email body. Neither of these formats are supported everywhere, but there are techniques and technology like my company’s video email express software that can optimize the use of both, for example by displaying an animated .GIF video in the case where HTML5 video isn’t supported by the mail client. The final method is to use a plain static image thumbnail in the email, linked to a landing page which features a video player.
How do the different methods impact A/B tests?
As more webmail and mobile mail clients support HTML5 video, more video viewers will see HTML5 video in email as opposed to an animated .GIF when you’re using a fallback method. Probably the best illustration (since it is fresh) is Hotmail’s surprise rollout for HTML5 video support in summer of 2011. Since Hotmail previously only supported animated .GIF videos, and with Hotmail comprising, on average, 12% of the overall mail client audience (this will vary, and could vary greatly, based on your list), then if you ran a 50/50 split, you would not be measuring up to 12% of the animated .GIF clicks for the “video plays” because those clicks are going to the HTML5 video version.
HTML5 video’s dirty little secret
No big deal, you say? You can just have the 50/50 split measure clicks on the animated .GIF, add that to the clicks for the HTML5 video version, and your test is valid – you’re measuring what you want: video in email v. no video in email – all good!
Keep it apples to apples
The lesson here: if you’re trying to measure video in email v. static image in email, and you want to measure the lift of video based on a clickthrough rate metric, then make sure you are not using HTML5 video in the split of your list that includes the video version in the email. Only clicks on animated .GIF videos are measurable.
Of course, measurement is still important, and most email marketers probably prefer to deliver HTML5 video as opposed to animated .GIF video. If this sounds like you, then you will need to use another metric outside of clickthrough rate to get the data you crave. Other metrics that could be used would include:
- At what rate did people visit the site after receiving the email for each segment?
- Did time on site increase for one segment or another?
- Did one segment demonstrate a higher conversion rate or higher average order value?
Regardless of the metric you choose, if you want to get full HTML5 video in front of as many people as possible in the email body itself, then you will need to live with HTML5 video’s dirty little secret.