You’re probably familiar with the saying “if at first you don’t succeed, try, try again.” That’s A/B testing in a nutshell – evaluating two versions of something like an email subject line to determine which one is more successful. 

A/B testing lets you choose the elements of your email that work and tweak the ones that don’t. That could look like choosing the subject line that generates a higher open rate, for example. Every element, from headlines and CTAs to graphics and email templates, has room for improvement. Skip the guesswork and use data to understand exactly what content your customers respond to. 

Need a refresher on the benefits of A/B testing? Start with this guide, then follow these steps to run your first test! 

1. Decide Which Elements to A/B Test

There are many elements in an email campaign and, therefore, many possible combinations to test. To effectively compare campaigns, stick to one element at a time to get the most accurate results. For example, play around with various subject lines before moving onto message design. Here are some email elements you should consider A/B testing.

• Subject line: Its prominence in the inbox makes it a big influencer of open rates. Play around with length (60-70 characters is optimal) and word order (“Here’s a discount code for 20% off” vs. “Get 20% off with this discount code”).
  • Pro tip: Learn how to make your subject lines catchy – and clickable! 

• Content: What offer should you include? Will testimonials improve the conversion rate? Test different pieces of content until you find one that resonates the most with your readers.
• Length: When it comes to email, short and to the point is usually best, but you can use A/B testing to find that sweet spot.
• Send time: People open their emails at different times of day and even different days of the week, like on weekdays when they’re checking for work emails. Figure out when your target audience is scrolling through their inbox for greater success. (Use these effective send times as a starting point.)
• Sender name: In many cases, your emails will see a higher open rate if they come from a sender with a real name (e.g. [email protected]). Learn more about email personalization. 

• Message design, layout, and graphics: Images are a powerful way to get your message across, especially since people are more likely to look at visuals over text.
• CTAs: Should you use buttons or hyperlink text? What should the call to action say? Get answers with A/B testing.
  • Pro tip: Campaign Monitor found that using a button instead of a text link resulted in a 27% increase in click-throughs. 

2. Prioritize Tests: Small Changes = Great Improvements

When choosing what to test first, think about which part of the conversion process needs improvement. For example, if you have a low open rate, start by testing subject lines. If your unsubscribe rate is high, focus on creating new content that better engages readers. 

Think about which tests are likely to have the biggest impact and which ones you have the most confidence in. Some changes will be more obvious, such as switching a hyperlink to a button. Others aren’t as clear, like determining which email template will be best received. Start with tests for which you’ve already formed a hypothesis, as this will help you learn more from the process. Another factor to keep in mind is ease of implementation. Changing a subject line requires minimal effort, but redesigning email graphics is a bigger undertaking.

It can feel tempting to tweak a few elements at once. Just keep in mind that A/B testing is all about making small changes that lead to big improvements. Take Hubspot for example: They tested using a personalized sender name as opposed to a company name and generated 131 new leads from simply changing one small part of their emails.

3. Set the Testing Pool & Duration

Once you’ve figured out what you’re testing and the specific changes you’ll make, you need to consider your testing pool and the duration of the test. 

The size of your testing pool depends on the size of your email list and how confident you are in the result. The most common confidence level used in A/B testing is 95%, meaning 95 out of 100 future emails you send should deliver the expected result. Use a sample size calculator to generate your ideal sample size. All you need to do is input the total size of your email list, your confidence level, and the margin of error you want (3% is common). 

 

With A/B testing, it pays to be patient. Run your test for at least 24 hours to get significant results. 

4. Measure Your Results 

Are you familiar with the most important email marketing benchmarks? These metrics are key to understanding the results of A/B testing, as they can correlate to specific email elements and give you insight into what’s working – and what isn’t. Here’s an idea of which elements affect which metrics. 

• Open rate: Subject lines, send time, sender name
• Click-through rate: CTAs, content, template, length
• Conversion rate: CTAs, content, offers
• Unsubscribe rate: Content, length, design/template, email frequency

Ready to give A/B testing a try? Think of it as a way to get a leg up on other businesses in your industry. After all, only 31% of companies do A/B testing in their email marketing (Yesware). 

To get started, see what A/B testing services your email service provider offers. Many of these platforms, like Mailchimp, have built-in tools that make it easy to test various elements, set a testing duration, and automatically send the winning version to the rest of your email list. Get in touch with your email marketing platform to see what they offer. 

Looking for more email tips? Explore our email marketing guides.