How to share test subject lines like a newsletter with 2.5 million subscribers

Morning Brew hit 1,000,000 openings using this split test strategy. Also, learn how to do this in AWeber.

Stuck between two subject line ideas?

It's impossible to know which will give the best results without doing a split test – commonly referred to as an A / B test. A split test gives you real feedback on the subject line that attracted your audience more to open.

This is how the most successful marketers optimize their open rates.

Take Morning Brew, for example. Morning Brew is a popular newsletter with more than 2.5 million subscribers. Their daily newsletter features business and technical news of the day written for a business minded millennial audience.

In August 2020, Morning Brew reached a major milestone – 1,000,000 openings of its daily newsletter.

But how did the newsletter achieve this success?

Morning Brew's Secret to 1,000,000 Emails Opens

Morning Brew author Roby Howell spilled the beans on Twitter about how the popular newsletter tests the subject lines for every mail it sends.

This is how it works:

  1. The Morning Brew team tests 4 subject lines every morning. These subject lines are based on catchy, interesting sentences from the news of the day.
  2. Each of the 4 subject lines are sent to 3% of the audience at 5 a.m.
  3. The winning subject line will be sent to the remaining 88% of their subscribers an hour later.

This strategy allows you to quickly test your subject lines against a small segment of your audience and then send the "winner" to the remaining audience.

It's a smart way to get more openings for every email.

Would you like to follow in Morning Brew's footsteps to set up your own split tests like this one? Everything you need to know is here.

Psst … If you want to keep your split testing simple, start with a simple 50/50 split test. This will help you learn the ropes and try split tests before jumping into more advanced testing strategies like this one.

connected: The Ultimate Guide to AB Testing

How to Share Test Subject Lines Like Morning Brew

Testing subject lines like Morning Brew is pretty easy. However, there are a few steps that need to be taken outside of your email service provider to ensure that your results are statistically significant.

That means you need to make sure you have a large enough audience for the first part of your show. If your audience is too small for Groups A and B, the results will not be significant. If the percentage is too high, you could be missing out on the opportunity to send the winning subject line to more subscribers.

How to set the breakout percentage

To find out how many subscribers to get Version A and Version B of your newsletter, you need to look at your historical open rates and the size of your email list.

For this example, let's say your email list has 10,000 subscribers and you're seeing 20% ​​open rates historically.

Use the Optimizely Calculator to change the "Baseline Conversion Rate" to 20% to take into account the historical open rate and calculate the ideal audience size for A&B.

If groups A and B must each have 1,100 subscribers, it means that each group is 11% of your total audience.

In this case we are planning a split test of 11% / 11% / 78%.

connected: Your guide to writing the world's best email subject lines

Determination of a subject line winner

Once versions A and B have been sent, the next step is to determine the winner. The winner will then be sent to the remaining 78% of your audience.

To evaluate what would go to the remaining group, review the results of each email and score them on a split test calculator.

For example, let's say the first results are:

Group A: 1,100 emails sent, 278 emails opened, 21 bounces

Group B: 1,100 emails sent, 245 emails opened, 33 bounces

If you look at the numbers, it's obvious that Version A gives better results than Version B. However, you should check that there is enough data to determine if the results are statistically significant (and not random).

The Neil Patel AB test calculator allows you to capture and plug in the results. Be sure to subtract the number of bounces from your total emails sent. With roving bounces you can measure the actual opening rate. When an email bounces, there is no chance of opening.

Ideally, you want to aim for statistical significance above 90%. But sometimes you have to move quickly. In this case you should aim for statistical significance of 80% or more.

Now you can be sure that you are sending Version A to the remaining 78% of your audience

This way you can maximize the number of openings. In this example, we would get 234 more openings if the winning email went to the remaining 78% of the subscriber list.

We came to this conclusion by applying the open rate (or the conversion rate in Neil Patel's calculator above) to the remaining audience.

7800 * .26 = 2028

7800 * .23 = 1794

2028-1794 =234

Those 234 openings could mean the difference between additional sales.

How to share test subject lines like Morning Brew with AWeber

Want to set up a subject line split test like Morning Brew for your own email marketing? Follow the steps below and get better results from your email marketing

Note: Split tests in AWeber are available to AWeber Pro users. Update your account to unlock this feature or contact our Customer Solutions team.

Step 1: compose your emails

The first step in split testing is to make sure that two versions of your email have already been created in the message editor.

Since this is a split test for subject lines, all other variables in your email should be exactly the same.

Connected: Email split tests you can set up in 1 minute

Step 2: set up your split test

To create a split test, hover over the Messages tab and click Split Tests. Then click the green "Create" button.

Next, name your split test. It is helpful to include a descriptive language in the name to distinguish one split test from another. This is also a quick way to get the takeaway from each split test so you can apply the findings to other emails.

Then select a segment of your recipients (if you don't want to send the emails to your entire list).

And finally, define your split segments using the formula used above. And click "Save".

Now it's time to choose your designs. Make sure your designs match what you have defined in your subject line name.

You can then schedule your broadcast split test message for segment A and segment B.

Once you've sent your emails, you can set a specific waiting time to use the steps above to determine a “winner”.

You can then send this winning email to segment C.

Click here for more step-by-step instructions on how to test subject lines in AWeber.

Comments are closed.