A/B Email Testing

From Net-Results Support

A/B testing is commonly used to gauge the performance of different subject lines, content sections, or other variations in your marketing emails. Net-Results makes running A/B tests simple and does the heavy lifting for you.

How A/B Email Testing Works in Net-Results

Setting up an A/B email test is pretty straightforward. Things start off exactly the same as in drip and nurture campaigns with the Settings and Participants tabs. The differences are all found on the Actions tab - where all the action takes place! ;)

We'll send your emails, "Email A" and "Email B", to an equal number of recipients.

  • These recipients are selected at random from your campaign's participants
  • You are able to choose what percentage of your campaign's recipients take part in the test
  • Example: If you have 10,000 total participants in your campaign and you choose to have 20% of them (2,000 total) participate in the A/B test, then 1,000 people will receive Email A and 1,000 will receive Email B

After a configurable wait period, the email that performed better in the test (the "Winner") will be sent to the rest of your recipients.

  • You may choose to have Net-Results wait for a period of 1 hour - and up to 7 days - before the Winner is determined
  • How the winning email is chosen is up to you: the email with more opens or the email with more clicks
  • Sticking with our example where your test went out to 20% of your campaign's participants, 80% of them, or 8,000 people, will receive the winning email when the wait period is over

Once the Winner is sent your campaign is over and is automatically retired.

Then it's time to visit the Campaign Performance Dashboard to check out your results!

You can also view the results while the campaign is still in its wait period, you just won't see results for the Winner until it has been sent.

A B Email Testing.png

What Should You Test?

You can test small changes or entirely different emails with Net-Results. Here are some of the common elements marketers like to test...

  • Specific offers (Example: “Save 20%” vs. “Get free shipping”)
  • CTAs (calls to action)
  • Subject lines
  • Testimonials to include (or whether to include them at all)
  • Email layout (placement of elements, single column vs. two column, etc.)
  • Personalization (Example: “Mr. Johns” vs. “John” vs. "Hello")
  • Body text
  • Headlines
  • Closing text
  • Images

Building Your Emails

You can use any of your existing Net-Results Emails in your A/B test campaigns. There's nothing special you need to do in your emails to make them ready for A/B testing. Simply create the emails you would like to A/B test just as you would with any other email.

Tip: If the variations between your A and B versions will be minor, get your "Email A" version built first, then use the "Save As" button to make a copy. Change the aspects of the email you'd like to test and you can use this as your "Email B".

Both Drag & Drop and Custom HTML Emails may be used in your A/B Email Test campaigns.

Configuring your A/B Email Test Campaign

The Settings and Participants Tabs

Both of these tabs are much like the Nurture/Workflow or Drip Campaigns. Configure these two tabs as you normally would. Click here for more information on the configuration options for these two tabs.

The Actions Tab

  • Choose Your Emails - You will be prompted to select Email A and Email B for your test.
    The drop-down list of emails is sorted by the most recently modified emails. You can type in the field to quickly find any email you're looking for.

  • Set Your Test Size - Using the slider, choose the percentage of recipients that you'd like to receive the A and B emails (you're also choosing the percentage of recipients that will receive the "Winner" email by dragging the slider).

    Note: The statisticians tell us that, for a statistically valid test, your test should include at least 2,000 participants. Net-Results does not set a minimum number of recipients and will still run your campaign with a smaller set of test recipients.

  • Set Your Wait Period - Set how long you'd like Net-Results to wait before determining which of the two emails is the winner.

  • Choosing a Winner: Opens or Clicks - Choose the metric you'd like Net-Results to use to determine the winning email. We'll count the total number of opens or clicks at end of your wait period. Whichever email has more is then the "Winner". That winning email is then sent to all the remaining participants in your campaign - those that did not receive email A or B initially.

Important: In the event of a tie between Email A and Email B, Email A will be declared the winner.

Additional Options

  • Send a Notification: Net-Results can send a notification via email (or webhook) to recipients of your choosing whenever:
    • Your Winner is determined
    • Someone opens your email
    • Someone clicks a link in your email
    • Participants qualify for your campaign

  • Lead Scoring: You may also have Net-Results adjust a Lead Score based on:
    • Email opens
    • Email clicks