How to set up an A/B test for your newsletters

In this article, you will learn how to set up an A/B test for your newsletter using systeme.io.

To get started, you will need the following:

• A systeme.io account
• A newsletter (How to send or schedule a newsletter)

Setting up an A/B test is one of the most effective ways to optimize your newsletter's performance. Rather than relying on assumptions, this method allows you to compare different versions of an email to identify what actually resonates with your audience.

Whether you want to improve your open rate, increase clicks, or maximize conversions, A/B testing helps you make decisions based on concrete data.

In this practical guide, you will discover how to easily create an A/B test on systeme.io, step-by-step, to optimize your campaigns and achieve better results.

1. Understanding the process

On systeme.io, A/B testing follows a simple and efficient workflow. You create two versions of your email: Variant A and Variant B, each with one specific difference (such as the subject line, content, or call to action).

When configuring your A/B test, you can choose between two different testing approaches:

• Optimize: This approach finds a winner for you. The two variants are sent to a sample of your audience, distributed according to a ratio you define (for example, 10% of your list, split equally between the two versions). After a defined waiting period, the platform analyzes the performance based on a key indicator (best open rate or best click rate) and automatically sends the winning variant to the rest of your contacts.
• Compare: This approach is built for pure analysis. Both variants are sent to 100% of your selected audience, split evenly (50/50). There is no waiting period and no automated winner selection, allowing you to directly evaluate the performance of each version side by side.

2. Preparing your A/B test

Before launching your A/B test, it is essential to prepare it properly to obtain reliable and actionable results.

The first step is to define a clear goal: do you want to improve your open rate, increase clicks, or optimize conversions? This goal will determine which element you should test.

Next, it is highly recommended to test only one element at a time. This could be:

• The email subject line
• The sender's name
• The email content
• The call-to-action (CTA) button

Testing multiple elements simultaneously makes results difficult to interpret and limits the relevance of your conclusions.

It is also important to choose a representative audience. The sample used for the test must be large and consistent enough to reflect the overall behavior of your contact list.

Finally, take the time to define the test duration and the success criteria (open rate or click rate). This allows you to clearly identify the winning variant and automate the sending process to the rest of your audience.

3. Creating and configuring your A/B test

Creating your newsletter and setting up an A/B test in systeme.io is intuitive and takes only a few steps.

Step 1: Create the initial newsletter

Under the Emails tab, choose Newsletters (1), then click Create (2).

After creating your newsletter, write your first version, which will be Variant A, by defining your subject line, content, and links.

Step 2: Create the variant

Once the first version is finalized, click Create an A/B test (3), then click Confirm (4) to create Variant B.

The newsletter is then duplicated identically. At the top of the page, you will need to configure the following settings:

1. A/B test settings

First, choose your test type. You have two options:

a. Optimize: Select this to let the system automatically find and send a winner.

If you choose Optimize, you must configure the following:

• A/B test duration (5): This is the period during which both email variants are sent to the initial sample of contacts. This duration should be long enough to gather a representative number of responses to reliably determine which variant performs best.

Tip: For standard campaigns, a duration of 48 to 72 hours is usually sufficient. Avoid choosing just a few hours; allow at least 24 hours so your contacts have time to open your emails.

• Email ratio (6): It defines the percentage of contacts who will receive Variants A and B. For example, if you choose 5% of contacts with the selected tag(s), those contacts will receive Variants A and B, while the remaining 95% of the list will receive the winning variant.

The ratio must be high enough to provide statistically significant results, but small enough to limit the impact if one variant performs poorly.

• Determining the winning variant (7): You can choose one of two criteria:
  • Best click rate: To measure actual engagement with your links.
  • Best open rate: To measure interest in the subject line and sender name.

b. Compare: Select this to test a 50/50 split without an automated winner.

If you choose Compare, you do not need to configure a duration, ratio, or winning criteria. The system will simply send both variants to 100% of your selected audience, split evenly (50/50), so you can compare the results side by side.

Important: Select your success criterion before starting the test. The winning variant will be the one that scores highest on that criterion and will be sent to the rest of your list.

2. Contacts settings

a. Send emails only to contacts who registered over X days ago (8): This option lets you limit the newsletter to contacts who registered more than the specified number of days ago. Enter the number of days to define how old a contact must be to receive the email

b. Include these tags (9): Select the tags that contacts must have to receive the newsletter. Only contacts with at least one of the selected tags will be included

c. Exclude these tags (10): Select the tags that should prevent contacts from receiving the newsletter. Contacts with any selected exclusion tag will not receive the email, even if they match the included tags

Scroll further down, and the newsletter copy will appear in the Variant B section (11), and you can go back to the first version by clicking on Variant A (12).

Modify this version to introduce the specific element you want to test (subject, CTA, content, etc.), ensuring you only change one element to maintain test reliability. Don't forget to save your changes.

4. Launching the campaign

Before launching your A/B test, a complete review is essential to ensure your campaign runs smoothly. Review the following essential settings:

• Target: Ensure the selected contact list is correct and matches your goal.
• Sender: Verify that the sender name and email address are clear and recognizable.
• Email subject lines: Confirm they are relevant and clearly differentiated between Variant A and B.
• Links: Test every link to avoid errors, incorrect redirects, or broken pages.
• Display: Preview your emails on both desktop and mobile to ensure an optimal user experience.

Once verified, you can proceed with:

• Save and run A/B test (13): If your campaign is ready, you can launch the A/B test immediately.
• Save and schedule (14): Plan the send for a strategic date and time to maximize engagement.

Before launching, you can cancel the test at any moment by clicking Cancel A/B test (15).

After launching, let the test run without intervention to ensure reliable results. The platform will automatically analyze performance and determine the winner.

5. Modifying and stopping an A/B test

Once an A/B test is launched, you can no longer modify the email content or the general test settings. You can, however, stop the A/B test at any time by clicking the three-dot menu, then clicking Stop A/B test (16).

If the variants have not yet been sent (neither the initial test versions nor the winning variant), the send will be automatically canceled.

6. Analyzing results and optimizing future campaigns

Once your A/B test is complete, it is essential to use the results to improve your long-term performance.

You can check the stats by clicking the 3-dots menu, then clicking Stats (17).

Start by analyzing the key indicators you defined, such as open or click rates. Identify the winning variant and note the performance gap between the two versions.

Beyond the simple result, take the time to understand why one version performed better:

• Was the subject line more engaging?
• Was the message clearer?
• Was the call-to-action more visible or convincing?

By adopting this logic of continuous improvement, each campaign becomes more effective than the previous one.

7. Resending the newsletter

To improve your overall open rate, you can easily resend your A/B test newsletter to subscribers who didn't open it the first time.

Click the three dots menu (...) next to your completed A/B test and select Resend to contacts who did not open (18).

Schedule it, and click Duplicate and schedule (19).

Important:

• This action will only resend the winning variant to contacts who didn't open it.
• Contacts who received the losing variant during the initial test are automatically excluded from the resend.

In summary: Test, analyze, understand, and optimize. This is the key to sustainably maximizing the impact of your newsletters.