How to A/B test an email subject line

A subject-line A/B test sends two subject variants to small subsets of your audience, measures performance, and sends the winning subject to the remainder. Webanto runs the entire test on the same durable workflow engine that powers automations.

1. 1

   Pick a meaningful variant

   Don't test trivial copy changes (a comma here vs. there). Test real hypotheses: question vs. statement, short vs. long, personalised name vs. category. The lift you can detect statistically grows with how different the variants are.

2. 2

   Open the campaign and enable A/B testing

   In your campaign draft, click 'Enable A/B test'. Enter both subject variants and select 'Subject line' as the variable being tested.

3. 3

   Choose sample size

   10-15 percent per variant is a reasonable default for a 5,000+ list. Smaller samples produce more noise; larger samples reduce the winning subject's lift on the remaining audience.

4. 4

   Choose the winning metric

   Open rate is fine for a subject-line test because the only thing the recipient sees before opening is the subject. Don't pick click-through rate here — that's measuring body content, not subject.

5. 5

   Set the test duration

   4 hours minimum, 24 hours typical. Most opens happen in the first few hours, so longer tests don't add much signal. Avoid setting it to less than 2 hours unless your list is very large.

6. 6

   Schedule the send

   Schedule the campaign. The cron-based winner declaration runs automatically when the test duration elapses; you don't need to be online.

7. 7

   Review the results

   After the test, Webanto shows you the open rate of each variant and which one was sent to the remainder. If the lift is under 5 percent it's likely within statistical noise; treat it as 'no difference'.

8. 8

   Document the result

   Add the result to a running log of subject-line tests. Patterns emerge over time: which themes consistently win for your audience, which don't.