A/B Testing Email Subject Lines: What Actually Moves the Needle

Most subject line tests compare two completely different lines — different length, different tone, different structure all at once — which makes the winner impossible to learn anything specific from. A real test isolates one variable at a time.

Test one variable, not the whole line

Comparing "Save 20% Today" against "Why Your Inbox Is Lying to You" tells you which line won, not why. Testing the same core message with and without a number, or with and without urgency language, isolates the actual lever that moved the result.

Sample size matters more than people assume

A test run on a list of 200 people can show a 5-point swing in open rate purely from noise. Smaller lists need either a longer test window or accepting that the result is directional, not conclusive — treating a small-sample win as proven can send you chasing a pattern that doesn't actually exist.

Open rate isn't the only metric that matters

A subject line that wins on open rate but loses on click-through or conversion didn't actually win — it just got more people to open an email that then underdelivered on what the subject line promised. Track the metric you actually care about, not just the easiest one to measure.

Personalization tests need a real baseline

Testing a name-personalized subject line against a generic one only tells you something if the rest of the email and audience segment are identical. Confounding variables — different send time, different list segment — quietly invalidate a lot of subject line tests that look clean on the surface.

Build a running log, not a one-off test

A single test result is a data point, not a rule. Logging results across multiple campaigns over time — which structures, lengths, and tones actually correlate with better performance for your specific list — produces a far more reliable picture than any single A/B test alone.