A/B Testing

Which of these two buttons is more appealing to you?

start-here-salt-btn start-here-asa-btn

Personally, I wouldn’t click either one. The problem is, my preferences only tell me what I like. They don’t tell me much about my audience.


I like to think I have excellent taste and great empathy for my audience. Even if I’m not flattering myself, those things only help so much.  Yes, there are basic rules you can follow to build clear and effective communications, just like there are user-interface guidelines.

But ultimately, personal taste is just a focus group of one, and heuristics and rules of thumb are things that worked before but might not work again. If you want to know what works in a specific situation at a specific time, you have to test it.

When someone suggested it to me, I thought adding a button telling people where to click in an email was a terrible idea. I may have said “if you can’t figure out how to click a text link, we can’t help you anyway.” But we tried it anyway, and emails with buttons did significantly better than the ones with plain text links, and now we use them in all kinds of places.

Recently we’ve been testing subject lines for a series of emails that go to a group of student loan borrowers recovering from default. I had been reading about loss aversion and thought it would be a good idea to try that angle. Among the tested subject lines were these two:

Don’t let your student loan success slip away!

Hold on to your student loan success!

I thought that the ominous warning about danger might be a stronger appeal, but it wasn’t. We got a significantly higher open rate for the second subject line.

Our team has also been testing the emotional tenor of our messaging. Our original messages had aimed to provide plain, clear facts about student loan payment plans, in accordance with our brand goal of being reliable and trustworthy. However, a just-the-facts approach wasn’t driving satisfactory engagement numbers. By highlighting the psychological benefit of not having to worry about payments, we have increased clickthrough rates on our messages, and encouraged more student loan borrowers to investigate and sign up for affordable payment plans.

You can’t test everything, of course. There isn’t time, and there aren’t always enough people in your audience to get a significant result. But if you’re not testing at all, you quite literally don’t know what you’re missing.