Of course I’m not suggesting for a moment that your own testing efforts aren’t doomed as well, but you’re here now, so there is hope. 🙂

I make my living advising Fortune 500 companies on what to test on their Web sites and how to get the best results from their A/B and multivariate tests. My week is filled with meetings where I get to hear internal debates between executives and design teams about so-called best practices and the latest thinking on button design.

As you can imagine, I hear some real zingers in those meetings about A/B testing.

Here are a few examples that I’ve heard just this past week:

  • “Test everything.”
  • “Version B was the winner, but we’re not sure why.”
  • “We rarely get a win.”

I’d love to speak up in those meetings and tell the know-it-alls that they’re mostly full of shit when it comes to experimenting on their Web sites (I know this because I’ve facilitated hundreds of A/B tests, many of which bombed and forced me to re-examine my approach to testing). But if you’ve ever done consulting for any length of time, you know that it’s best to take a softer approach. And so I gently try to steer them in a different direction in their thinking. And try and try and try.

Problem is, taking the softer approach weighs on your soul after awhile. So I’ve decided to reclaim my inner peace on this blog — with the occasional rant about clients who shall remain nameless — and start out by sharing what I really believe with you:

If you find yourself repeating the statements above, your testing program is doomed.

“Test everything” is bad advice and here’s why

Plain and simple, this is lousy advice that I see shared all over the Web. The intent makes sense, perhaps, but the execution is wrong. It’d be better worded, “Every area of the user experience should be considered for testing” (with the operative word being considered).

I don’t mean to pick on anyone specifically, but I’m hoping that the nice folks over at Plenty of Fish have thick skin to match their thick wallets (just kidding, guys).

On April 3rd, ‘pofben’ penned an intriguing post at ads.pof.com, titled “Throw Everything You Know About Ads Out The Window“. Ben surmises that because a hand-drawn image of a car outperformed a more polished version of an advertisement for “Need For Speed World” — in terms of clickthroughs — that it’s worth testing everything.

I get it: If an image that looks like it was drawn by a 6 year-old can produce lift over something that took some time to put together, what other crazy opportunities might you be missing on your site? It seems somewhat plausible that everything is, in fact, worth testing.

That post generated a lot of comments. And many of the commenters correctly identified another, better (IMO) explanation for what happened. The commenters feel there is a novelty effect at work here. The low fidelity image got people curious because it’s not something they see everyday on the Web. If this kind of image started to show up everywhere, the curiosity factor goes away — along with the clicks.

The problem here was the conclusion Ben came to… that even the craziest ideas are worth testing. Had he included a conversion metric beyond the initial ad clickthrough, it’s very likely he would have reached a different conclusion — that the unconventional ad design was really only responsible for a curiosity click (minimal business value).

Ben’s not the first person to deliver this test everything advice, either. It’s a general notion floating around that you should test everything, because you just never know. It’s true; you never know. But you also don’t have unlimited resources to conduct A/B tests, do you?

After all, there are internal costs to running tests… time to develop the creative… time to code the variations… time to evaluate the results. There is an opportunity cost for every test, as you could always be testing something else. And I’ve personally noticed that there is a psychological toll for every test that doesn’t produce lift – in other words, deliver 7 or 8 losing tests in a row and see how your testing manager and creative team feel.

And when you open things up to testing everything, it feels pretty daunting, no? How many elements does your Web site include? Coming up with initial ideas may not be too difficult, but your selections for testing end up looking pretty random (BOOM, you’re doomed!).

What I’ve found in my experience is that this advice to test everything is usually given by folks who don’t really know where to start. That’s okay though – we’ll let them struggle (in other words, just politely ignore their advice). Because luckily, there is a place to start.

Alternative to “Test everything”: “Test where there’s evidence of change required”

By “change” I mean user behavior change.

A longer form of this statement could be “Test where you see evidence of user behavior that you’d like to modify, and then run an experiment to drive that behavior change.” See why I went with the abbreviated version?

Breaking that statement down further…

In order to optimize your site with a level of predictability, you have to:

  1. Understand [the undesirable] visitor behavior
  2. Create hypotheses around why the behavior is occurring
  3. Attempt to change the behavior via an alternate experience

I’ll touch on all three steps, but in reality there is enough material on these topics to fill a small book.

1. Understand [undesirable] visitor behavior

This is the easiest part of the process. To understand your site’s visitor behavior, your primary source of information is Web analytics data. Your analytics reports can give you a good glimpse into problematic user behavior on a particular page (form completion, bounce rate, exit rate) or across a series of pages (product feature usage, sales funnel drop-off).

The best places to start looking are your busiest pages (i.e. heaviest traffic) and those pages along your primary conversion path. You can branch out from there later.

2. Create hypotheses around why the behavior is occurring

There is a lot of stuff your analytics can’t tell you. For example, they won’t tell you what’s accounting for your home page bounce rate. They won’t tell you why people are abandoning your payment page. Any time you want to understand why a certain behavior is occurring, you’ll have to take an educated guess or actually observe someone using your site.

Again, Web analytics can point to a problem, but you really need to watch users interact with your site to pinpoint the issue(s) and understand what’s behind the behavior.

A fallback to directly observing people using your site is performing a heuristic review, but they tend to be a lot less fun and less effective unless you’re experienced at connecting heuristics to your copy & design elements.

One of the things you’ll notice while watching people using your site is that some problems are pretty easy to spot – like people struggling to find a link they need. But there are also problems you won’t be able to spot, because the issues are occurring in the minds of your visitors. To uncover those types of issues, you need to talk to your test subjects.

3. Attempt to change the behavior via alternate experience

Once you know where a problem exists and have some ideas as to why, you need to figure out how to modify the visitors’ behavior. Your options fall into two main categories:

a) If you see evidence that the obstacles your visitors are experiencing are interaction based (i.e., related to the usability of your pages), then you should fix the usability problems that are creating the undesirable behavior.

b) If you see evidence that the obstacles your visitors are experiencing are psychologically based (i.e., related to their motivations, objections, fears, anxiety), then you should change your messages or placement of those messages to address the underlying resistance.

Fixing usability issues is something you can learn to do by reading about user experience design and best practices. If a button is not visible, set up your test to make it more prominent.

If your checkout process doesn’t include a progress indicator, test a design revision that provides contextual feedback.

If you’re using a two-column design on your home page, set up a test using a single column to provide a better eye path for your visitors.

Addressing visitors’ psychological resistance to your product or service requires a little more work. You’re not going to glean much from quantitative data. No, to get inside the minds of your visitors, you’ll need qualitative data, and to get that, you’re going to have to talk to a handful of them. Egad! (At least that’s how I used to feel – before I realized just how massive a conversion lift I could achieve by tapping into visitor motivations and emotions.)

An example to bring it home

1. Identify the undesirable visitor behavior

Let’s imagine that your home page has several calls to action, one of which leads to your free 30-day product trial. By examining your home page clicks in Google Analytics, you believe there are too few people taking you up on the free trial offer. To reach your desired monthly goal for free trials, you need to increase home page clickthroughs by at least 30%.

2. Create hypotheses around why more people aren’t clicking your main CTA

Let’s assume that you have no friends to observe as they attempt to navigate your home page. 🙂 Let’s also assume that you have no money for enlisting the help of UserTesting.com or KISSinsights (no affiliations here) to better understand the behavior you find problematic.

This really leaves you with some educated guesses, any of which could be correct. So you create a list of your assumptions:

  • The main CTA is not sufficiently visible on the page
  • The main CTA is competing for attention with the other CTAs on the page
  • Visitors are not ready to give our software a try because they need to know more
  • Visitors are not sure what they’ll need to do in order to gain access to our trial
  • Visitors may believe they’re going to get hounded by us after the trial period is over

Looks like a decent list of reasonable assumptions. Now it’s time to validate them.

3. Run a few tests to validate your assumptions and change visitor behavior

Now line up a test idea for each assumption, respectively:

  • Test a more prominent button (e.g., color, placement, surrounding whitespace).
  • Remove or subordinate the secondary CTAs on the page to focus visitors’ attention
  • Test a version of the page where you provide a more detailed explanation of your offering before you ask people to click
  • Test a version of the button copy that lets them know where it leads and why they should be excited about what’s on the other side
  • Assure visitors that once the trial period is over, communications also end (via copy placed near the button)

Keep in mind that these should remain 5 separate tests, each one giving you a single, useful insight into how your visitors think. Win or lose, every one of the above tests will reveal something about your visitors’ mindset!

And besides generating lift for your home page CTA clicks, you can apply what you’ve learned on other pages or in other marketing initiatives – skipping right over the first two steps and into deeper validation of what you learned on your home page CTA tests. Milk what you’ve learned before moving on to learning more.

Edit (based on feedback from this post’s early reviewers): If you’re start-up founder, I totally get that you have lots of other stuff to do besides analyzing visitor behavior and trying to modify it. But if you invest some time into learning this discipline and you get it right, you’ll be in a position to out-market and out-convert your competition — without having to out-spend them. Remember, their testing program is already doomed. 🙂