Growth Marketing

Testing Ain’t ALL That


Who would dare to blaspheme what is arguably the ultimate technique for optimizing one’s website copy? Surely Joanna wouldn’t let a statement like “testing ain’t all that” simply slip through in a guest blog post… And it couldn’t come from someone who earns a living helping clients run a successful testing program… could it?

It could. And it does.

Split testing is often lauded as a panacea for website optimization.

I mean, you get to (cue sinister Transylvanian voice) experiment on your visitors… test your theories… and ultimately see the impact of your copy changes on visitor behavior – using statistics to confirm that the difference is real!

Yes, testing is amazing. And no, I’m not going to try to talk you out of testing completely.

But testing ain’t all that.

Instead, stand by as I make a case for an equally powerful – and typically much faster technique – for optimizing your website copy…

Fact Is, Testing Has Many Limitations

Testing is a terrific way to validate assumptions you may have about your visitors… but it can also take awhile to get there. See, testing requires traffic. It requires patience (and no peeking at the results!). It requires a testing tool and proper set-up of that tool. And it often requires careful interpretation of the results…

In my testing travels, I’ve seen startups and large companies run tests to inform their visitor theory (in other words, to learn what makes their visitors tick).

For example, you might test a new headline that focuses on your most popular product feature. The results of the test can tell you whether or not that feature (hopefully worded as a benefit!) resonates with your target audience…

And if you have a bunch of cool product features, you might choose to run a headline test using 5 variations… with each headline focusing on a single benefit of your offering.

Whether you run 5 headline variations as an A/B/…/F test, or run 5 separate A/B tests sequentially, it’s a perfectly valid way to learn about your website visitors.

But here’s the rub: Unless you have sufficient traffic to support testing those 5 headline variations, you may be watching the data roll in for quite some time. Weeks or even months may pass – all in the pursuit of informing your theory about what product features resonate best with prospective customers.

I Don’t Want To Give You The Wrong Impression Here

There is absolutely nothing wrong with split testing to learn about your visitors, but it’s typically not the most efficient way to get inside your visitors’ heads.

There are other ways to learn what will resonate with prospective customers. Ways that don’t require a ton of traffic or time to execute. In fact, you can start tomorrow and have answers within days.

To find out what will resonate with your website visitors, look to your existing customers. Learning what led to your customers’ decision to buy from you will inform your visitor theory more quickly (and I’d argue more reliably) than any series of split tests. More specifically, you want to understand why your customers chose you over all other competing solutions (or chose you over doing nothing at all).

How can you learn from your customers?

Interview Them
(It’s Not as Terrifying as It Sounds)

There are a couple of ways to go about interviewing, neither of which requires you to be a trained interviewer. You just need to know what questions to ask.

OPTION 1: Interview Your Customers Directly

Whether it happens on the phone or face to face, this is a fun and effective way to inform your visitor theory. Like usability testing, where as few as 5 participants can reveal 80% of the problems on a website, you can learn a TON about your customers from a handful of discussions… and then apply what you learned to persuade your website visitors that they too should buy from you (like the rhyme?).

OPTION 2: Interview Your Customers Indirectly

By indirectly, I mean interviewing your customers without speaking to them. The most common approach is an email-driven survey. But I hate the word ”survey”. Surveys are cold and impersonal, and they have abysmal response rates. Nobody wants to take a survey, but people like to be interviewed. So call it an “interview” when you invite your customers to respond.

Tweet that easy-peasy and useful copy tip

Interviewing Is Easier & Faster Than A/B Testing

(Especially with These Qs to Guide You)

Compare the interview approach to split testing… where each test should be designed to reveal 1 thing about your visitors (remember that split tests are univariate, meaning you test a single variable per test)… a well-designed customer interview can generate a ton of useful insight, all from a single exchange!

Below is a sample 12-question interview that has worked well for us. We’re sharing it in the hope that it’ll kick-start your decision to give this approach a try.

You can use it as is, or modify it to suit your business. Try it out with some customers on Skype, or turn it into an email interview using your favorite survey (ugh!) tool.


1. Thank you for choosing us! How long have you been using PRODUCT/SERVICE?

  • I’m on a free trial
  • Less than 2 months
  • 3-6 months
  • 7-12 months
  • 1-2 years
  • More than 2 years

2. How long ago did you first realize you needed a solution similar to what you’ve purchased from us? What was going on in your world that caused you to go looking for a solution?

3. More specifically, what problem were you trying to solve?

4. After realizing you needed solution, what did you do or where did you go to begin the process of getting it?

5. How did you first hear about us?

  • A colleague/co-worker
  • A friend/family member
  • A search engine
  • One of our partners
  • Other:

6. What are the top three (3) reasons you chose us over others like us?




7. What one (1) feature were you MOST EXCITED to use when you got started with PRODUCT/SERVICE?

  • Feature 1
  • Feature 2
  • Etc.

8. What one (1) feature do you find yourself USING MOST?

  • Feature 1
  • Feature 2
  • Etc.

9. What 3 words would you use to describe the benefits of PRODUCT/SERVICE to your friends or co-workers?

10. Have you ever used one or more of our competitors?

  • Yes
  • No

11. If you don’t mind, please tell us the name(s) of the competitor(s) you’ve used.

12. What do you think we can learn from our competitors?


That’s it! If you use the questions in a phone interview, you can simply ask the questions – don’t worry about reading out the full set of responses on questions 1, 5, 7, and 8.

Shoot to complete 5-8 direct interviews or 12-15 email interviews. You’ll need more email interview completions because the open-ended responses will be less thorough.

Once you’ve completed a series of interviews, look for themes in your customers’ responses. Themes will give you a level of confidence in turning the interview responses into new website copy. Also look for language that you can “steal” from your customers and use directly on your site.

If you like – remember, we still think testing has its place in your optimization efforts – you can always validate what you’ve learned from your interviews in a split test. 🙂

The trick (and time saver) with this approach is this:

You’re not experimenting your way to that learning.
You’ve taken a shortcut by engaging with your customers.

Think I’m crazy… like a fox? What kinds of questions or techniques have you used to learn what makes your visitors tick? Leave your advice below!

About the author

Lance Jones

Lance Jones is a CRO and SaaS growth expert.

  • In a perfect world, we’d all be able to interview thousands of clients and then test each element of our copy. (And I could eat chocolate for breakfast without getting dirty looks…) Unfortunately, that’s not always possible for small companies with very little traffic. I read CopyHackers and think, “Man, I’d love to test copy for my clients” but it’s a rare case when they have enough traffic to get solid results. Your proposed solution is straightforward and simple. Swiping the survey now. Thank you!

  • Mark @ Make Them Click

    Testing is what gives us Hollywood movies, but it never gives us Tarantino or the Coen Brothers.

  • Spook SEO

    Instead of split testing, simply interviewing your customers or perhaps sending a very short questionaire can actually make things easier for you.

    Through this, you no longer have to spend too much $ or wait several days. You can actually get the information in minutes.

  • Steven Macdonald

    Hey Lance, great post!

    It’s quite funny that we have blogged about similar experiences with A/B testing. My post went live on a few days back, that reads just likes yours:

  • I have to say, I positively LOVE the reframe of “survey” to “interview”.

    I believe that talking one-on-one with at least one of your perfect prospects every single day in person or on the phone/Skype is one of the most valuable things any business owner can do.

    It’s so easy for us to get a chip on our shoulder and think that we know it all or WORSE assume we are our customer. When we call into this trap, we tend to think that we’re gonna be able to imagine the need, identify it and make a product and people are gonna buy it.

    The reality is, that when you start asking people questions that you’ve given here, they say things that you wouldn’t have thought of, yet are amazingly simple and almost dead obvious as is revealed in the following story . . .

    13 Random Strangers Off The Street Put Millions Of Dollars Into Tony Robbins Bank Account

    Tony Robbins, the world famous personal development guru surrounds himself with so many positive people that sometimes he forgets where his perfect prospect is. And all he has to do to snap himself back into the right mindset is think about this revolutionary eye-opening experience he had . . .

    His television show “Personal Power” was the most successful infomercial that had ever been done. It’d broken all the records and he was really proud of it but they wanted to come out with a new show and he wanted to make sure they reached even more people.

    So he thought, “Why don’t we bring in a focus group – a group of people who don’t have an answer to the question of “Who is Tony Robbins?” and lets have a professional moderator really find out what they think of our show and find out if we’re reaching them or not and figure out to impact even more people.”

    They arrange for this to happen and now he’s in Marina Del Rey in a room behind a two way mirror and nobody in the focus group knows he’s there. Tony watches the moderator
    interviewing 13 people and he learned some AMAZING lessons.

    Behold Tony Robbins Having His Ego Body Slammed

    Tony figured that based on the response they had to their show that these people were all gonna love his show too. Not the case.

    When the moderator interviewed them in advance, one guy happened to mention, when they asked the participants if they were ever involved in any personal development seminars and stuff like that, and the guy mentioned he’d been to seminar of someone else’s before and listened to some tapes.

    You should’ve seen how the mob jumped all over this guy. They started prodding at him implying that he was dependent on somebody else and how THEY didn’t need it because they were already confident and balanced.

    It was incredible the amount of emotional negativity they dished out towards this man.

    Then the moderator asked the group if they’d heard about Tony Robbins. One person in the group had heard about him, read his book and was enthusiastic. Other people were bothered by that.

    Then they showed the film and after doing so the response was pretty amazing.

    They had everyone fill out a piece of paper and on it was a drawing of people talking with each other and the moderator told the group, “These people are talking about the show that you just watched. Please write down what they’re talking about.”

    Next they gave them a chance to score Tony and other people on the show on terms of their sincerity, their vitality, their passion, etc. Well, Tony got really high scores and so he assumed that when the conversation started, everyone’d be really enthusiastic.

    They. Were. Not.

    One man stood up and said, “I love him. I want those tapes. This seems fantastic. It seems different than everything else.” As soon as he did, you can imagine what happened. Other people in the group jumped him saying, “What are you crazy? Don’t you know these people are trying to rip you off? They’re just trying to make money off of you! This is TOTAL SCAM!!!”

    They went on and on and on about how if this guy bought these tapes, he was gonna get ripped off.

    The moderator asked, “What do you mean? Why do you think it’s a scam?”

    They said, “All those people on there talking about their results are all PAID for their testimonials! Those people are saying that stuff because they got paid to do so! That isn’t what really happened for them!”

    Tony couldn’t believe that out of the 13 people, that 7 of them thought they were paid endorsements when not one of their testimonials was paid for.

    Then the moderator said, “How can this guy get ripped off if the tapes come with a money-back guarantee and if he doesn’t like them, he can send them back and get a full refund?”

    The group shot back, “You’ll never get your money back. That’s a total lie. That’s a total scam too! There’s no way!”

    They carried on and on and on saying “Well if this guy is so successful, why is he teaching other people? Why isn’t he just keeping this for himself?”

    The Light Bulb Turns On In Tony’s Delusional Mind

    Tony thought, “This is AMAZING! I had no idea people thought this way.”

    If he’d used his intelligence, he would’ve remembered that seven years prior to this, that was how he used to think which was ironic because one of the ladies there was real
    emotional talking about how this must be a rip off saying, “Yeah, I believe that Tony Robbins is sincere but he’s a LIAR!”

    This caused Tony to think, “WHAT?! How can I be sincere, AND be a liar?”

    She said, “He’s sincere and I believe he really cares and all that but he never lived in 400 square foot bachelor apartment in Venice.”

    By the way, they did this focus group about 10 minutes away from where he used to live. 2516 Pacific Avenue, Venice, California. Pretty amazing.

    The bottom line is she went on and on and on and she was so intense and shewas so angry, so upset, even though she’d never met him, and he was taken aback. But after being taken aback he got real curious.

    He thought, “What is this dynamic? Why are these people so emotional?”

    The moderator worked with the group for almost two hours. After conversation after conversation after conversation asking each person if they’d see any value in these tapes at all, one older gentleman said, “No. I’m retired and I’m happy about it,” with a terrible frown on his face as he’s saying this.

    So after a while, the moderator asked, “What would convince you that these tapes were at least worth listening to and then sending them back for your refund if you didn’t want them?”

    They said, “We need some real evidence that this stuff works – not just paid testimonials!”

    The moderator agreed with them and told them he’d be right back. He left the room and walked into where Tony was sitting and asked him if he wanted to talk to these people.

    Tony said, “I guess so. These guys are pretty mean but okay.”

    Tony Robbins Walks Right Into The Eye Of The Storm

    And so Tony walks into the room and you should’ve seen the state change when they saw him.

    There was the gasp of disbelief and they all got excited saying, “It’s him, it’s him,” nudging each other. Tony sat down and said hi to them and they said hi back and smiled back at him being very nice after they’d just beat the hell out of him and his show.

    Tony said, “I want to thank you all for being here because you just gave me an unbelievable education. See this mirror here? It’s not a regular mirror. On the other other side I was watching and listening to your feedback on my show. Fortunately, I don’t take it personally but I had no idea people could be this pessimistic. All those people you see on the show are sincere. No one was paid for their testimony. The bottom line is that they’re all sharing because they got results. They’re all sincere.”

    Then he turned to the woman who said he’d never lived in Venice before and said, “2516 Pacific Ave. Apt. 3A” Check it out on your way home, it’s only 10 minutes away.”

    He said, “I needed to hear this. I needed to hear your responses to this because my goal is to reach more people. Some of you said I’m doing this to make money and you’re absolutely right. I’m a business man. I make money, but I get to do something that I love to do that makes a difference in the process. If I wasn’t making any money, I wouldn’t be
    reaching any people, so yeah, certainly I want to do well but I really want to impact people and I’m not impacting you guys. You aren’t hearing my message. I want to reach you. Help me. How can I reach you? ”

    It’s was fascinating.

    They said, “Well, tell us more about what you do.” And Tony told them what he did and why he did it and sure enough in about 5-10 minutes they’re telling him, “This is what you should do. You should be on that show. Don’t have as many of those testimonials because you’re really great. We really like you.”

    And Tony said, “But I’m the same guy who was on the infomercial.”

    They said, “It’s different having you here in person. Now we know you’re real. You’re not like all those other scams.”

    They kept talking about all those other scams and he listened to that. To the one man who was retired, Tony asked, “Would you never get some tapes like this?”

    He said, “I would’ve got them years ago. Years ago I used to get tapes all the time. I used to go to the seminars. I’d give my all and it wouldn’t work and I finally got tired of trying to give my all. I don’t need that. I’m happy the way I am,” with this terrible scrunched up look on his face.

    Tony thought how interesting this was that this man kept giving, kept tryingand it didn’t work and it didn’t work and it didn’t work and it dawned on him!

    The Great Wall Of Skepticism That Keeps Your Conversion Rates Low

    He thought about why these people were so emotional, so angry about trying to prove this stuff wouldn’t work. Tony finally realized what it was. These people had been disappointed so many times that now they’re deathly afraid of getting excited.

    Their brain tells them that if they get excited, they’re gonna take a fall – “If I get all jazzed and pumped up about this thing, what’s going to happen to me is what always happens to me – it’s not gonna work out and I’m gonna look like a fool! I’m gonna tell my friend about this great business opportunity and then the company goes under and my friend is gonna hate me forever! Or this guy’s gonna get me thinking I can really succeed and I’m gonna go out there and make some investment or put myself on the line and really stretch myself and then I won’t pull it off and then I’ll really be a failure and I’ll feel really crappy so I’ll just stay right here where I am and not do ANYTHING and watch my life drift away.”

    He turned to these people after he’d listened to them talk for awhile and he said, “You’re paid to be here and you don’t have to take anything from me, you can get up and leave at any time but I think the real reason you guys wouldn’t order these tapes in not because you didn’t believe in the guarantee, not because you think these people are lying. I think it’s because you guys are afraid to get excited. I think you’ve been disappointed so many times that you’ve given up and . . .

    I think the real reason you wouldn’t buy these tapes is because you don’t believe in you.”

    Tony looked over the room and of the thirteen people, 11 are shaking their head, ‘yes’, and the other two kinda saying they don’t agree with him.

    Tony tells the 2 people, “You don’t have to agree with me, but what do you think?”And after contemplating deeper they came around and said, “Yeah, you’re right.”

    After the focus group did Tony’s show, they did one for a diet product. And the one woman who’d been so against Tony’s show in the beginning, when asked later if she’d use the diet product, she said, “No, because the truth of the matter is, I wouldn’t succeed at this until I used something like Tony’s product and learned to love myself more.”

    It was unbelievable the totally different view this woman began to look at things from.

    As they were closing out on time, Tony said, “I’ve kept you here longer than we were supposed to but I’d like another 20 minutes with you and I’ll pay you for your time or you can leave, or I can give you a set of my tapes instead.”

    And guess what? Out of the thirteen people there, guess how many people wanted the tapes? Every. Single. One.

    Only one person had said they wanted the tapes before talking with him and
    even that had shifted when the mob turned on him for saying so. But there was a problem. Tony only had twelve sets of tapes with him.

    So he said, “One of you can’t have the tapes right now, who of you would be willing to wait to get the tapes?” The older guy said, “I’ll wait to use them because I probably can’t benefit from them that much anyway.”

    What was really fun was that by the time Tony was done speaking twenty minutes later, after he’d got a chance to really chat with these people and as they’re leaving, the older gentleman who said he was gonna wait for his tapes, fought with another woman to get the last set of tapes so this woman that really wanted them had to wait because this old guy took the tapes and bolted with them.

    Tony got several things out of this eye-opening experience . . .

    One, he got to see how negatively conditioned, pessimistic, and skeptical a random group of Americans could be.

    Second, he saw why people would be negative.

    He never understood why before.

    He just thought people were being negative and he’d make people wrong because of how hard he worked at being positive. He realized this negativity was oozing out of their fears.

    He realized that when someone is negative, they’re in a fearful state of mind. It’s the same
    thing that happens when someone is throwing their big ego around – it reveals how insecure they are.

    These people have tried things and they didn’t work and they have a lot of references-experiences where they tried something and it was painful and now their brain has linked up and says, “You get excited, you really go for it, you’re just gonna get disappointed. You’re not only gonna get financial pain but you’re gonna get pain in the form of shame.”

    And this is why those people were so emotional about making Tony’s product wrong. They didn’t want to believe because they were afraid to believe. And all of this was unconscious.

    Third, he learned from the old guy, that everyone still wants to dream.

    Everyone still wants to make it happen.

    Everyone still wants to slay the dragon and be the hero by doing ordinary things extraordinarily well. EVERYBODY WANTS THAT!

    And everybody somewhere in their gut believes this is possible for them and they’re waiting for someone to come along and kick them in the ass and show them how it’s still possible for them to dream and be heroic.

    And now he’s got more reinforcement for his mission and what he does.

    He tells this story to remind you of the kind of environment you live in. You’re pretty positive if you’ve read this far. But if someone around you is being negative, they’re not wrong, they’re not bad, they’re not a negative person. They’re just fearful.

    Now Tony Robbins is a force to be reckoned with. I don’t know if you’ve ever been in a room with him but it an experience you will never forget. So one might think, “Well, of course Tony Robbins can get these people to open up to him but people are going to want to open to me like that. I’m no Tony Robbins.”

    And as Henry Ford says, “Whether you believe you can or can’t, you’re right.”

    There is one outstanding tool that I’m aware of that helps copywriters/business owners get in touch with what makes for a good interview and how to get nervous, shy, and apprehensive people to open up deep and wide to go beyond giving you surface level fluff responses and give you the real emotion-rich answers that will make your copy sing but it is off the market.

    It is called Secret Six Interviewing and it was put together by a former Canadian TV talk show host turned phenomenal copywriter by the name of Shaune Clarke and he used to sell it for $800 bucks but now the sales page is up, but there is no shopping cart attached to it as Shaune has poured all of his focus into training speakers.

    There is no other course I know of that was directly applicable to helping copywriters become better interviewers and this is why I would recommend that anyone interested in improving this crucial skill, go find Shaune on Facebook or Twitter and ask if there’s any way that you can buy this from him. It is worth the effort as the training is world class and in a league of it’s own.

  • Now now, you’re confusing things here.

    A/B testing is to validate hypotheses, not to discover what makes them tick. What you describe here is part of any normal CRO process (employed by anyone who knows what they’re doing) – BUT it is no replacement for A/B testing. Split testing the only way to validate what actually works.

    After you conduct the interviews etc you’re still left with hypotheses – and that’s where A/B testing comes in.

    • Lance Jones

      I actually have a different take on it, Peep. In my experience — which is based on running the optimization program for Intuit and now overseeing optimization programs at 30 of the Fortune 500 (as a leader in Adobe Global Consulting) — most organizations do, in fact, use split testing to discover what makes their visitors tick.

      And I think at its core, this is a valid approach. It just takes a lot of time, traffic, and resources to pull it off.

      At its core, TESTING = LEARNING (about people; your visitors).

      The way you describe it is also correct: TESTING = VALIDATING HYPOTHESES.


      Every test comes with learning about the test participants. You can, in fact, test your way to learning a TON about your target audience, but as I say above, it’s very expensive.

      I’m writing about a shortcut to all that testing. You can use qualitative interviews to significantly narrow the field of possibilities (hypotheses), and that’s what I’m promoting here — especially to businesses that don’t have the traffic to run a big series of tests.

      I hope that helps clarify my position.

      • oligardner

        Nice click bait title 🙂

        What you’re describing *is* the CRO process. Gather data, form a hypothesis – *then* test. Be it insight from an interview, poll, live chat, analytics, heatmaps etc.

        So anyone testing without a concrete hypothesis is simply doing it wrong. That’s just common sense. Anyone doing tests based on guesswork is just another example of natural selection – we need people to suck at things in order to see how fabulous we are.

        I think this post could have been written in one sentence.

        “Don’t start an A/B test without gathering customer insight first.”


        What you do need to be equally wary of is basing your hypothesis on the comments of the vocal minority. You are still inferring something dangerous from your qualitative research if those who wouldn’t convert anyway are responding to your questions. Some people are just lonely and need to talk to someone 🙂

        Tests that only touch the surface of customer behaviour will take a long time regardless. You won’t reach statistical significance quickly if you haven’t used the data correctly.

        So I’d say that just as it’s important to ask the right questions, it’s also as important to know how to dissect them into something useful.

        Good conversation going on here.

      • Lance Jones

        I obviously chose the right blog title if Oli from Unbounce and Peep from ConversionXL replied. 😉

        This entire post was in response to (1) the poor testing practices that I observe (e.g., running many tests to arrive at a learning that could’ve been better derived through other techniques), and (2) the challenge of optimizing copy without a lot of website traffic.

        There is a cost to split testing. In many cases, there is a platform/software cost. There are people costs (hours x $). And there are opportunity costs (i.e., how else might my time or dollars be better spent?). Testing volume is constrained by budget limits and by traffic.

        I’m advocating for using a technique to optimize website messaging (i.e., user research via phone or email) that is less “expensive” and requires no website traffic.

        I’m definitely not suggesting that testing should be overlooked.

        We’re all on the same page here, but I coming at it from a slightly different angle.

        Thanks for jumping in!

  • Yassin Shaar

    Awesome post. A couple of questions i’d add:

    – What were your biggest doubts and hesitations before buying (or hiring us) ? (This is how i identify the main sources of anxiety and any objections)

    – How is your life better thanks to it? Which tangible improvements in your life or business have you seen? (This will tell me the end-benefit my product provides in the words of my customers. If some say really nice things, i hit them up for testimonials or case studies.)

    – Were there questions you needed answers to, but couldn’t find any ? (this will open my eyes for missing info on my landing page or website)

    And one last question is

    – What else would you like to buy from us (if we were smart enough to offer it)? (Ideas for new products or services my customers maybe ready to pay for)

    I usually use qualitative data to formulate test hypothesis and then run A/B testing to validate it quantitatively.

    Business owners need to be always talking to their customers, i even believe they need to sell their own services/products every now and then to keep their machine oiled. We need to stay close to our customers and identify any emerging trends.

    One awesome book on understanding your cusomer’s buying decision making process is “Roadmap To Revenue” by kristin.

    Keep up the great work lance

    Make your day great

    • Lance Jones

      Yassin, thank you so much for the awesome suggestions. We’ll be sure to put those to good use in the future!

      You make an interesting observation: “I use qualitative data to formulate test hypotheses and then run A/B testing to validate/invalidate it quantitatively.”

      In my opinion, that’s exactly how you should go about optimization. But I wrote this post because I see so many companies (large and small) skipping the first part (qualitative data) and just relying on split tests to run everything by their visitors — which is expensive… in terms of time and resources.

  • I find myself grinning at the main message of this post. Many of my clients’ websites do not have the traffic to run tests in an expeditious fashion. On the oft chance they do, I’m like a pig left alone in an abandoned farmers market.

    I’m quick to run surveys on a website to glean the information needed to make choice decisions for a product or service. It’s obvious you guys do the same here. Why else would you focus on USP so much? 😉

    • Lance Jones

      Hi Hubert! If you’re referring to the little poll we’ve been running in the Copy Hackers page footer, then you get a gold star for your astute observation. We had a pretty strong “gut feeling” that many people struggle with writing a value proposition for their product or service, and our informal poll (now with 530+ responses) confirmed it!

      • It’s awesome to have your hunches validated by science, or more appropriately, your customers.

        I actually am going to be conducting a survey similar to the one you’re suggesting, but I was going to do 12 in-person interviews. I had planned to let my online survey run until I finished the in-persons. Your 5-8 suggestion is intriguing. Is there any statistical reasoning behind the range?

      • Lance Jones

        That recommendation is based on Jakob Nielsen’s assertion that you can identify 80% of the usability problems on a website with as few as 5 usability study participants. And just like identifying usability problems on a website, getting a ton of insight from qualitative-focused interviews don’t require a large sample. After about 8 people, you’re going to start hearing the same types of things again and again. Beyond 8, each additional interviewee may provide a little incremental insight, but you’ll see diminishing returns.

      • Ah, right. Cool, thanks Lance!

        My survey isn’t focused on usability, but the theory still likely applies. I guess I’ll know when I conduct survey #9. LOL Thanks again!

  • That’s all well and good, Lance, but how can I interview my customers?! Surely there must be a good way of getting these questions in front of my website visitors in an non-intrusive way that provides them with incentives to answer the questions and instant gratification!! Have you heard of anything like that?

    But seriously folks (if you didn’t get that joke you’re not a die hard copyhackers fan … ) this issue of testing at low volume is, I think, one of the most important messages in online marketing today.

    We’ve had the “always be testing” mantra for a long time dominating the collective conscience of digital marketers and it’s GREAT because we’ve finally broken free from the tyranny of HIPPO (highest paid person’s opinion) when discussing tactical details of an online marketing campaign… but if you’re not working with a client that already has 10,000 uniques a month you’re kind of up shit creek without a paddle.

    I posted something similar to this on my blog which talks about data as being a “scarce resource” required for decision making. The most important thing is to determine first what you *can* measure in sufficient volumes, then determine what decisions can be made based on that information (that’s actually worded better than it is in the article … )

    • Lance Jones

      Hello Mr. Dooley… whatever do you mean? 😉

      Thank you for the well-disguised plug for our latest endeavor, “Kyvio”.

      I’ll return the favor and recommend that people have a read of the article you posted. Great advice there!

  • scott

    Elegant solution to the need for enough traffic to get decent results. I really wish I had thought of it some time ago 🙁

    Nicely done!

    • Lance Jones

      It’s never too late to talk to your customers! They — and your website copy — will love you for it, Scott.

  • Bryan Hackett

    Good article. I like the sample interview format.

    I agree that qualitative methods are better when you are in the early stages of understanding your customers and the problem space.

    It is better to save split testing until you enough traffic to give a meaningful level of confidence that the test results reflect reality.

    • Lance Jones

      Thanks for commenting, Bryan. We’re all about testing here at Copy Hackers, but we also understand the limitations of testing… especially if you’re a startup with low website traffic volume. This approach to learning is a fantastic alternative to running a series of tests.

  • danielgonzalez

    Awesome post! Thanks for sharing your interview template. Consider it Swiped!

    • Lance Jones

      Happy to help… and swipe away, Daniel!

Copywriting tutorial

Blog writing software

Jo will be teaching conversion copy at…

Experience Inbound in June in Milwaukee
Learn Inbound in Sept in Dublin
Content Jam in Oct in Chicago
StartCon 2018 in Dec in Sydney

We built a million-dollar business on blogging

Amazing blog posts build businesses and print money. Now Copy Hackers is teaching indies and teams to write kick-ass posts in half the time. Get notified when we're live.

Unsubscribe anytime. 100% privacy. Powered by ConvertKit