Five interesting conversion mistakes we’ve made and seen

This article is based on the webinar we co-presented with Rand Fishkin from Moz. Rand’s section of the talk has been moved into a separate article, “Does CRO mess up SEO? Or vice versa?” The article you are currently reading began life as a transcript, but transcripts are hard to read. We believe that the content is too important to be buried in a transcript, so we have heavily copy-edited it. Think of this as a transcript-article. A transcr-article.

If you’d prefer, you can get the slides, video and podcast of the webinar here.

How we discovered these five conversion mistakes

Karl: Our “Clients and Results” page shows how we have grown businesses in many industries. When you’re a consultancy, you get to see many projects. If you’re smart, then every time you see something go wrong—or every time you do something wrong—then you update your methodology to prevent that mistake from ever happening again.

So we are reflective. We look at projects and think two things:

• “If we had to repeat that project again, what were the ‘vital-few’ fruitful activities that we carried out?”
• Whenever something goes wrong, we ask ourselves, “How can we prevent that from happening again?”

Some of the mistakes I’m about to describe are common. Fortunately, we have processes to prevent them from happening. And now you can prevent them from happening too.

Mistake #1: The “Keitai Denwa” Problem

We came across the “Keitai Denwa” Problem years ago, before we formed Conversion Rate Experts. My co-founder Ben and I were running a cell-phone company that had U.S. and Japanese branches. This is one of the Japanese company’s pages:

I have used Google Translate so that you can understand what the page is saying, at least in a crude Google-Translate kind of way:

I’d like you to think of how you would change this page to increase its conversion rate. The page sells phones for Japanese people who are traveling abroad. The customers use the phone on their travels and then, when they return home, they keep the phone in a desk ready for their next trip.

Before you read any further, take a moment to think about how you would increase the conversion rate of the page in the image above.

Here’s the punchline to the story: In about 2003, we installed analytics software on this site. The software revealed that fewer than 10 percent of our visitors were arriving from the keyword we were targeting, “kaigai keitai,” (which means “phone rental”). Most of the visitors—more than 90 percent of them—were arriving on this page from the keyword “keitai denwa,” which means “cell phone.”

So they were searching for something that we didn’t even offer: normal domestic cell phones. This was because, thanks to our focus on SEO, we were number one in Yahoo! (Japan’s top search engine at the time) for a keyword we hadn’t even targeted.

“Keitai denwa” was a ridiculously valuable ranking to have. Most of our visitors were looking to buy a domestic mobile phone, and then realized that we sold phones only for travel. So they left. It was painful to discover that, for a long time, we had had one of the web’s most valuable rankings, but we had squandered it because we didn’t sell what the visitors were looking for. It also was an effective way of learning that the only way you can increase a website’s conversion rate is to understand who the visitors are, what they want, and why they aren’t converting. By the time that we discovered this, it was too late—the ranking was starting to slip. We didn’t sell domestic phones, and before we could start, the domestic carriers displaced us in Yahoo!

The “Keitai Denwa” Problem taught us that you can’t critique a site without knowing about the visitors. Frustratingly, even once you’ve learned that lesson, even when you think, “I get it now. It’s obvious,” it’s easy to slip back into making judgments without having the data. Every day, it takes an incredible amount of discipline to avoid making this mistake—not just when you are writing a webpage, but also when you are giving a talk, selling a product, or even writing an email. It’s possible to learn this rule one day, and then break it the next. No matter how many marketing techniques you use, you’ll always fail if you don’t know your audience.

Mistake #2: The “Cinema Foyer Effect”

The following image shows A/B split-test data for a page.

We noticed that sometimes, no matter what you do on a page, you can’t get a win. And the longer you leave a test, it just stays inconclusive. The answer comes from the world of science. Scientists often carry out what’s called sensitivity analysis, which is a way of measuring which variables are affecting the results and which aren’t.

Imagine you are in an industrial control room, facing a row of thirty levers. Presumably each of the levers does something. When you pull one of them, something happens somewhere else. But now imagine that some of the levers weren’t connected to anything. No amount of pulling them would have any effect on anything. On your website, some of your pages—or page elements—can be like those disconnected levers. Not every page element will change the number of orders you receive, no matter how hard you pull it. An obvious example would be a page that receives no visitors. If no one ever visits your “Terms and Conditions” page, for example, then no amount of yanking on it will increase your profits, because it’s not connected to your goals. That’s an obvious example, but your site will have many other “disconnected levers” on it. You may even discover that your homepage is a disconnected lever. Sensitivity analysis is about changing variables and spotting which of them are connected to the goal and which aren’t. On a website, many of the page elements aren’t connected to anything and don’t affect the conversion rate, no matter what you do.

We had a client that came to us after a site-wide redesign. The client’s new site had a completely different appearance, but the conversion rate hadn’t changed. It transpired that the whole website, for reasons that I’ll hint at later, was a red herring. We identified the real levers (which was the onboarding process of the software itself) and instead worked on those.

How can you spot this problem? You know you have this problem when nothing you do to a particular page (or page element) makes a difference. All of your tests on it are inconclusive; none of them win, and equally importantly, none of them lose either. For example, you might discover that everything you do with testimonials makes no difference. You’re pulling on a lever that isn’t connected to anything.

A common example of this is what we call the “Cinema-Foyer Effect.” Imagine that you own the cinema shown in the following image.

See the poster for the movie called “New York”? Imagine that on some days you put the “New York” poster there, and then on other days you put a “Fast & Furious 6” poster there. My guess is that the poster would give no measurable uplift to the number of people who go to see Fast & Furious 6. Why? Because if you were to ask the cinema’s visitors how they decided which movie to go to, their answer wouldn’t be, “I looked at the posters, then made a decision.” They would say things like

• “A friend recommended this movie to me.”
• “It was recommended on a movie-reviews podcast that I subscribe to.”
• “I saw the trailer for it last time I came to the cinema.”
• “I saw on IMDb that it has high ratings.”

The posters in the cinema foyer would probably not get mentioned. The posters may have been near to the transaction, but they didn’t influence the decision. The decision happened in the customers’ minds, when they saw the persuasive content (the friend’s words, the podcast, the trailer or the IMDb page). To influence the customer’s decision, you must first identify the persuasive content.

So what can you do?

• Step 1. Be aware that you will sometimes be barking up the wrong tree.
• Step 2. Identify where and when the conversions are really happening. That’s the most important point.
• Step 3: Ask customers when and where the persuasive content was. Then find ways of improving, duplicating and amplifying that content. Don’t give up just because it’s not in the most convenient place for you. Don’t be like the drunk who drops his keys in a ditch, but then looks for them under a streetlamp because it’s lighter there.

A good example of this phenomenon is Dropbox. People often ask why Dropbox’s homepage contains a sign-up form and very little else. I believe that it’s because by the time someone arrives at Dropbox’s homepage, they are already persuaded. They aren’t thinking, “Oh, this looks an interesting service, I wonder what it does.” By the time they get to Dropbox’s homepage, they may have already used Dropbox—perhaps a friend has sent them some files—so they know what it is. Or maybe their friend has explained to them what Dropbox does. Maybe they’ve read reviews elsewhere.

So what Dropbox has discovered is that it doesn’t help to give detail on this page, presumably because of the Cinema-Foyer Effect. This page isn’t where the conversion happens. And so the best that Dropbox’s homepage can do is to skip to the installation process. Dropbox’s marketing team can focus its conversion efforts on identifying where the persuasive content is, and what it is, and then improving and amplifying it. Because Dropbox’s service is naturally viral—and because users love it—then Dropbox’s team is right to focus on enhancing the virality and customer satisfaction. It does so by continually optimizing its referral mechanisms (like its tell-a-friend program) and the usability of the product itself.

So with your business, identify what and where the persuasive content is.

Mistake #3: Returning customers can cause deafening noise

We discovered this problem with one of our first clients, a company that sold health supplements. As with many of these examples, we wish we could show the actual page we designed, because it was clearly much better. It felt like we had “cracked the walnut with a sledgehammer.” We had usability-tested it, and the users were emphatic that it was much more persuasive. The page contained mountains of proof and the product’s benefits were explained much better. Then we ran an A/B test, which didn’t reach significance. We were confounded. And then we realized where we had gone wrong. On any given day, about 95% of the company’s customers were repeat customers, people who had taken the health supplement before and who were returning to the site simply because they had run out and needed to reorder.

If a customer has already taken, say, ten bottles of a health supplement, and they are just returning to reorder, then the only thing the website needs to do is to let them place the order. Nothing on the landing page is going to make much difference. We realized that no matter how persuasive our new page was, the only thing that it could do was to increase the number of new customers—the green guys in the following slide:

So we ran the test again, but this time we measured the number of new customers. Back then, this was difficult to do; these days, it’s much easier. We did it by analyzing the company’s database and tracking new accounts. We discovered that our new page was generating 114% more new customers than the control. So we had more than doubled the conversion rate of new customers to the site. But our success had initially been hidden in the noise of repeat customers.

So if your business has a lot of repeat users, whether it’s because your service involves existing customers logging in (as with a SaaS company) or because you have many repeat customers, then be wary of this problem. Whenever you try to increase the rate of new customer acquisition, make sure that you track the new customers. Otherwise, your successes will get lost in the statistical variation from your existing ones.

Mistake #4 is not having ADA on the project

What do we mean by ADA? ADA stands for authority, duty, and ability. The concept arose from a conversation we had with the online strategist Mark Nunney, and describes a problem that we see in many companies. When we speak to a new prospect who’s thinking of signing up with us, we now want to know that the prospect’s team has the following three attributes:

“D” stands for “duty”: You need one person on the project who is ultimately responsible for increasing conversions.

“A” stands for “authority,” the person whose permission must be sought to do all of the following:

• Make changes
• Make decisions
• Spend money
• Do whatever is needed to keep the project moving.

Authority is often missing from a conversion team, usually because the person who has authority has delegated the project to people who don’t.

The other “A” stands for “ability,” the person or people who can either do the work themselves—or control those who can. These tend to be the designers, copywriters, developers and analysts.

If you are setting up a conversion team, ensure that the team has all three of the ADA attributes.

In addition, we notice that the rate of progress of a project is inversely proportional to the number of people on the project; large teams move excruciatingly slowly. We believe that this is for two reasons: (i) A large team is a clue that no one knows who has the duty (no one can confidently say, “This isn’t my project”). (ii) No one knows who has the authority. Beware of people who claim to have the authority but who always “pass decisions by” someone else.

To counter this problem, Amazon’s Jeff Bezos famously instituted the two-pizza rule, insisting that any team should be small enough to be fed with two pizzas. (Our friend Brian eats like a horse; if he were to join Amazon, he’d be forced to work alone.)

In terms of lean-ness, you’d ideally have all three ADA attributes within one person, because then there would be no need for lengthy hand-offs, emails, and meetings. As Paul Graham says, “Imagine what Apple was like when 100% of its employees were either Steve Jobs or Steve Wozniak.”

If one or more of the ADA attributes is missing from your team, fix it. Progress will be painful until you do.

Mistake #5 is “Switching to the beauty lane”

Amazon’s website looks pleasant, but it’s not what you would call beautiful; its design is largely functional. This is true of most successful pure-play web companies (that is, companies that are successful because of their websites, not companies whose websites are being subsidized by other activities). Just look at Twitter, Reddit, Facebook and Craigslist. They all have functional design.

Now let’s look at what you might call a beautiful website. Take Rolex’s website. Or the website for M&Ms. Both look polished and elegant, as do most websites whose primary purpose is “branding” rather than functionality or selling. Now imagine how difficult it would be to make changes to either of those sites. They are so complex and polished, you couldn’t just say, “Let’s see what happens if we move this image to the left and add some text to the right.” There would be many considerations about the site layout, the text, and the font substitutions.

We often hear people say that they would like their website to be both functional and beautiful—as if Amazon, Twitter and Facebook haven’t considered doing that. The following image shows why this is so dangerous.

This is the first time we have shown this graph. We think it’s profound. Start by looking at the green blocks, which represent a website that’s designed for functionality, like Amazon et al. The axis that runs into the distance represents time, and the height of the bars represents the conversion rate.

Imagine that with your functional site you run an A/B test and get a win, so your conversion rate rises, as shown by the next green bar. Then you run another test, and you get another win. Each time you run a test, you increase the conversion rate, so those green bars rise like a steep staircase to success.

Now let’s assume that you redesign your functional-looking page, making it beautiful. And let’s assume (generously) that your beautiful page wins an A/B-test and has a 20% higher conversion rate. So far, so good; your business has grown, and you get a site that you’re proud to show to your friends.

But your next A/B test may take two or three times longer to create because you can no longer simply change the text. So your progress follows that of the orange bars in the chart above. Each time you change a webpage, you need to redesign the graphics, and consider the effects on the layout, and check with the branding department, and discover that you are going to break the clever CSS tricks that were need to make the previous navigation bar work in Internet Explorer, and then go through several rounds of QA-testing, and discover that you have broken the mobile version … and everyone is blaming each other, and your developers leave, and you wonder why it’s so hard to find good developers these days, and your competitors start to overtake you.

Welcome to the slow lane.

Most web companies—certainly more than half of them—are stuck in the slow lane. Their websites become elegant and beautiful—more beautiful than Amazon et al (surely that’s a clue that something’s wrong, no?)—and from then on things move more slowly, because every page, to beat the control, has to look elegant too. When you move from functional design to beautiful design, you may get an increase in conversion rate, but you do so at the cost of having to make every subsequent challenger page beautiful—otherwise every challenger page will miss out on that 20% beauty lift. The 20% beauty lift becomes a crippling tax on your rate of progress, maybe reducing your implementation rate by 50%, 70% or even more. Some companies are literally unable to make changes to certain pages, because they have wrapped themselves in so much complexity that progress reaches gridlock.

We see this time and time again. It’s really hard to persuade people of the benefits of being agile. The only thing we can point towards is the graph above, to this article we wrote about the subject, and to all those crazily successful companies that we’ve just shown. Of all of the mistakes that I’ve mentioned, this is the one that happens most often. It’s also infuriatingly hard to prevent people from making it—because the alternative is so alluring.

I hope I have persuaded you to resist the urge to change from the fast lane (the functional lane) to the slow lane (the beauty lane).

Q&A

“What processes do we have in place to uncover the Cinema-Foyer Effect?”

Karl: I would say the main way of uncovering it is to ask customers that question about what persuaded them. You can ask the question in many ways:

• On a phone call with an existing customer.
• As an on-page survey.
• As a survey afterward to customers.

The question and the way of asking it are largely independent of each other: you can ask most questions with most types of survey tool. But those are the most popular ways of asking it.

Also, you can identify that effect by testing removing page elements and seeing whether they made a difference. That’s often useful. If you de-clutter a page, testing what happens if you remove an element, you often discover that the page doesn’t need to contain all of the things that are currently there.

Rand: In addition to online surveys, you mentioned the in-person survey or the over-the-phone survey. I have found time and time again that the qualitative data from watching two or three people go through your process and asking them to talk their thoughts out loud, and to say what they’re thinking, can be even more valuable—or at least as valuable—as the hundreds or thousands of data points that you get from the automated tools and from surveys.

When people talk out loud, suddenly designers and developers and your usability specialists and your conversion rate optimizers, all of a sudden go, “Oh my gosh, I had no idea that when visitors were looking at that they thought X, Y, and Z.” And no one thinks to include that question in a survey or to give it as a response, so it can be pretty valuable.

How much did you like this article?

What’s your goal today?

1. Hire us to grow your company

We’ve generated hundreds of millions for our clients, using our unique CRE Methodology™. To discover how we can help grow your business:

• Read our case studies, client success stories, and video testimonials.
• Learn about us, and our unique values, beliefs and quirks.
• Visit our “Services” page to see the process by which we assess whether we’re a good fit for each other.
• Schedule your FREE website strategy session with one of our renowned experts.

Schedule your FREE strategy session

2. Learn how to do conversion

Download a free copy of our Amazon #1 best-selling book, Making Websites Win, recommended by Google, Facebook, Microsoft, Moz, Econsultancy, and many more industry leaders. You’ll also be subscribed to our email newsletter and notified whenever we publish new articles or have something interesting to share.

Browse hundreds of articles, containing an amazing number of useful tools and techniques. Many readers tell us they have doubled their sales by following the advice in these articles.

Download a free copy of our best-selling book

3. Join our team

If you want to join our team—or discover why our team members love working with us—then see our “Careers” page.

4. Contact us

We help businesses worldwide, so get in touch!

© 2024 Conversion Rate Experts Limited. All rights reserved.