I admit, it’s a provocative headline. But it’s true.
However compelling the message, however great the copy, however strong the sales argument… the way a page is designed will have a dramatic impact on conversion rates, for better or for worse.
Before I go any further, I want you to look at three versions of the same offer page:
I know, they won’t win any design awards. They weren’t intended to. But they are functional and familiar. A reader going to any one of these pages will be able to quickly figure out what the message is, and what they are being asked to do.
Version A is the original.
Version B follows the same basic layout, but we made some minor copy changes.
In version C, we changed from a one-column format to two-column format. We wanted to test the impact of bringing more of the page content onto the first screen.
Be honest with yourself and decide now whether B or C beat A, and by what percentage#section2
Don’t scroll down and look for the answer. You’re a designer, an expert in web design. So put your money where your credentials are and write down some figures now.
Write down a percentage by which B did better or worse than A. And a percentage by which C did better or worse than A.
The design choices you make have a profound impact on results#section3
I imagine you have some way of measuring the success of your site. Maybe it’s about sales. Maybe it’s based on readership. But one way or another, your site has a purpose.
But I don’t think most designers truly understand the effect their design choices can have on achieving that purpose.
And yes, I’m sure you do some usability testing. And that likely gives you some broad, if sometimes confusing insights into what’s working and what isn’t.
But do you test different page designs?#section4
By testing, I don’t mean asking a few folks around the office; I mean doing a live test that demonstrates—with hard figures—what site visitors actually do.
Testing like that is a beautiful thing. There is no space for fancy arguments. An expert’s credentials and opinions mean squat. When you serve alternative versions, one after the other, and measure reader actions, you get the real deal. You get what is.
Do you do that? It’s a scary thing.
But if you are serious about achieving your site’s purpose, and if testing can show you which version of a page does best, then where is the argument not to test?
Here’s how design choice can make a difference#section5
Here are just a few of the design elements we have found can make a significant difference to the performance of a web page:
- The position and color of the primary call to action
- Position on the page of testimonials, if used
- Whether linked elements are in text or as images
- The amount of “white space” on a page, giving the content space to “breathe”
- The position and prominence of the main heading
- The number of columns used on the page
- The number of visual elements competing for attention
- The age, sex and appearance of someone in a photo
OK… now for the results of the test.
| Page A | Page B | Page C | |
|---|---|---|---|
| Percent of traffic | 34% | 33% | 33% |
| New sales | 244 | 282 | 114 |
| Change | N/A | 15.57% | -53.28% |
Version B, with the minor copy changes, resulted in a 15.57% increase in sales—that represents a big revenue jump for a site with high sales volumes.
Version C, in which we changed the regular, one-column format into a two-column format, resulted in 53.28% fewer sales.
That’s an astonishing reduction in sales and revenues, resulting from a design change that was intended to improve the performance of the page.
Now, just pause for a moment and think of all the design choices you have made over the last year, and the reasons why you made them. And think about the huge impact those choices might have had on the performance of the sites you worked on.
Some concluding thoughts…#section6
The figures from this test are shocking. But they are not exceptional. Design changes really do have a huge impact on conversion rates.
Here are a few things to consider:
If you have some pages on a site which are critical to its overall success, instigate a program of A/B split testing. You cannot afford to guess; you have to know.
Be aware that however strong the copy and text on a page, its performance is very much dependent on the way in which it is presented. In other words, design choices can enhance or diminish the power of the words.
Talk with your writers. Ask them how they think the message would best be presented. Then test some different versions. A good writer should have some strong instincts when it comes to the layout of the text.
One way or another, it’s important to accept that none of us—neither designers nor writers—know what the “best” page design or copy is until we test.
In a business environment where marketers demand an accountable performance from every web page, it’s time to put aside the assumed expertise of design and copy “gurus.”
The way forward is to test, and let our readers show us which designs work best, and which copy works best.
While this may be uncomfortable for some, the end result is that we will become much better web designers and writers.
So is this another argument to realigning?
I DID guess the order on the usability. The way the eyes flow etc. I have just been reading a book on usability so im in that mood.
OK, I think the third option is more confusing because of the two colums.
But as far as I see the first one has much better layout – better whitespace handling more readable fonts etc.
So maybe the better impact of v2 is because of the prominent link and the customer statements.
So I would like to see a new test ( 🙂 ) with the layout of a combined with the link and the customer statements (but not in italics…).
With forms and with web sites in general, I really focus on how the user follows the copy in the page. Too much emphasis is placed on jamming as much stuff “above the fold” as possible. I say forget the fold in most cases. If you can guide users through a process with large bold call to actions, and not clutter the screen with excessive information, the experience will be naturally easier to follow.
We can really only deal with one thing at a time – a design should reflect that.
I had a few thoughts about the results of the test. The first thing I noticed when viewing v3 was not the two collumns but the word “Free” apearing at the top-left of the page. This one word sends chills down my spine every time. I actually think that that one word could have as much to do with the sales dropping as the design.
To test the impact of the design fairly, the actual content should probably remain. Any changes to copy may have as much to do with the results as design changes. (as was stated for v2)
I think post #4 nails it regarding the copy. The biggest thing I noted was the v2 design’s use of the big “Click Here” link, something very much missing from the other two versions.
I’d like to see the results again with the same copy in each design.
I actually preferred v3 to the others.
I too think that pageC was the best one regarding page layout. However, it missed immediately visible user testimonials.
PageB had a clear call to action (‘would you like names, … click here’), wich is why it was successful in my opinion. However, if it were formulated differently, like a short paragraph without too much marketing speak or dreaded ‘click here’ links, it would have been even more successful.
Personally i would have put a short description of the offer at the very top of the page. What is it, what are its benefits, its strongpoints, …
Then a 2 column layout, left hand side the sample report, right hand side a call to action: “You can order a report as shown in the sample of your area. If you like to know if there are sex offenders in your area, fill in the the form below”. Directly under it the form with a clear checkout button.
At the right hand side i would have put the narrow sidepanel with user testimonials, like on pageB. Not italicised and formatted like on pageC.
Too bad this cannot be tested 🙂
While I agree that design choices matter and have a profound impact on the overall effectiveness of a site, I can sum this article up in one sentence (instead of thousands of words):
“Design choices matter — we’ve researched it, so be careful.”
And to compare A, B, and C is ludicrous. For proper multivariate or A/B element testing (copy changes, headers, etc) you can’t implement them all at once and hold the pages against each other.
This article offers nothing more than a statement that design choices matter — and who doesn’t know that by now?
To be more specific, you’re testing is the IA of the page. What needs to be stressed is that overall visual design doesn’t vary too much, while the information architecture is the main factor being tested.
I also agree with the above comments that by changing the copy, the results are suspect.
Want to know why Design B was better? CLICK HERE!!!
While I agree with the above posts that this was not a true comparison as regards to design, I did still find the article interesting and a nice reminder of how our design choices might affect the bottom line for companies.
My question is, as a small, one-person design-firm typically working with smaller clients, how would I go about setting up these types of tests? How much does this type of testing cost and how is it implemented?
I design sites as an enthusiastic amateur but do have one commercial site in my portfolio (which does its job well) so may I comment as a web-design aware user of the net.
I scored B at +20% and C at -50% (without peeking honest) and did so in less than 2 minutes. I am not a speed reader so I did not even read the text which casts doubt on some of the above comments. My calculations were made on feelings I sensed immediately.
So, what did I sense? Put simply, value for space.
I looked at A first and thought that it was OK, if a bit basic, then when I saw B I saw more content because it uses the space better. I did not even realise that C was a 2 column design because that part of the page was lurking around and below the bottom of my viewport. All I saw was much less of the content than on either of the other designs.
Are not A and B 2 column designs as well?
BTW The use of a Click Here link in B is scandalous.
I’m the Senior Web Designer for “Crutchfield”:http://www.crutchfield.com and we always A/B or A/B/C test any major interface or design changes. It’s hard to argue with the numbers when sales have dropped suddenly on one variation.
This has saved us a number of times. When everyone involved is absolutely convinced that one particular version will win, it often does not and we quickly yank it from the site before losing too many tens of thousands of dollars. Intuition is not enough, and often users choices seem to be counter-intuitive.
I think it’s important to include A/B testing in the mix, and I swear by the 3-prongs: usability testing, site metrics, and A/B tests. They are all useful, but most useful when used together to compliment one another. Using only one can be very misleading — particularly when using site metrics alone without other data to complement it.
Try looking at the first two layouts in 800×600. Then try looking at C. No one likes scrolling horizontally for content and I have a feeling your target market is mostly at 800×600.
I think the article did a great job of explaining how important it is to conduct tests like this – but let’s get some details on how exactly to conduct A/B/C tests.
What questions were the test participants asked? Seeing the results table is great, but let’s see what criteria contributed to those percentages! Explaining a simple A/B/C testing process could really help small shops/individuals start benefiting from testing today!
Thanks!
irrelevant, it seems to me that any ethically minded designer would refuse this contact in the first place. “I love my predator report” – shocking.
Pardon my typo I meant contract not contact.
sorry, but i failed to find any useful information in this article. was your test purely a numerical one, or were any comments/reasons gathered as well? if not…what was the point? and B is not just a change in copy…you’ve managed to make the layout superbly cramped, and the small italics on the right-hand side are almost illegible. almost as if you deliberately tried to hit B with the ugly stick, prompting most design-conscious readers to think “well, between B and C, i’d go with C”, only to then set up your big *surprise* revelation that B actually performed better. Would have been a lot fairer if B and C had been identical, safe for the difference in columns used.
on a sidenote: “interesting” choice for a case study. Sex offenders, eh? Next month, I’ll be expecting a “thorough” analysis of the usability of adult sites (“you’d think the site with the image of huge breast would perform best, but in fact it was the one with the more trustworthy, professional, sober credit card entry form…shocking”).
I am an amateur web designer since about 1996, but my day job keeps me busy and away from designing. I read ALA to keep up. This article taught me two important lessons:
1) *Users Need Directions* – As another comment said we need to guide the user through the process of a website whether it be a form or just information in general. The big difference to me between A and B was the line:
bq. “Would you like *names*, *pictures*, and *addresses*? Click Here”
As much as we had to admit it, our users and client do like the “*Click Here*” links no matter how sacriligeous it may feel to include it in the page.
2) *Predator Report* – the one comment above said that this job should not have been accepted because of that line alone. However again, we need to take into account our audience. Are we going to incite people into anger or just push them along into getting the full report for $10. Personally, I do not want to inflame anyone into getting angry enough to do anything rash. This site and copy does not seem to do that to me.
Again, I am just a “beginner” with no formal design training or graphic education. I have picked up what I know from reading, trying and listening to sites like ALA. Thanks for a great job on a site that provides a tremendous amount of information for the average web designer.
Predator Report — the one comment above said that this job should not have been accepted because of that line alone.
If you were referring to my post, that’s not actually what I said or intended the post to be taken to mean. I was rather questioning the sites “business model” for want of a better phrase and wondering why anyone would be prepared to assist in the promotion of such. Is cash king to the extent that those of us who have a choice, in as much as we are not directed by managment, leave our ethics at the door when we sit down at the keyboard? This is an entirely different discussion to the intended one so I shall sign off here.
I question the reliability of this test. I’d need to know more about the testing methods you used and how many testers there were, and from what demographics etc…
There isn’t enough evidence in this article to just take your word for it.
One thing, that may play a part in this example and others like it is how the demographic views a web site. Ever noticed a tendency of younger audiences to load a web page and immediately scroll up and down quickly until they find something that catches their eye? How about older viewers who like to read word for word top to bottom in a book fashion. It would be interesting to see the results of each experiment broken down by age group, as each generation is now begining to show general tendency to view interactive content in their own way–atleast in my expierence.
Version B has the giant call to action: “Would you like names, pictures and addresses? Click here”. I think that is *the* largest factor causing the difference in sales. Secondary to this is the testimonials. My experience tells me that any time you put testimonials it instantly gives a lot more credibility to a product than without.
Freakin’ bizarre choice of subject matter for a commercial test! And done without apparent irony. This is the sort of thing I’d expect to be hawked by the virus that just dumped a hundred links on a web site of mine. When did the boogie man go mainstream?
I’d love it if the company I work for had the resources to do this kind of testing. As it stands, I usually run a couple of speculative, mostly functional layouts past a handfull of friends and ask for feedback on design, usability, and stuff like that. It’s cheap and actually gives me great leads more often than you’d probably think. But my friends and associates aren’t exactly the target audience for the sites of my clients. But I do the best I can, and I’m sure it’s better than nothing.
But I’d like to do more. But how do I do it? Particularly, how do I do it on a limited budget? What questions do I ask? Do I show everyone each version of the site or do I split up my testers into groups equal to the number of site versions I have to test? How many people should I use as my test group? “Don’t Make Me Think” suggests that testing with more than 5 people is a waste of time, from a usability standpoint. Would you agree with this as applied to this type of user testing? Others have mentioned the fact that the differences between A, B, and C produce too many variables to be measured properly. Is this truely an issue? Should as many things be held constant as possible so that we’re sticking closely to the scientific method?
A case study of the results of such a test is all well and good, but I think a case study of how this information is gathered and computed would be even more helpful. Feel like writting a follow up or should I just go subscribe to MEJ?
I have to agree with Derek, comment 22. The biggest draw to B is the call to action. This is something that many designers fail to put in their designs. What do you want the user to do? Give them the opportunity to do what you want them to do. It’s not rocket science. Even a “bad” design with a obvious call to action can work.
This is a very good illustration but I a left wondering if the table layout and the copy are two different variables that should be tested as such. Maybe a test with the same text and different layouts first then change the copy to see who gets the cheese.
Numbers like these without some estimate of the error in the measurement don’t mean anything? How many users were tested?
The copy and the order are important things. There is a lot of factors that modify user experience, credibility and influence sells.
It would be better to run a thorough test. For example, provide design like “C”, but”¦ realigned. Rearrange blocks or put testimonials like it is done in variant “B”.
Test them all with the same copy. Test the copy separate from design or layout. Test different headlines, different testimonials.
Of cource, we should consider who were our “testers”. I mean, gender and age differences, for example.
By the way, I like variant “C” — nice looking. I believe it can be realigned, filled with better-selling copy and it can become, at least, better than “A”.
Are the images labelled correctly? Based of the reported search time, B was taken before A and C. It appears that A must have taken place after C because the number of Registered Sex Offenders increased from 32 to 33. This means the original version was the more cramped version, the two column was the one that brought in slightly higher sales, and the more spacious version had the worst sales.
I feel that with all this standards compliancy and usability talk going around we forget the importance ‘good design’ plays in all this (it’s all part and parcel). Now I liked this article, and I’m thinkng of how I will go about split testing layouts (picking users at random and using cookies would be my first guess). But back to my original point. Perhaps the reason C did so badly was because it was badly designed. It’s common knowledge that forms aren’t the most inviting aspect of web design and giving it center stage would (in my opinion) be a major ‘turn off’ for anyone. Could the form be streamlined? could an image have been used instead? Sorry to be a kill joy, I really did enjoy the article, but in the words of someone else,
‘It’s like choosing between three pairs of dirty underpants, when what you need is a clean pair of underpants’
I found this article quite shocking in it’s choice of example. Surely you could have found a more neutral subject matter? What’s next? Advanced CSS layouts for a page on land-mine effectiveness?
Hi Nick,
This was a really nice article, but I made some texts with those examples that you gave us. I ask some co-workers to see and just choose, in what site he feels more safe and confortable to buy.
Anybody here, besides me, speakes english, everybody choosed to buy in PageC.
Ok, I have to agree, if they don’t speak english, they don’t will buy correctly, and my tests was a bit wrong.
But I thought if the culture don’t make the things go different…
Leaving C aside, I refuse to believe that B is the redesigned version, and that A is the original. How B is an improvement on A is certainly beyond my understanding.
Here’s a suggestion: why not fill the page with a photograph of a dirty old man, leering disgustingly at the camera? If that doesn’t improve sales, I’m not sure what will.
I’d be willing to model for that. I’ve been working on leering and perversion in my spare time. I’m 25, but it’s amazing what a good makeup artist can do.
Example C says “and costs *just $10*” directly towards the top. The other examples didn’t specify price until below the form.
*In Example C*
I believe the users may immediatly leave the site when they realize they need to pay.
*In B & A*
On the otherhand if the user has already started filling out the form, they’ve already invested some time into it and seen all the information in the top section. When the $10 issue comes up they may grumble to themselves. However at that point they already have understood how unique the report is.
_*C loses conversions in the first 2 paragraphs*_
_*A & B loses conversions after the form*_
As with any marketing its important that the audience realize the value in the service. Example C tells people the price before explaining the value, which loses people before they can read any further.
Hey, to me, page C looked the best. It shows you a sample report clearly there, and you enter your details alongside – who could fail to see what to do with that?
What I really don’t understand is the author’s comment on
bq. Position on the page of testimonials, if used.
Who believes the testimonials they read on a website like that?
And as for the “click here” link on B attracting the most customers – how hard would it have been to do it properly and make the whole sentence “click here for …” into a link?
bq. What questions were the test participants asked? Seeing the results table is great, but let’s see what criteria contributed to those percentages! Explaining a simple A/B/C testing process could really help small shops/individuals start benefiting from testing today!
The point about A/B testing is that the users don’t realise they are part of a test.
What you do is configure your server to randomly send one of two versions of the page each time it is requested – so roughly half of users will see one version and half will see another. You then track how many sales result from each version.
The advantages of this method of user testing are:
– You can test a large number of people with very little effort.
– You can see the genuine effect the site has on people’s actions. When users knowingly test a website, they don’t necessarily react in the same way they would if they were just *using* it.
The main disadvantage is that you don’t find out *why* users preferred one version over another (I suspect that in many cases, the users wouldn’t be able to give you a coherent reason either…), or which aspects of each version work best.
And, especially if you have a relatively small user base, it can be very error prone.
Users _did not prefer_ one version to another just because they worked only with one version, I suppose. And this is good, because when we ask user to comment the design, user _tries to look like an expert_ and his words become unreliable.
Maybe a little «exit-poll» after buying could help. Ask users how did they know about your site, what made them buy your product, etc. I realize this is not the best and not the most exact way.
The second trick is log analysis. I believe that users do not click «buy it» immidiately. Some users click other links, they try to learn more about what they’re going to pay for. So user’ paths may be useful.
And (I repeat) *we need _the same copy_ if we’re testing different designs* and the same design if we’re testing different copys.
I’ve personally seen +/- 20% fluctuations on a day-to-day basis regardless of the changes made. It seems like there are so many other variables that could impact the results (# of visits, day of week, month, etc.).
It would be nice to see more advanced statistics (i.e., trend analysis) to better validate the numbers in this article.
I’m not sure what we are to learn from this – it seems like the kind of thing a pure marketer brings to a meeting as evidence to prove that good design is a waste of time. “See! All you need are ‘click here’ links and testimonials from people with names like ‘Justice’! And will someone please make that copy bolder, redder and more emotionally manipulative!”
Sorry.
Wouldn’t it have made more sense to test the same basic design with a small variation for each? I suppose that wouldn’t have supported the argument, but at the basest level, design B has the edge over the others cos it has an explicit call-to-action, an obvious link smack in the middle of the page.
The problem with running an A/B (or A/B/C) test in this fashion is that it doesn’t show you a way forward. It would be interesting to now try an A/B/C split with the previous winner (design B) as the control, and two variations; one with the “Click here” link removed, and one with “names, pictures and addresses” as a link.
As others have pointed out, you have to know the raw numbers involved to accurately interpret the results. The demographic segment that uses the site would also be handy to know.
Frankly though, this kind of thing seems a bit out-of-place – and tawdry – for ALA.
“In a business environment where marketers demand an accountable performance from every web page, it’s time to put aside the assumed expertise of design and copy “gurus.”?”
Doesn’t that cover pretty much the entire ALA readership?
“[…] let our readers show us which designs work best, and which copy works best.”
Anyone remember the car that Homer Simpson designed?
“[…] the end result is that we will become much better web designers and writers.”
“Better” is a very subjective term, but I’d like someone to show me how the triumph of design B could possibly make us better or worse designers and writers.
as said above if this is a layout test, then copy needs to be the same. If this is a copy test then layout needs to be the same for adequate controls in testing. This isn’t really a decision made about ‘designs’ here because the call to action and the copy is different on each item.
I would also like to see the test with all three including the click now in the same positioning, and a test with none of them including the click now. I think that would make a more compelling article than this (nothing personal to our author who put in his time and effort).
btw I thought C was the most visually appealing while still providing the baseline information for the site within a decent amount of breathing room.
as a followup to my previous post above here is my thoughts on all the designs:
design A: I think this one is functional and provides a good choice. Personally the call to action not being immediately visible, and action required at the bottom of the page you could argue the negatives of.
design B: I actually thought this was the worst of the 3. there is no white space, it comes off as too cramped and riding up on me. It needs some room to breath in my opinion and the right columned text size, while I like it, would be the first thing I would receive as a complaint [:(].
design C: I thought this was the most visually compelling, to me, of the three anyhow. It traded off and compromised well with visual appeal, breathing room, and still providing instant access to the neccessary information. Again the call to action could be questioned. I believe this choice would always be a designers choice, whether a functional end user choice is another discussion.
While the use of white-space, font color, and layout contribute greatly in any project, could the answer to the swing in sales be security?
I stay as far away from on-line store fronts in the development world, but I do look for the visual cues when I am a consumer.
A :
-No lock on credit card field
-No Verisign logo
-No security policy info
B :
-Lock on credit card field
-Verisign logo
-Authenticated Verisign symbol
-No security policy info
C :
-Lock on credit card field
-No Verisign logo
-Authenticated Verisign symbol
-Security policy info
Is that the full answer?
No.
The layout of C may have played a role in the flux of sales, but we may see the comlicated formula of online sales and marketing take a turn when the “sense of security” is included in the equation
i can’t even understand that the designer accepted this contract, so how could i understand that he disuss about the commercial impact of his design !
i am sure version c could be improved significantly. i wonder why the form has been put into the hot spot and not the example. would be interesting to see how conversion changes when the columns were switched. and it would be fair to give version c the same header and introduction as version b without mentioning the price etc.
I am working on a Technical Degree in Web Design and I eat this kind of information daily trying to digest it out as a style statment. I saw no difference in overall design in B or C except the column change. What am I missing? My goal is to be a w3 compliant designer and use CSS for everything but I keep getting hung up with wanting to put design elements in where they don’t belong. Any advise? From “Stranded in Wyoming”
This is digressing a bit from the article, but I must say that the idea for the site is probably by far the most disgusting thing I’ve seen in my entire life. It pretty much made me sick. Hanging out people like that…
Now I see why this might be a great buisness-model, with the parents and all, but holy crap. It really hurts to see an actual site like this.
I agree that the point that design choices matter is stating the obvious. However I personally took the articles’ overal point to be that a designer needs to test there layouts thoroughly in a practical, analytical and possibly statistical manner. I think this advice is apt as myself and most designers I know tend to test there layouts by showing to a few people and discussing their merits rather than actually running tests on them.
That has got me thinking about how I shall test layouts in future.
I do agree that the specific analysis would need to be more in depth than the article presents to be truly useful. You don’t want to just know that B got a bit more sales than A and many more than C, but also WHY that is the case.
Never mind the design choices, what about the ethical choices? I find it disturbing that you would design a website for, effectively, vigilantes.
I’m no more in favour of the people your example site names than the next guy (sorry about the wording, I’m allowing for filters), but there have been enough cases of mistaken identity leading to innocent people getting hurt that sites like this are a really bad idea. There are established legal channels for dealing with this sort of thing, and making sites like this is just pandering to vigilantism – and worse, profiting from it.
However valid the design issues, a bad choice of example.
but the Sample Report should have been on the left and the form on the right. Bah! scrap that idea there is too much on screen! Try putting the form on another page?
I liked the article, but it was not scientific. No real control, plus the data sample was too small. I think the best layout would have been a combination of B and C, with the prominent sales “copy” (Click Here) prominently displayed. In all three cases the yellow button at the bottom of the screen did little to impact first impressions. The call-to-action is the key element in Item B that likely did the work.
Leaving aside the question of the tastefulness or otherwise of the web page in question, is this article really on the level?
Out of curiosity I googled my way to nationalalertregistry.com, entered the zip code of 32225 and got….Version A. And just in case nationalalertregistry.com were still doing user testing a la comment #37, I tried accessing the same nationalalertregistry.com page via two other browsers. Same result.
So what’s going on here?
I still don’t understand why you don’t write this kind of article using figures from an hotel booking web site or anything more usual and less … subject to questions …
anyway :
B/ Is ugly. No space. Not readable. May be there is good idea in it, but the design itself is a mess i thing. Of course it improves conversion because of the “Click here” button. You could have the same result keeping the A design and simply adding this link at the top …
C/ Is not very good from a design point of view but the idea is ok. It would ask for a redeisgn. You can’t really have any conclusion on the figures you give if you don’t know more (duration of the test, target list, new users / recuring users …)
My conclusion reading this article, as many other comments, is that all is in the title and not more ;-( but it can be interesting for people who are not aware of that (i thing ALA readers are …)
Subject matter aside, am I the only one who thinks that all 3 of these designs are amateur and hideous?
Does fact that one terrible design tested better than two other terrible designs make this a valuable study of web design (especially when each version used completely different verbiage) ?
I could go into my garage and bottle 3 different homemade wines. I could then go down to the vacant lot beside the local liquor store and conduct a taste test with some ‘users.’ One of my wines would score higher than the other two. What would this prove? Obviously it proves that the wrong wine label design can cripple a wine.
^To be perfectly analogous with the subject matter I should probably note that these particular wines are being used prepping the user for a ‘Bum Fight’ which we are taping at the same location.^
Naturally these would be some valuable findings, and I would submit them to Wine Spectator magazine, who, being well regarded by vineyards and spectators alike would surely put the story on the cover with a provocative headline.
Come on, ALA, show some editorial control.
Despite what percentages say, I do really prefer PageC. PageA is also better than B, since B does really make me sick when I try to read it: line-height and white space in general are definitely too poor. This is the most important thing to be noticed in my opinion, more than the 2 or 1 coloumn layout. In fact I think that the best design might be a re-designed C version. Two changes might be:
1) Please take away that damn FREE word, i’m sure people associate it with SPAM. I do. You do. We do. They do. So it’s a stupid bug that corrupts the design of the page.
2) The 2 coloumns layout is ok, but why should they be put this way? As the natural flow of the reader tries to find the sample first, just put it on the left. I think it’s quite obvious, isnt it?
Of course many other changes are to be made, but these are really important im my opinion.
PS: When I see a “click here” link the very first thing I do is laughing. Guess why.
Do you think the fact pageC mentions the cost straight away, whereas the other two don’t mention cost until you have been sold what you get in the report, has an effect?
I think the space and look on pageC is easier to read than the other two, but I would stick to selling the report first and having the form underneath. I also think the headline in pageC makes the purpose of the page clearer. What about keeping the same text as B but laying it out with the spacing from C.
I think this is interesting but, I’m a bit put off by your scare mongering to make money. I just hope you don’t make mistakes and send people to the wrong addresses like has happened in this country(UK).
bq. Does fact that one terrible design tested better than two other terrible designs make this a valuable study of web design (especially when each version used completely different verbiage)?
This is _not_ about what factors make a good design. It is not about whether we, as authors, prefer one design to another.
It _is_ about how we find which designs work best for any given site (remembering that different sites have different demographics, so require different approaches).
The aim behind this article (as far as I can tell) was not to say “B is a wonderful design”, nor was it to promote vigilante-ism or to disgust readers by the choice of material. It was to show a *real example of A/B testing* and how the results might differ significantly from what we expect.
We could now discard the A and C designs, and refine B into B1 and B2, and see which of those performs best, and use that as the basis to improve the design further. Don’t think that the game is over just because we have completed Level 1.
Zach Harkey “said”:http://www.alistapart.com/comments/designcancripple?page=6#55 …
bq. To be perfectly analogous with the subject matter I should probably note that these particular wines are being used prepping the user for a “Bum Fight” which we are taping at the same location.
Heh. You said “Bum Fight.”
I can’t see having 3 different page designs, active at different times of the year (thus the market can shift significantly during those times) as yielding results worthy of legitimate note.
We all want to analyze how visitors navigate our website, and there are (somewhat limited) tools available to do so to an extent, but I feel that any site which doesn’t necessarily bombard the user with data, and instead takes an approach in which navigation and information are minimalized, will have visitors who might not make a purchase, but when they do, may both do so quicker, and return more often; due to an ease of use factor.
The article makes a good general point that design matters in drawing a user in/pushing them away, but comparing the three designs, as mentioned before, isn’t really fair.
The other question I’d ask is whether it’s actually fair to compare the performance of the three when we don’t know what controls were put on the tests. The three designs presumably weren’t all live at once, so who’s to say one design wasn’t just subject to a slow day/week of sales?
What other marketing was going on while each design was up on the site that may have affected results?
The designs change too much and (barring more info from the author) seem to have too few checks over the environment in which they were competing to make any real conclusions on the merits of each.
bq. The other question I’d ask is whether it’s actually fair to compare the performance of the three when we don’t know what controls were put on the tests. The three designs presumably weren’t all live at once, so who’s to say one design wasn’t just subject to a slow day/week of sales? What other marketing was going on while each design was up on the site that may have affected results?
The usual way to carry out A/B testing *is* to have both versions live at the same time. Roughly half the users will be served one version of the page, and half the other. That _should_ mitigate against the sort of other factors you mention.
I think B would be even better if the layout were more like A. Needs more whitespace. Layout C is just unnecessarily complex. The process of deciding on a small purchase is sequential: Read what it is, find out how much, purchase. But two columns forces the user to juggle all three things as soon as the page loads. You’re asking to fill out a form to buy something before it’s even clear what you’re selling.
However, given some of the (apparently) misinformed comments I think all three designs have a more serious flaw: It’s not clear what the product is. First, let’s make it clear that I know nothing about this company except what’s in the screenshots, and I don’t care if they exist or not. But if I understand the copy right (esp the “Background” section seen on the first design) this “report” is a compilation of publically available information that law enforcement agencies must provide to anyone who ask, in accordance with child protection laws in the US. Some states do a decent job presenting the information online by themselves. Others do a bad job, thus creating a market for third-party sites like this.
Those may or may not be good or effective laws (take it up with your member of congress) but the result is certainly not some random “creepy guy” blacklist compiled by fly-by-night spammers as some have implied. Calling the author/designer unethical seems both uninformed and unfair.
This article really missed the mark by not trying to really analyze why B was slightly better. As others mentioned here it probably had something to do with the testimonials being more avaiable and the huge “Click Here” steering the user past the jumble of content. Though personally I think the click here is just a cop out for a bad design with no clear path for the eye to follow. Sure C is confusing but so are the others—they just are better because at least there is a simple down direction to follow.
I agree with the article that real testing is great but really… how often can you get a client to pay for that? This article could have served better if it not only tested but analyzed the whys. For at least then we could walk away with a bit more understanding for use the next time a client won’t pay for design user testing.
This is sound advice for ANY business in ANY industry. Live testing is the only way to know what really works, and everyone should be open to the idea that release #1 may not be (and probably isn’t) the most effective version.
For the record, I used live testing in a phone card distribution business, and was able to come up with a protocol that allowed sales to increase linearly (to keep up with cash flow) in an almost unbounded fashion. The results after 18 months of this gameplan? 130% increase in sales, 400% jump in profits, 81% jump in per-unit profit.
When I read something like this, it just serves to drive home a point that I was lucky enough to have firsthand experience with. I hope everyone who reads this sees the value here.
I understand the point the author is trying to make but the methodology is appalling and acts to counter, the useful point that I think is lurking in the article.
To make a test meaningful you can only alter one variable. How can you tell what caused the difference in sales if both copy and layout was changed. As an earlier commenter suggested, perhaps it was the copy that had the real effect, not the layout as the author implies.
The major difference to the results was copy changes, not design. The authow even stated as much in the opening: “Version B follows the same basic layout, but we made some minor copy changes.”
To my eyes all 3 options looked confusing….didnt know where to look or what different area’s did.
Hire a professional web designer to actually design the site rather than tinkering with slight variations…. THEN see the results.
Its very difficult sometimes to determine what kind of layout to use for a particular site.
When I clicked the Marketing Experiments’ Journal link in our author’s profile it took me to alovely white background page with one sentence on it… No website is configured for this URL.
Why is that happening…?
I have a great website and i want to add articles source in this site, plz help me. I want articles on trade and business.
The author’s condescending remark, “I know they won’t win any design awards. But they are functional and familiar” betrays the offensive implication that designers simply decorate web sites with pretty graphics, with no understanding of what they are doing.
The article is not ‘provocative’ because it is suggesting something radical or novel, it is provactive because of it’s crass arrogance and ignorance.
Surely in 2006 we’re not still debating the value of design! The purpose of design (whether in print or web) is to aid the communication of ideas and information, and to create desire.
I would be amazed if an IA, information designer or graphic designer had been anywhere near the example pages that were shown. All three were terrible – it’s a wonder that site made any sales at all!
I am all for live testing (under rigorous conditions),,, but does the author of this article (which I have to say, has no place on this illustrious site) seriously believe that as designers we are just poking around in the dark hoping for the best?!
First: Back to the basic purpose of the article: improving website design. Still pertinent and still needing to be examined…even in this lofty year of 2006! In fact, what seems to be missing in relating to the comparision test comments– is the realization of what is the difference in the effects between A, B, C.
Too many of the comments are concerned with details. Without a doubt, the best version was site B. Why? Structurally it was ergonimically appreciated visually (meaning eyeballroll) and did not ‘yank’ the head down as was version A’s affect…which to a tired neck, is a complaint. Further, a minor frustration that the core of the info which immediately ‘weights’ seems to be sinking into the bottom of the computer screen.
Test it yourself. Where are your eyes immediately drawn to before you reavert to search the site? Looking right down at the bottom of the screen. Now you understand the effect of the neck being ‘yanked’.
In version B, the bulk that the eye first seeks is better positioned to the upper left, and is more effortless to view overall.
forgive me if this was mentioned already but I think b was most succesful simply because of the “want names pictures and address” link near the top. who wouldn’t avoid mucking through such a bad layout if they could click on a link and get what they really wanted? the rest of the design choices in my opinion are irrelevent.