The article makes a good general point that design matters in drawing a user in/pushing them away, but comparing the three designs, as mentioned before, isn’t really fair.
The other question I’d ask is whether it’s actually fair to compare the performance of the three when we don’t know what controls were put on the tests. The three designs presumably weren’t all live at once, so who’s to say one design wasn’t just subject to a slow day/week of sales?
What other marketing was going on while each design was up on the site that may have affected results?
The designs change too much and (barring more info from the author) seem to have too few checks over the environment in which they were competing to make any real conclusions on the merits of each.
The other question I’d ask is whether it’s actually fair to compare the performance of the three when we don’t know what controls were put on the tests. The three designs presumably weren’t all live at once, so who’s to say one design wasn’t just subject to a slow day/week of sales? What other marketing was going on while each design was up on the site that may have affected results?
The usual way to carry out A/B testing is to have both versions live at the same time. Roughly half the users will be served one version of the page, and half the other. That should mitigate against the sort of other factors you mention.
I think B would be even better if the layout were more like A. Needs more whitespace. Layout C is just unnecessarily complex. The process of deciding on a small purchase is sequential: Read what it is, find out how much, purchase. But two columns forces the user to juggle all three things as soon as the page loads. You’re asking to fill out a form to buy something before it’s even clear what you’re selling.
However, given some of the (apparently) misinformed comments I think all three designs have a more serious flaw: It’s not clear what the product is. First, let’s make it clear that I know nothing about this company except what’s in the screenshots, and I don’t care if they exist or not. But if I understand the copy right (esp the “Background” section seen on the first design) this “report” is a compilation of publically available information that law enforcement agencies must provide to anyone who ask, in accordance with child protection laws in the US. Some states do a decent job presenting the information online by themselves. Others do a bad job, thus creating a market for third-party sites like this.
Those may or may not be good or effective laws (take it up with your member of congress) but the result is certainly not some random “creepy guy” blacklist compiled by fly-by-night spammers as some have implied. Calling the author/designer unethical seems both uninformed and unfair.
This article really missed the mark by not trying to really analyze why B was slightly better. As others mentioned here it probably had something to do with the testimonials being more avaiable and the huge “Click Here” steering the user past the jumble of content. Though personally I think the click here is just a cop out for a bad design with no clear path for the eye to follow. Sure C is confusing but so are the others—they just are better because at least there is a simple down direction to follow.
I agree with the article that real testing is great but really… how often can you get a client to pay for that? This article could have served better if it not only tested but analyzed the whys. For at least then we could walk away with a bit more understanding for use the next time a client won’t pay for design user testing.
This is sound advice for ANY business in ANY industry. Live testing is the only way to know what really works, and everyone should be open to the idea that release #1 may not be (and probably isn’t) the most effective version.
For the record, I used live testing in a phone card distribution business, and was able to come up with a protocol that allowed sales to increase linearly (to keep up with cash flow) in an almost unbounded fashion. The results after 18 months of this gameplan? 130% increase in sales, 400% jump in profits, 81% jump in per-unit profit.
When I read something like this, it just serves to drive home a point that I was lucky enough to have firsthand experience with. I hope everyone who reads this sees the value here.
I understand the point the author is trying to make but the methodology is appalling and acts to counter, the useful point that I think is lurking in the article.
To make a test meaningful you can only alter one variable. How can you tell what caused the difference in sales if both copy and layout was changed. As an earlier commenter suggested, perhaps it was the copy that had the real effect, not the layout as the author implies.
Copy & paste the code below to embed this comment.
Jon Smith
The major difference to the results was copy changes, not design. The authow even stated as much in the opening: “Version B follows the same basic layout, but we made some minor copy changes.”
To my eyes all 3 options looked confusing….didnt know where to look or what different area’s did.
Hire a professional web designer to actually design the site rather than tinkering with slight variations…. THEN see the results.
When I clicked the Marketing Experiments’ Journal link in our author’s profile it took me to alovely white background page with one sentence on it… No website is configured for this URL.
The author’s condescending remark, “I know they won’t win any design awards. But they are functional and familiar” betrays the offensive implication that designers simply decorate web sites with pretty graphics, with no understanding of what they are doing.
The article is not ‘provocative’ because it is suggesting something radical or novel, it is provactive because of it’s crass arrogance and ignorance.
Surely in 2006 we’re not still debating the value of design! The purpose of design (whether in print or web) is to aid the communication of ideas and information, and to create desire.
I would be amazed if an IA, information designer or graphic designer had been anywhere near the example pages that were shown. All three were terrible – it’s a wonder that site made any sales at all!
I am all for live testing (under rigorous conditions),,, but does the author of this article (which I have to say, has no place on this illustrious site) seriously believe that as designers we are just poking around in the dark hoping for the best?!
Copy & paste the code below to embed this comment.
P R
First: Back to the basic purpose of the article: improving website design. Still pertinent and still needing to be examined…even in this lofty year of 2006! In fact, what seems to be missing in relating to the comparision test comments—is the realization of what is the difference in the effects between A, B, C.
Too many of the comments are concerned with details. Without a doubt, the best version was site B. Why? Structurally it was ergonimically appreciated visually (meaning eyeballroll) and did not ‘yank’ the head down as was version A’s affect…which to a tired neck, is a complaint. Further, a minor frustration that the core of the info which immediately ‘weights’ seems to be sinking into the bottom of the computer screen.
Test it yourself. Where are your eyes immediately drawn to before you reavert to search the site? Looking right down at the bottom of the screen. Now you understand the effect of the neck being ‘yanked’.
In version B, the bulk that the eye first seeks is better positioned to the upper left, and is more effortless to view overall.
Copy & paste the code below to embed this comment.
clay davis
forgive me if this was mentioned already but I think b was most succesful simply because of the “want names pictures and address” link near the top. who wouldn’t avoid mucking through such a bad layout if they could click on a link and get what they really wanted? the rest of the design choices in my opinion are irrelevent.
74 Reader Comments
Back to the ArticleAdam Perfect
The article makes a good general point that design matters in drawing a user in/pushing them away, but comparing the three designs, as mentioned before, isn’t really fair.
The other question I’d ask is whether it’s actually fair to compare the performance of the three when we don’t know what controls were put on the tests. The three designs presumably weren’t all live at once, so who’s to say one design wasn’t just subject to a slow day/week of sales?
What other marketing was going on while each design was up on the site that may have affected results?
The designs change too much and (barring more info from the author) seem to have too few checks over the environment in which they were competing to make any real conclusions on the merits of each.
Stephen Down
The usual way to carry out A/B testing is to have both versions live at the same time. Roughly half the users will be served one version of the page, and half the other. That should mitigate against the sort of other factors you mention.
Bryan Costin
I think B would be even better if the layout were more like A. Needs more whitespace. Layout C is just unnecessarily complex. The process of deciding on a small purchase is sequential: Read what it is, find out how much, purchase. But two columns forces the user to juggle all three things as soon as the page loads. You’re asking to fill out a form to buy something before it’s even clear what you’re selling.
However, given some of the (apparently) misinformed comments I think all three designs have a more serious flaw: It’s not clear what the product is. First, let’s make it clear that I know nothing about this company except what’s in the screenshots, and I don’t care if they exist or not. But if I understand the copy right (esp the “Background” section seen on the first design) this “report” is a compilation of publically available information that law enforcement agencies must provide to anyone who ask, in accordance with child protection laws in the US. Some states do a decent job presenting the information online by themselves. Others do a bad job, thus creating a market for third-party sites like this.
Those may or may not be good or effective laws (take it up with your member of congress) but the result is certainly not some random “creepy guy” blacklist compiled by fly-by-night spammers as some have implied. Calling the author/designer unethical seems both uninformed and unfair.
Colin Fahrion
This article really missed the mark by not trying to really analyze why B was slightly better. As others mentioned here it probably had something to do with the testimonials being more avaiable and the huge “Click Here” steering the user past the jumble of content. Though personally I think the click here is just a cop out for a bad design with no clear path for the eye to follow. Sure C is confusing but so are the others—they just are better because at least there is a simple down direction to follow.
I agree with the article that real testing is great but really… how often can you get a client to pay for that? This article could have served better if it not only tested but analyzed the whys. For at least then we could walk away with a bit more understanding for use the next time a client won’t pay for design user testing.
Chris Pearson
This is sound advice for ANY business in ANY industry. Live testing is the only way to know what really works, and everyone should be open to the idea that release #1 may not be (and probably isn’t) the most effective version.
For the record, I used live testing in a phone card distribution business, and was able to come up with a protocol that allowed sales to increase linearly (to keep up with cash flow) in an almost unbounded fashion. The results after 18 months of this gameplan? 130% increase in sales, 400% jump in profits, 81% jump in per-unit profit.
When I read something like this, it just serves to drive home a point that I was lucky enough to have firsthand experience with. I hope everyone who reads this sees the value here.
Alex Walker
I understand the point the author is trying to make but the methodology is appalling and acts to counter, the useful point that I think is lurking in the article.
To make a test meaningful you can only alter one variable. How can you tell what caused the difference in sales if both copy and layout was changed. As an earlier commenter suggested, perhaps it was the copy that had the real effect, not the layout as the author implies.
Jon Smith
The major difference to the results was copy changes, not design. The authow even stated as much in the opening: “Version B follows the same basic layout, but we made some minor copy changes.”
Maak Bow
To my eyes all 3 options looked confusing….didnt know where to look or what different area’s did.
Hire a professional web designer to actually design the site rather than tinkering with slight variations…. THEN see the results.
aliyu aliyu
Its very difficult sometimes to determine what kind of layout to use for a particular site.
Shashank Sondhi
When I clicked the Marketing Experiments’ Journal link in our author’s profile it took me to alovely white background page with one sentence on it… No website is configured for this URL.
Why is that happening…?
adjava john
I have a great website and i want to add articles source in this site, plz help me. I want articles on trade and business.
Damian Stephens
The author’s condescending remark, “I know they won’t win any design awards. But they are functional and familiar” betrays the offensive implication that designers simply decorate web sites with pretty graphics, with no understanding of what they are doing.
The article is not ‘provocative’ because it is suggesting something radical or novel, it is provactive because of it’s crass arrogance and ignorance.
Surely in 2006 we’re not still debating the value of design! The purpose of design (whether in print or web) is to aid the communication of ideas and information, and to create desire.
I would be amazed if an IA, information designer or graphic designer had been anywhere near the example pages that were shown. All three were terrible – it’s a wonder that site made any sales at all!
I am all for live testing (under rigorous conditions),,, but does the author of this article (which I have to say, has no place on this illustrious site) seriously believe that as designers we are just poking around in the dark hoping for the best?!
P R
First: Back to the basic purpose of the article: improving website design. Still pertinent and still needing to be examined…even in this lofty year of 2006! In fact, what seems to be missing in relating to the comparision test comments—is the realization of what is the difference in the effects between A, B, C.
Too many of the comments are concerned with details. Without a doubt, the best version was site B. Why? Structurally it was ergonimically appreciated visually (meaning eyeballroll) and did not ‘yank’ the head down as was version A’s affect…which to a tired neck, is a complaint. Further, a minor frustration that the core of the info which immediately ‘weights’ seems to be sinking into the bottom of the computer screen.
Test it yourself. Where are your eyes immediately drawn to before you reavert to search the site? Looking right down at the bottom of the screen. Now you understand the effect of the neck being ‘yanked’.
In version B, the bulk that the eye first seeks is better positioned to the upper left, and is more effortless to view overall.
clay davis
forgive me if this was mentioned already but I think b was most succesful simply because of the “want names pictures and address” link near the top. who wouldn’t avoid mucking through such a bad layout if they could click on a link and get what they really wanted? the rest of the design choices in my opinion are irrelevent.