I prefer persistent A/B versions (the way Google Optimizer runs). Both ways are valid, but when you look at the results of your test, be sure to note which way (persistent or not) the test was run.
For example, if I’m testing an account setting, I want it to look the same for a user each time they log in. This will help me measure the effects of the setting and its success. If it changes each time, it may confuse the user, which will add a new variable to your test results (and may invalidate the results, depending upon how you’re measuring success of each version).
Hubspot’s way could still be helpful in some cases, but I wouldn’t use it for any A/B test that examines user workflows or other actions that may be repeated by the same user. Hubspot’s way could work for things like sidebar ad text or other content the user may see once – but it really depends on the test.
This is a new area to me. I have used Adwords for a while but for some reason the concept of testing different scenarios in a systematic way never really clicked with me until last couple of weeks.
Now I am on a mission to learn as quickly as possible.
12 Reader Comments
Back to the ArticleLara Swanson
I prefer persistent A/B versions (the way Google Optimizer runs). Both ways are valid, but when you look at the results of your test, be sure to note which way (persistent or not) the test was run.
For example, if I’m testing an account setting, I want it to look the same for a user each time they log in. This will help me measure the effects of the setting and its success. If it changes each time, it may confuse the user, which will add a new variable to your test results (and may invalidate the results, depending upon how you’re measuring success of each version).
Hubspot’s way could still be helpful in some cases, but I wouldn’t use it for any A/B test that examines user workflows or other actions that may be repeated by the same user. Hubspot’s way could work for things like sidebar ad text or other content the user may see once – but it really depends on the test.
digitalark
Hi Lara
This is a new area to me. I have used Adwords for a while but for some reason the concept of testing different scenarios in a systematic way never really clicked with me until last couple of weeks.
Now I am on a mission to learn as quickly as possible.
Thanks for a great article.
Simon