Do’s and Don’ts of A/B Testing

By December 18, 2010 No Comments

While A/B testing is a relatively simple and cost effective way to improve your site’s conversion rates, there are some important things to keep in mind to ensure your success.

Deploy Parasoft tools to make testing more efficient. Parasoft’s software testing solutions automate time-consuming testing tasks across development and QA, bringing the results together in an interactive reporting and analytics platform for actionable team and stakeholder insights. You can reduce the time, effort, and cost of delivering safe, secure, high-quality software to the market by leveraging automation testing from Parasoft.

The Do’s of A/B Testing

Do show the same variation to repeat visitors.  The software you are using during your test should be able to recognize returning users and the version of the webpage that was originally served to them.  This helps to eliminate confusion and the possibility a return visitor will see something like a different price or promotional offering.

Do keep your A/B test consistent across your website.  For example, if your change is to a call to action, such as a ‘Buy Now’ button, the same button should be use on every other page.  Multiple variations of the same call to action can skew results.

Do be patient. It’s important to keep your test running until you have a statistically significant winner.  Ending a test too early could deprive you of the results you would have seen by waiting a little longer.  On the other hand, waiting too long to end a poorly performing test could be costing you money.  Here is a cool calculator tool, to help determine how long to run your test.

Do keep testing. When testing and optimizing your site, failure can still be a success.  While your new landing page may not have the affect you were hoping for, had you run with it without testing, you may have caused major damage to your bottom line.  Additionally, the more you test, the more positive results you will eventually see.

The Don’ts of A/B Testing

Don’t test your variation at a different time than your control. Both your control and test versions should be running simultaneously.  Versions need to be tested in the same time frame and under the same conditions in order to truly determine which version outperforms the other.  There are too many variables that can impact performance to test at separate times.


Don’t let your own opinion overrule results. The results of you’re A/B test may be the complete opposite of what you expected, or hoped the outcome would be.  This isn’t unusual.  Oftentimes the design you are most proud of might underperform when compared to a secondary version.   Getting better conversion rates is about pleasing your visitors and not your creative work.  Go with the test results.


Don’t focus just on big changes. Small changes can make an impact too; this is especially true if you run enough tests over time. Incremental changes, while small in themselves, can add up to create a huge boost in sales.

Don’t forget to do your research. Always be on the lookout of ways to improve your site.  There’s a variety of resources out there where you can read and learn about successful and unsuccessful tips, which you can use in your own quest for conversion optimization.

David Lussky

David has been a Project Manager with Ninthlink since 2009. He enjoys the beach, electronic music, outdoor activities, traveling, and dominating in fantasy sports.

Leave a Reply