Research Methods: A/B Testing

A/B testing is an excellent way to gain first-hand insight while comparing and choosing the right design for a multitude of projects.

Also known as “Split Testing,” this method involves gathering a group of at least ten users to compare two different website designs in order to understand which one will perform best (Pannafino McNeil, 2017). Since this technique requires two versions of a design, this test is not suitable for preliminary usability testing, functioning like a product test. The original design serves as the control or the “A” and the new design, “B” serves as the variation in this test.

Watch Lesson 1 in the 1966 Design of Experiments series by Professor J. Stuart Hunter.

This technique has existed since the early days of marketing in the 60’s to and was developed even earlier in the 1920’s by statistician and biologist Ronald Fisher while running agricultural experiments.

So why is this research method so tried and true? Well, just like us humans love intuitive design, the same applies to how we learn. Although A/B testing was formally defined in the Twentieth Century, the act of comparing a new idea or invention, and comparing it’s use alongside a predecessor or similar concept or object is something people have done as long as innovation has existed. We want to know if this novel idea is worth the time and energy as opposed to what already exists.

A/B testing at its core consists of three components:

  1. What are you testing?

  2. What features you are measuring?

  3. What are the desired outcomes or metrics from the test?

These questions will help to define and assess your experiment and the following results.

In web design, A/B testing can be used for virtually any two designs you would like to compare. From to website building to newsletter and social campaigns to advertisements, it’s a simple way to learn how users will respond to the new version of a design.

When preparing to conduct an A/B test, researchers must first gather participants. James Pannafino and Patrick McNeil advise no less than ten in their book, UX Methods: A Quick Guide to User Experience Research Methods. From there, lay out your ground work, what designs you will be testing, the metrics you will be testing such as clicks or conversions, and then you can begin. These tests can be done in-person or on an online platform, since the users will be participating online anyway. The test itself is short and sweet: simply have participants choose which version they prefer and collect the results to analyze and come to your conclusions.

In a case study explored on website optimization company, Crazy Egg, the wall decal company, WallMonkeys used a heat map while designing a newly optimized homepage. While testing using the heat maps, they learned about what their website visitors clicked the most. Their search bar was identified as the most-clicked item, but the actual homepage was not as busy. The company then proceeded to A/B test with a new design informed with their new knowledge, replacing their stock imagery with a more engaging visual illustrating the possibilities of their product. WallMonkeys saw their conversion rate increase by 27%. By honing in on specific metrics and establishing clear and simple goals, they were able to achieve successful results. What’s even more compelling is that they ran another test after moving their most-clicked item (the search bar) to the middle of their homepage, and the company saw a 550% increase. (Crazy Egg, 2022).

Another example of A/B testing is this case study from the group video chat social networking app, Houseparty. While seeking a solution to a common error users faced while connecting to contacts, the app found a new design that would prevent users from unintentionally denying what was essentially the entire purpose of the app: finding friends. While the initial grey default iPhone pop-up gave no context and forced users to accept or deny the contact function, their new A/B-tested pop up allowed users to read why the permission is important, and was not as invasive-looking as the previous version. Houseparty saw two times more users adding friends on their first day and a 15% increase to access contacts. (Hubspot, 2021).


References:

Gallo, A. (2017, November 27). A refresher on a/B testing. Harvard Business Review. Retrieved February 13, 2023, from https://hbr.org/2017/06/a-refresher-on-ab-testing

JL. (2022, August 28). Case study: Wall monkey a 550% revenue conversion increase. Crazy Egg. Retrieved February 12, 2023, from https://support.crazyegg.com/hc/en-us/articles/360056296334-Case-Study-Wall-Monkey-a-550-Revenue-Conversion-Increase

Pannafino, James, and Patrick McNeil. 2017. UX Methods: A Quick Guide to User Experience Research Methods. CDUXP LLC.

Riserbato, R. (2021, October 12). 9 A/B testing examples from real businesses. HubSpot Blog. Retrieved February 18, 2023, from https://blog.hubspot.com/marketing/a-b-testing-experiments-examples

What is A/B testing? Mailchimp. (n.d.). Retrieved February 13, 2023, from https://mailchimp.com/marketing-glossary/ab-tests/

What is A/B testing? Predikkta. (n.d.). Retrieved February 10, 2023, from https://www.predikkta.com/learn/ab-testing.html

Previous
Previous

Research Methods: Photojournal

Next
Next

Is an Emoji Worth a Thousand Words? 🧐