The Scientific Method and Qualitative UX Research

I love discovering that something I think is true is actually false. Some people call it being "wrong," but I call it "science" (aka "learning"). It's a special moment: your mind rattles and your cognition widens just enough to entertain a world a little different than you originally thought. 

For example, for much of my adult life, I didn't think there was water on mars . I consider myself someone of average intelligence, so I was surprised to learn that water may be flowing through the beautiful red planet. During this moment of learning, I did not think less of the scientists who came before who had not discovered this finding yet. Instead, I acknowledged that science is a process of constant discovery, and that I must trust the process. 

As a UX researcher, I believe we have the opportunity to be more respected, accurate and impactful when we use scientific methods for our research.

Running water may be running through Mars. Image from Creative Commons.

Running water may be running through Mars. Image from Creative Commons.

But isn't qualitative UX research, by definition, not scientific?

Data may be qualitative, but the methods for acquiring them can be scientific in nature (which I will explain in detail later in this article). In science, we create falsifiable hypothesis statements and conduct experiments to prove or disprove them. The process of science consists of experiments, but the method of science will continue until a more logical problem-solving method appears. 

Some UX researchers, such as Jon Kolko, argue that design research is not science. Research that is intended to drive the design of a new product may be nonscientific and still valuable. In contrast, research that is intended to develop new and accurate knowledge may benefit from scientific methods.

UX research gives the practitioner experience and empathy. Scientific research gives the world knowledge. These are two very different outputs. Each output is relevant to a different part of the organization's conducting research.

Why would more scientific methods help?

While UX design and qualitative UX research may be growing as a field, the respect for UX research "insights" are just as vulnerable to doubt as any other industry, including the scientific research industry. Poor scientific research is critiqued regularly: in fact, this critique is a critical part of ensuring our knowledge of the world is accurate. The reason we don't have to constantly question whether or not a scientific finding is correct is because scientists critique each other to ensure that the methods have the highest likelihood of accuracy.

How might we blend scientific methods into our design processes? Image designed by Atvur.

How might we blend scientific methods into our design processes? Image designed by Atvur.

What would more science in UX research look like?

Integrating the scientific method into qualitative UX research doesn't necessarily require any formal training in science. Instead, consider integrating the following elements into your qualitative UX research practice.

1. Start with a clear question.

Start with a good question that can be answered. Ideally, there are a number of these questions that comprise the focus of your study.
e.g. How will users attempt to turn on our prototype of a new bluetooth device if they don't have printed directions?  

2. Create a hypothesis.

Create a hypothesis of how you imagine a person will behave. A good hypothesis is like an "if. . . then. . . " statement. 
e.g. If users are handed a bluetooth speaker, then they will will turn the device around in their hands, press the buttons on the device, and/or ask for the moderator's help. 
 

3. Identify your sample.

Clearly detail how you will choose the people in your sample. Indicate the pros and cons behind why those people are accurate representations of the audience.
e.g. We will interview 30 music listeners who are randomly sampled from across the globe. We will utilize a stratified sample of the population in an effort to ensure multiple user backgrounds are recruited.

4. Identify your testing method.

Clearly detail how you will test your hypothesis. Methods can include sample sizes, sample methods, and examination methods.
e.g. We will go into their homes and perform a contextual inquiry whereby we will ask them to turn on the bluetooth device. We will videotape their behaviors.

5. Identify methodology weaknesses.

No method is perfect, but not acknowledging the limitations is imperfect and unprofessional. Be ready to defend your sample size, your specific examination methods, the accuracy of your data, and the precision of the data. 
e.g. We believe we will not be able to accurately access a complete sample of users in southern asian countries. As a result, our findings may be different were we to perform the study within Asian countries. 

6. Prove or disprove your hypothesis. Propose revised hypothesis.

The end goal of your research is to clearly indicate how your data proves or disproves your hypothesis. 
e.g. Hypothesis disproven. We observed some participants actually tried to plug in their device. Our revised hypothesis is "If users are handed a bluetooth speaker, then they will will turn the device around in their hands, press the buttons on the device, try to plug the device into an electrical socket, and/or ask for the moderator's help. 
 

7. Interpret data with professional judgement.

Despite popular opinion, subjective interpretation plays a major role in scientific papers. The end goal of your research is to clearly indicate how your data proves or disproves your hypothesis. Often titled the "discussion" section of in scientific papers, this provides the scientist with an opportunity to interpret results and compare them against other experiments. 

Do you have suggestions or additions for a more scientific UX practice? Let me know and I'll reference you in an extended version of this article.