The Flaws of Forced Selection Exercises in Applied Field Research

Question

During research experiments, do users make less authentic decisions when they are forced to answer within a closed system?

For the purposes of this paper, closed and open systems refer to the user's ability to answer using a closed set of answers (e.g. multiple choice test) or an open set of answers (e.g. verbal response). An example of a closed system is a closed card sort or a multiple choice test

 

Participants

  • Intercepting visitors of a website
  • n#= 36

 


Process

 

Answer question in semi-closed system
Participants are shown 4 images and asked to answer the question "If I asked you to choose the object that is most like object X, what would you say?" Participants could choose to select an answer that did not force them to choose between A, B, and C. This answer is called answer "D." These answers are recorded under the title "round 1."

Round 1 Participants where asked to click on the object that is most like object X. If participant clicked on D, he would proceed to round 2.

Round 1
Participants where asked to click on the object that is most like object X. If participant clicked on D, he would proceed to round 2.

 
Round 2 Participant would be shown the same question, but without option D.

Round 2
Participant would be shown the same question, but without option D.

Answer question in fully-closed system.
If the user chose answer D, he was asked the same question again, but without the ability to answer "D."  These answers were recorded under the title "round 2."


Results

Semi-Closed System

Fully Closed System


Observations

  • In the semi-closed system, 14 respondents (38 %) chose "D: Each object has elements that are similar to object X."
     
  • When those 14 participants were asked to asked the same question again without the opportunity to answer "D," participants are forced to respond differently or abandon the study.
     
  • The distribution of answers between options A, B, and C in round 1 are different than the distribution of answers from round 2.
 

Interpretation

When researchers force participants to make decisions in closed systems, the participant may react differently than he might have in a semi-open or open system. As a result, the accuracy of data from closed systems may be less than data from open systems.


Further Reading

  • http://digest.bps.org.uk/2010/09/what-do-i-want-dont-ask-me-choice.html