The Flaws of Forced Selection Exercises in Applied Field Research
During research experiments, do users make less authentic decisions when they are forced to answer within a closed system?
For the purposes of this paper, closed and open systems refer to the user's ability to answer using a closed set of answers (e.g. multiple choice test) or an open set of answers (e.g. verbal response). An example of a closed system is a closed card sort or a multiple choice test.
- Intercepting visitors of a website
- n#= 36
Answer question in semi-closed system
Participants are shown 4 images and asked to answer the question "If I asked you to choose the object that is most like object X, what would you say?" Participants could choose to select an answer that did not force them to choose between A, B, and C. This answer is called answer "D." These answers are recorded under the title "round 1."
Answer question in fully-closed system.
If the user chose answer D, he was asked the same question again, but without the ability to answer "D." These answers were recorded under the title "round 2."
Fully Closed System
- In the semi-closed system, 14 respondents (38 %) chose "D: Each object has elements that are similar to object X."
- When those 14 participants were asked to asked the same question again without the opportunity to answer "D," participants are forced to respond differently or abandon the study.
- The distribution of answers between options A, B, and C in round 1 are different than the distribution of answers from round 2.
When researchers force participants to make decisions in closed systems, the participant may react differently than he might have in a semi-open or open system. As a result, the accuracy of data from closed systems may be less than data from open systems.