How To Moderate Effectively In Usability Research

1. Deliver findings that achieve the research goals.
2. Give the participant enough “leash” for the study to be as representative of their real-world experience as possible.
3. Keep the participant within the scope of the study in order to achieve the research objectives as efficiently as possible.

How Moderating Can Affect Research Findings

The way in which the moderator facilitates the research session is critical. In the lab, most findings occur as a result of two activities: observing what the participant does and asking (probing) questions. It turns out that the balance between these two activities is critical. In my experience, the best findings in a research session are concentrated in two areas: what the participant does and the feedback they volunteer — unprompted. These are the gems, the golden nuggets that come closest to revealing how the user would experience the design realistically.

As moderators, we often concern ourselves with what questions to ask (according to our own agenda), while being careful not to ask leading questions. But for effective moderation, this is just the beginning. Moderators have the ability — and the power — to thwart research goals by what types of questions they ask and when they ask them. A certain type of question, as well as the timing of the question, can have a powerful influence on what happens during the session and, consequently, on the findings.

By nature, moderators are in the business of asking questions. Their job is to probe. So it’s tempting to ask too many questions, to ask the wrong question and to ask a question at the wrong time. The problem with the last two is that it can shape what users think about (in the moment) and even make them think of things they wouldn’t ordinarily think about — thereby potentially taking the session in a direction it wouldn’t normally (or realistically) have gone.

Remember that participants are compensated for their feedback. Because of this, they will readily provide an answer to any question we ask and will willingly follow our lead. Skewed findings are a serious issue, of course, because they jeopardize our ability to identify the true problems with a design and can lead to faulty decision-making by our business partners.

I once observed a moderator who believed the best way to get reliable results from a group of users was to ask each one the very same questions in the very same way. Each research session, then, became a rigid question-and-answer period, in which users interacted relatively little with the actual website that was to be tested. The key in usability testing is to observe behavior. It’s what people do that is most revealing. To this end, the moderator stands much more to gain by getting out of the way and allowing participants to interact with the website as they normally would, rather than interjecting with too many questions along the way.

The beauty of qualitative research is that it enables you to understand where and why users are encountering problems with a design. Each user is different, and the moderator should have ample opportunity to probe the particular experience and thoughts of each user. To derive the greatest insights, though, the moderator needs to be flexible and astute enough to know how to get the most from each research session.

Facilitating Effectively

To facilitate effectively, a moderator needs many skills and has to make many considerations. The scope of this article is limited to discussing a few that I’ve found have particular influence on research findings.

Starting the Research Session

How you start a research session is important. For a long time, I was accustomed to starting sessions with five or six questions that were related to the topic of the research initiative. If the study was geared to a certain type of investment account or a certain type of trading activity, for example, I might start with a series of background questions to learn more about the participant’s history or experience with the topic.

A potential problem with this, however, is that participants can glean from such questions what the study is about and what the scope of the session will be. Many, then, are quite happy to tailor their behavior and responses to what they think we want from them. Again, participants are compensated for their feedback, so they will want to please and be agreeable, to appear competent and knowledgeable and to be attentive to what we want from them.

In light of this, I have since given more thought to how I start research sessions. Now, I typically ask two to three very general questions, mainly to build rapport with the participant.

Probing Questions

Research sessions most reflect the true user experience when participants are allowed to interact with a design the way they normally would. The main job of the moderator, then, is to observe what the participant does and to note any feedback they volunteer (unprompted) along the way. I usually save most of my questions for the end of the session, when I can circle back and review the participant’s experience, or I will ask short probing questions along the way.

When probing, the moderator takes the participant’s lead and simply queries for clarification or for deeper insight into what the participant has already done or voluntarily verbalized. The scope of probing questions should always be limited to what the participant has already voiced or experienced. For example, if a participant says, “This is really great,” I might probe with, “What makes it great?” If a participant says a word or phrase that could be interpreted in multiple ways, I might ask for clarification. I wouldn’t ask a completely unrelated question that would take their line of thinking in a different direction.

One way to gauge whether your question is appropriate is to consider whether you’re asking about something they’ve already said or done. If not, then think twice about whether it’s really appropriate at this point. Remember that you can circle back later in the session to ask the question.

Another reason why probing questions are important is that they help you to check your assumptions. Being human, we are always assuming things, often without consciously realizing it. When a participant uses a particular term or offers positive feedback, it’s easy to assume that you understand what they mean or that you know why they’re pleased with the design (without even realizing it). Assumptions that are not validated by the participant, however, contribute significantly to faulty research findings.

Open-Ended vs. Close-Ended Questions

Note also that probing questions are open-ended, not close-ended. Close-ended questions yield a “yes” or “no” answer, while open-ended questions invite a broader response that yields deeper insight. Close-ended questions run the risk of directing or leading the interaction, with the likelihood of tainting your findings.

Leading Questions

As mentioned at the beginning of this article, there is a constant tension between allowing the participant to be in the driver’s seat and keeping them within the scope of the study. Your goal might be to see whether participants discover a feature or element of the UI that you’re not sure is immediately evident, for example.

In this situation, I’ve heard moderators ask questions like, “Do you think you could do [such and such a thing] on this website?” But why would someone ask whether something is possible unless they already knew it is possible? The question gives the answer away.

Instead of asking the question in this way, think of other ways to get at the answer. Users will often eventually discover the functionality on their own. And if they don’t, perhaps you could devise a task or ask some other type of question that helps them to stumble across the functionality. For example, if you want to see whether a user will discover that a chart is interactive by mousing over the chart, perhaps you could devise a task that requires the user to pass the mouse over that area of the screen — maybe by asking them about a particular data element in the chart.

Achieving Balance

In any given research session, the moderator faces the constant challenge of determining what to ask and when, on the one hand, and when to simply stay quiet and observe, on the other. The truth is that there is no one perfect way to moderate a study. Just like in design, there are multiple ways to do it, and each has its strengths and weaknesses (and potential outcome). The challenge is to think on your feet so that you can decide in the moment whether to ask a question or to hold off and simply observe.

More often than not, I’ve found that when I wait and allow for silence, the participant ends up expressing something of value. Give participants enough time to absorb the design and gather their thoughts. (This will often take more time than you expect because you’re already very familiar with the design.)

I’ve seen moderators ask too many questions, which leads the participant to behave less naturally with the website being tested. Instead of engaging with the website, they are unsure whether to move forward or wait for the moderator’s next directive. This can cause the session to become rather disjointed and make it difficult to ascertain how the session would have gone had the moderator done more observing than questioning.

Helping Participants

Participants will sometimes get stuck during a study. They might encounter something unintuitive and not know how to continue. In these situations, it’s not uncommon for the participant to ask for help, such as an explanation. But when is it appropriate for the moderator to help the participant get unstuck?

My rule of thumb is to provide an answer when it would help me achieve the research goals. Would I learn more by allowing the participant to experience some discomfort and asking how they would expect it to work, or would I learn more simply by providing an answer and getting their thoughts on the design’s intent? Most often, I learn more when I don’t answer too soon (or at all).

Managing Your Business Partners

Moderating effectively is also about managing your business partners and assessing what they want to take away from the study. They will often have an agenda. They have difficult decisions to make, and if they can leverage research findings to make them easier (or to back up their point of view), they will.

Your job is to make sure that the research findings are “real” and to manage what your business partners take away from the study. I’ve seen business partners focus on their own personal pet peeves during research sessions and specifically look for participant behavior and responses that support their agenda. When you probe about things that really don’t matter to the participant (but that matter to the business), your business partners might attach undue importance to the answers in order to make their own job and decisions easier.

Be cognizant of the dynamics of human psychology — both in the participants and in your business partners. Moderating effectively requires considerable skill!

Conclusion

How you moderate will have a significant impact on the quality of your research findings. The goal, of course, is to get realistic findings — findings that represent as closely as possible the real-world experience of users with the design being tested. Keep in mind the following variables:

• Questions asked at the start of a research session could unduly influence the participant’s behavior and thought process later on.
• Ask probing questions that don’t lead the participant in a direction they wouldn’t otherwise have gone.
• Your own assumptions could inadvertently affect the findings.
• Certain types of questions give the answer away.
• Responding to the participant’s questions can affect your ability to understand their needs and expectations.

An effective moderator understands how the nature and timing of questions can influence — and sometimes even drive — the outcome. Moderating well means being able to think decisively in the moment and to make critical decisions about what to ask and when.

Another factor is being aware of the various players in a research study and how their respective goals can affect the integrity of the research protocol. The moderator’s goal is to get realistic findings; the user’s goal is to appear competent and to please; and the business partners’ goal is to leverage the study to make their jobs easier — which is why they sometimes encourage moderators to ask participants to speculate about what they would do in a particular scenario. (Generally, asking participants what they would do is not productive because people are notoriously poor at predicting behavior.)

A skillful moderator is adept at managing these potentially conflicting goals in order to ensure the integrity of the research and equally adept at ensuring that their findings appropriately inform their business partners’ decisions.