Choosing right: moderated vs. unmoderated usability studies

Usability testing is a UX research methodology. It’s used in a Human-Centred Design (HCD) process to evaluate a digital product by testing it on potential users.

As we test products during the design phase to eliminate usability issues before the development process, we frequently run usability tests – both moderated and unmoderated. As we navigate the complexities in B2B UX research, the distinction between these methodologies becomes not just relevant, but vital.

Through this exploration, my aim is to explain the differences between moderated and unmoderated usability studies, offering you a pathway to informed decision-making that aligns with both your project goals and available resources.

Moderated usability studies.

Moderated usability testing involves the presence of an interviewer (moderator), typically someone from the UX team, and their active involvement in conducting the testing. The moderator gives tasks to the respondent in real time and asks clarifying questions during the test. Such tests can be conducted both virtually and in person, and they create a deep understanding of why the solution works, to inform further iteration.

When to conduct moderated usability studies:

• Detailed feedback and interaction. At the discovery phase of the project, we may want to test initial designs to understand if the suggested solution meets the workflow of users. We also run moderated studies when we want to understand the reasoning behind completing scenarios, and we tend to ask follow-up questions to get deeper insights. For example, a moderator can ask sales reps to perform tasks such as entering a new lead. As participants interact with the CRM, the moderator observes their actions, asking for detailed feedback on the interface, or any difficulties encountered. This direct interaction allows the moderator to probe deeper into the reasons behind user behaviours.
• Complex tasks and user flows. When we design complex workflows, such as web platforms or mobile apps used by B2B clients, we evaluate very niche scenarios. An example can be the evaluation of procurement tasks, where the moderator asks a participant to simulate a full cycle of a procurement job. As the participants navigate through this cycle, the moderator closely observes their interactions with the platform, asking probing questions at each step of their journey.
• Specific user groups or behaviours. There are plenty of use cases when a moderator should be directly involved for the test to succeed. We always moderate accessibility tests, and tests involving participants with disabilities. Another example can be conducting tests in the user’s normal environment, such as in hospitals or on construction sites, where we can also take into consideration the environmental aspects. When analysing a healthcare project, we consider different roles, since nurses and administrative staff may use the functionality of the product in diverse ways.

Limitations:

• Higher costs and time requirements. Moderated sessions can extend from 45 minutes to as much as 1.5 hours, adding to their time-intensive nature. The UX researcher, acting as the moderator, must dedicate time to prepare for the study, conduct it, and then analyse and synthesise the findings. In-person studies often incur greater expenses, including venue hire and participant travel reimbursements. Additionally, the moderator can involve another member of the UX team to take notes and capture further emotional responses and behaviours from the user, providing an alternative perspective. Given their comprehensive nature, moderated studies deliver outcomes that go beyond simple usability improvements. They can include the development of detailed user flows and the analysis of quantitative aspects of the study, such as administering usability questionnaires, post-test, to measure usability metrics.
• Possible moderator bias and observer effect. Potential moderator bias and the observer effect present challenges. The involvement of a moderator might skew participants’ responses. Participants could hesitate to express critical feedback, despite the moderator’s efforts to foster a supportive environment that encourages sharing both positive and negative observations. This openness is crucial for validating initial hypotheses and enhancing usability. The awareness of being observed can also influence participants’ behaviour, affecting the genuineness of their interactions. Although attempts are made to simulate the participant’s natural usage environment, the mere fact of observation implies that their behaviour might not completely reflect their genuine interactions with the product.
• Limited scale due to resource intensity. As previously noted, moderated studies can be undertaken both remotely and in person. Remote studies might exclude participants with restricted internet access or limited technological skills. In-person studies require additional planning, and our experience suggests fewer individuals are inclined to participate in face-to-face sessions, owing to geographical constraints.

Unmoderated usability studies.

Unmoderated usability testing is conducted without the involvement of a moderator. The respondent independently completes a series of tasks, and their actions are recorded for analysis. There are paid platforms to conduct these types of studies. These platforms provide tasks and questions to users which were initially written by a UX researcher and records their responses and actions during every session. Unmoderated tests require more meticulous planning than a moderated study, since you cannot rely on human judgment to adapt the study procedures on the fly.

When to conduct unmoderated usability studies:

• Specific basic task evaluation. Unmoderated tests typically last up to 20 mins where respondents complete few tasks based on the use case, then answer follow-up questions. We can test the user experience of filling out forms, such as sign-up, checkout, or feedback forms. We can also test a new feature (if it’s not complex and we just want to understand first impressions) and naturally observe how the user interacts with the app without needing obtaining deeper details.
• Quick results. Since unmoderated studies can be conducted by all users during the same time, they’re much faster to complete. What’s more, a UX researcher can review them all together and analyse the findings in a consolidated manner. This synchronous data collection and review process significantly shortens the feedback loop, allowing UX researchers to quickly identify patterns, trends, and outliers across all user interactions.
• Affordable budget. Unmoderated studies are typically half as expensive as moderated ones, since both participants and moderators can review them when available. Plus, we can get quick results on small questions such as UI adjustments. By eliminating the need for live facilitators and scheduling individual sessions, these studies significantly reduce planning, correspondence, before-and-after studies, and enable participants to complete them in a suitable time.

Limitations:

• Less control over the test environment. In unmoderated studies users can complete the study whenever they want, and we cannot dictate the rules. Participants might complete tasks in vastly different settings: a quiet office, a bustling coffee shop, or a home filled with distractions. These variations can influence not only the performance and comfort of the participant, but also the quality and reliability of the data collected. What’s more, participants may be more prone to distractions, impacting task focus and completion. For example, respondents may pause the study to answer a phone call, accidentally mute the microphone, or even multi-task during the study, such as checking emails or engaging with social media. These interruptions can significantly alter the way tasks are approached and completed, leading to fragmented or incomplete data.
• Limited opportunity for follow-up or probing questions. Since we cannot engage with participants in real time, to delve deeper into their responses or observe behaviours directly, meticulous planning becomes crucial. This preparation phase often requires more time and consideration, as it’s important to formulate questions that are both specific and, at times, closed-ended to elicit the most informative responses. Crafting these questions with precision aims to compensate for the lack of interactive dialogue, ensuring that the insights are as relevant and actionable as possible. However, this approach inherently limits the depth of insights compared to moderated sessions, where dynamic questioning can uncover unexpected but valuable user feedback.
• Potential for incomplete data or misinterpretation. During an unmoderated study, participants interact with a platform – there is no one to guide them and ask follow-up questions. This lack of interaction can lead to scenarios where participants misinterpret the tasks or provide feedback that lacks context, making it challenging for researchers to draw accurate conclusions. In complex B2B environments, where the specifics of tasks and features are critical, such gaps in understanding can compromise the validity of the study’s findings, making it challenging for UX researchers to identify genuine usability issues versus user errors or misconceptions.