Large-scale Qualitative Research
Evaluating the user experience against competitors:
A Benchmarking Study
Our client wanted to evaluate the usability of their product compared to two primary competitors in the field.
We gathered qualitative data and usability metrics from 90 users to answer their research questions.
Through remote, moderated interviews, Amplinate evaluated the usability of the product and competitors’ products through a pre-established set of tasks.
Our results enabled our client to compare between competitors and product iterations, capture performance improvements and declines, and identify target problem areas to be addressed.
This research was part of ongoing benchmarking studies conducted by the client and informed whether the product was meeting key long-term goals and scores.
Research Goal & Questions
The client needed to gather baseline usability metrics for core experience tasks for their product and two primary competitor products. The following measures were captured for 12 tasks:
Objective Measures: Pass/Fail, Time on Task
Subjective Measures: Ease of Use, Visual Appeal
System Measures: System Usability Scale (SUS)
Our Approach
-
N=90 participants from the US
Daily to monthly users of the specific type of software tested
-
90-minute remote, moderated sessions
2 platforms/products tested per session (counterbalanced)
Participants completed up to 12 tasks per platform while following the think aloud protocol
-
15 weeks
Example Scores Matrix from the final deliverable
Impact
Suggested UI improvement to enhance discoverability and usability was added to customer roadmap.
High-impact presentation to UXR stakeholders, Design, PM, and the Partner Head of UX research.
The client had a goal of a SUS score of 85 or above. We measured their progress toward that goal and provided specific guidance on improvements that would steer them to achieving that goal.