There is plenty of information on the web related to usability testing. The majority of this content encourages UX researchers to collect data exclusively related to what they observed during the testing session. While collecting this type of information is highly valuable and certainly necessary, not enough content on the web encourages UX researchers to use standardized usability questionnaires after each usability test. There are standardized questionnaires that have been specifically designed to assess participants’ perceived usability and satisfaction of products and systems.
Benefits of using standardized questionnaires are:
Quantification: Standardized measurements allow practitioners to report results in finer detail than they could by using only personal judgment.
Scientific generalization: Standardization is key to generalizing a finding from a sample to the greater population.
Communication: It is easier for researchers to communicate findings when referring to standardized metrics.
- Quick Comparisons: By using standardized questionnaires, it’s easy to compare different design iterations throughout the development process.
Perils of not using standardized and systematic metrics are that over time, throughout the project development process, the researcher can become de-sensitized to pertinent usability issues and fail to document them. Furthermore, these questionnaires provide an alternate way for participants to provide their subjective evaluation of a product or system.
Popular standardized questionnaires include: Software Usability Measurement Inventory (SUMI), Post-Study System Usability Questionnaire (PSSUQ), and the System Usability Scale (SUS). What is the difference between these questionnaires and which should you use? See for yourself:
Software Usability Measurement Inventory (SUMI)
The SUMI is a 50-item questionnaire that measures users’ perception of the Efficiency, Affect, Helpfulness, Control and Learnability of a system. The SUMI is highly reliable (.92) and is currently available in 12 languages. Using the SUMI requires purchasing a license that is approximately $700 a month.
Sample SUMI question
The system responds too slowly to inputs: Agree Undecided Disagree
Post-Study Usability Questionnaire (PSSUQ)
The PSSUQ is a 16-item survey that measures users’ perceived satisfaction with a product or system. Obtaining an overall satisfaction score is done by averaging the four sub-scales of System Quality (the average of items 1-6), Information Quality (the average of items 7-12), and Interface Quality (the average of items 13-16). The PSSUQ is highly reliable (.94) and is entirely free.
The PSSUQ Survey
The System Usability Scale (SUS)
The SUS is perhaps the most popular standardized usability questionnaire, accounting for approximately 43% of unpublished usability studies. It is a 10 item questionnaire designed to measure users’ perceived usability of a product or system. The SUS is highly reliable (.91) and is entirely free.
The SUS Survey
To score the SUS, subtract the scale position from 1 on all oddly numbered items, and subtract 5 from the scale position on all evenly numbered items, then multiply the sum of all items by 2.5 to get an overall SUS score that ranges from 0-100.
Which Questionnaire Do I Use?
Determining which questionnaire to use depends on budgets and project goals. Use the SUMI if the project budget permits and if measuring users’ perceptions of control when using a system is important to the project. We think the other dimensions measured by the SUMI can be objectively measured by other means and at a cheaper price, and there will be more on this in an upcoming webinar.
Use the PSSUQ if measuring users’ satisfaction is important to the project. However, the PSSUQ should be used carefully as it is susceptible to the “acquiesce bias”– which is the fact that people are more likely to agree with a statement than to disagree with it. Therefore, it is important that questionnaires carefully mix positively and negatively worded questions. All questions in the PSSUQ are positively worded.
Use the SUS if measuring the users’ perceived usability is important to the project. An added bonus to using the SUS is that recent psychometric analyses shows that items 4 and 10 reliably measure the dimension of perceived “learnability”. Therefore, by measuring these items separately, it’s possible to gain an understanding of users’ perceived usability and learnability of the product or system being studied.