Re-designing a usability testing plan

Usability testing for a website measures how easy it is to use, how accessible it is, and how accurately it reflects the users' expectations through a series of questions and tasks, followed by a close observation. The biggest difference between the traditional questionnaire is that in usability testing, the tester carefully watches how the user actually, er, uses the website. Observing the real behavior as opposed to listening to how the user thinks he or she behaves, provides truly useful data.

Here I examine the first draft of usability testing plan for the NJIT website (a technology university based on New Jersey) and makes suggestions for the plan. The testing plan is structured in two parts: the procedure part and the question/task part. I will also examine how the testing plan can be applied to other commercial sites, such as Amazon.com.

The Procedure: Before

1. Meet participant, introduce yourself, and thank them for their participation.
2. Record demographic data.
3. Briefly explain the purpose of the session.
4. Ask participant to fill out the Participant Consent form.
5. Start the interview with general questions.
6. Ask specific questions from the NJIT Usability Testing Questions document.
7. After the last question is answered, thank the participant.
8. Escort the participant out of the testing room.

The Procedure: After

Conduct the following four tasks even before the usability test takes place.

• Introduce yourself.
• Ask participant about his demographic data.
• Ask general questions.
• Ask him to fill out the participant consent form.


1. Meet participant and thank them for their participation.
2. Ask specific questions from the NJIT Usability Testing Questions document.
3. After the last question is answered, thank the participant and escort him or her out of the testing room.

Summary of changes in the Procedure

The point of the above changes is to save time, and to make the experiment more spontaneous.
Saving time does not need explanation - we all have equally 24 hours, and spending a participant’s precious on-site time for “fixed” information is a waste - they should be completed even before the usability testing occurs, together with the consent form agreement and in the participant’s leisure time.

Making the experiment more spontaneous is to collect genuine response from the participant. If he or she spends longer pre-test time in the usability testing room filling out forms and answering questions, he or she might absorb expectations and might even have an insight about how the website is constructed. Needless to say, the most accurate responses are collected when the participant’s attention level is still high (unless the test’s purpose is to measure the website’s performance toward people with fatigue).

The testing questions: Before

Opening questions

• Feedback on the big photo image(s) on the homepage
• Feedback on the news & events shown on the homepage
• Feedback on the special-interest features
• Feedback on the general mix of content on the page
• Feedback on use of colors, and visual “balance” of design elements
• What they would expect to find behind each Primary Nav links

Tasks

• Apply online
• Find a list of available majors
• Find information about NJIT academic focus (what makes NJIT different from other schools in the region)
• Find out about the student body; would they fit in at NJIT?
• Find directions to NJIT
• Find information on student clubs 
• Schedule a visit to NJIT
• Find information on financial aid and tuition cost
• Find a list of places to go and things to do in Newark/NYC region

Closing questions

• From an internal page: How would you get back to the homepage?
• Is the breadcrumb navigation apparent? Is it clear how to use it?
• Do you have any other observations about the NJIT website?

The testing questions: After

Opening questions

• Feedback on visual appeal and design
• Feedback on the news & events
• Feedback on the special-interest features
• Feedback on the general mix of content on the page
• What they would expect to find behind each Primary Nav links
• Feedback on navigation including menu and breadcrumbs

Tasks

• Apply online
• Find a list of available majors
• Find out about the student body; would they fit in at NJIT?
• Find directions to NJIT
• Find information on student clubs 
• Schedule a visit to NJIT
• Find information on financial aid and tuition cost
• Find a list of places to go and things to do in Newark/NYC region
  

Other observations

• Do you have any other observations about the NJIT website or any other issue?

Summary of the changes in the testing questions structure

I will change the three-part structure into a two-part structure plus one closing question (Do you have any other observations?). Unless absolutely necessary, all questions should be asked at the beginning when the participant is highly alerted. The exception is the observations request, and it does not have to be limited to the website design. The participant might have some opinions about other issues such as the usability testing scheme, tester’s attitude, and university website in general. Any of these information would be useful for both the website and the usability testing.

I added a feedback on navigation and merged the breadcrumb question from the original “Closing questions” in it.

Also, I deleted the academic focus question which sounds impressive but is ultimately pointless: how many people read organization policies to find out how unique the university truly is? Instead, more attention should be paid to where the users actually go to find out their true interest.

Applying the usability test documents to a commercial site

All the previous suggestions were made for a university website, but most of the items also apply to commercial website as well. Therefore I would not change the overall structure, but will add the following items in the Tasks corner. Of course, each question or task should be modified according to the actual information in the commercial website.

• Complete a purchase of a randomly selected product
• Enter a user comment/review
• Reach customer support desk and ask imaginary questions
  

The completion of the purchase is about testing the e-commerce functionality thoroughly. Unlike university, customer are likely to complete the purchase in a single action, and when they cannot do that, they are far more likely to abandon the website and look for alternative options.

Entering a user comment/review is to test how much the website encourages collaboration  with and input from the users. Everybody say they treat users as collaborators, not just buyers, but do they live up to that hype?

Reaching customer support desk is one of the most important factors in improving customer satisfaction. Here, time is the critical factor - when observing the time, the testers should count the time the participant took to complete each transaction (this is especially true for reaching out for the support desk, where real customers likely have low tolerance). A commercial website is about helping customers get what they want as fast as possible.