Re-designing a usability testing plan

Usability testing for a website measures how easy it is to use, how accessible it is, and how accurately it reflects the users' expectations through a series of questions and tasks, followed by a close observation. The biggest difference between the traditional questionnaire is that in usability testing, the tester carefully watches how the user actually, er, uses the website. Observing the real behavior as opposed to listening to how the user thinks he or she behaves, provides truly useful data.

Here I examine the first draft of usability testing plan for the NJIT website (a technology university based on New Jersey) and makes suggestions for the plan. The testing plan is structured in two parts: the procedure part and the question/task part. I will also examine how the testing plan can be applied to other commercial sites, such as Amazon.com.

The Procedure: Before

1. Meet participant, introduce yourself, and thank them for their participation.
2. Record demographic data.
3. Briefly explain the purpose of the session.
4. Ask participant to fill out the Participant Consent form.
5. Start the interview with general questions.
6. Ask specific questions from the NJIT Usability Testing Questions document.
7. After the last question is answered, thank the participant.
8. Escort the participant out of the testing room.

The Procedure: After

Conduct the following four tasks even before the usability test takes place.

• Introduce yourself.
• Ask participant about his demographic data.
• Ask general questions.
• Ask him to fill out the participant consent form.


1. Meet participant and thank them for their participation.
2. Ask specific questions from the NJIT Usability Testing Questions document.
3. After the last question is answered, thank the participant and escort him or her out of the testing room.

Summary of changes in the Procedure

The point of the above changes is to save time, and to make the experiment more spontaneous.
Saving time does not need explanation - we all have equally 24 hours, and spending a participant’s precious on-site time for “fixed” information is a waste - they should be completed even before the usability testing occurs, together with the consent form agreement and in the participant’s leisure time.

Making the experiment more spontaneous is to collect genuine response from the participant. If he or she spends longer pre-test time in the usability testing room filling out forms and answering questions, he or she might absorb expectations and might even have an insight about how the website is constructed. Needless to say, the most accurate responses are collected when the participant’s attention level is still high (unless the test’s purpose is to measure the website’s performance toward people with fatigue).

The testing questions: Before

Opening questions

• Feedback on the big photo image(s) on the homepage
• Feedback on the news & events shown on the homepage
• Feedback on the special-interest features
• Feedback on the general mix of content on the page
• Feedback on use of colors, and visual “balance” of design elements
• What they would expect to find behind each Primary Nav links

Tasks

• Apply online
• Find a list of available majors
• Find information about NJIT academic focus (what makes NJIT different from other schools in the region)
• Find out about the student body; would they fit in at NJIT?
• Find directions to NJIT
• Find information on student clubs 
• Schedule a visit to NJIT
• Find information on financial aid and tuition cost
• Find a list of places to go and things to do in Newark/NYC region

Closing questions

• From an internal page: How would you get back to the homepage?
• Is the breadcrumb navigation apparent? Is it clear how to use it?
• Do you have any other observations about the NJIT website?

The testing questions: After

Opening questions

• Feedback on visual appeal and design
• Feedback on the news & events
• Feedback on the special-interest features
• Feedback on the general mix of content on the page
• What they would expect to find behind each Primary Nav links
• Feedback on navigation including menu and breadcrumbs

Tasks

• Apply online
• Find a list of available majors
• Find out about the student body; would they fit in at NJIT?
• Find directions to NJIT
• Find information on student clubs 
• Schedule a visit to NJIT
• Find information on financial aid and tuition cost
• Find a list of places to go and things to do in Newark/NYC region
  

Other observations

• Do you have any other observations about the NJIT website or any other issue?

Summary of the changes in the testing questions structure

I will change the three-part structure into a two-part structure plus one closing question (Do you have any other observations?). Unless absolutely necessary, all questions should be asked at the beginning when the participant is highly alerted. The exception is the observations request, and it does not have to be limited to the website design. The participant might have some opinions about other issues such as the usability testing scheme, tester’s attitude, and university website in general. Any of these information would be useful for both the website and the usability testing.

I added a feedback on navigation and merged the breadcrumb question from the original “Closing questions” in it.

Also, I deleted the academic focus question which sounds impressive but is ultimately pointless: how many people read organization policies to find out how unique the university truly is? Instead, more attention should be paid to where the users actually go to find out their true interest.

Applying the usability test documents to a commercial site

All the previous suggestions were made for a university website, but most of the items also apply to commercial website as well. Therefore I would not change the overall structure, but will add the following items in the Tasks corner. Of course, each question or task should be modified according to the actual information in the commercial website.

• Complete a purchase of a randomly selected product
• Enter a user comment/review
• Reach customer support desk and ask imaginary questions
  

The completion of the purchase is about testing the e-commerce functionality thoroughly. Unlike university, customer are likely to complete the purchase in a single action, and when they cannot do that, they are far more likely to abandon the website and look for alternative options.

Entering a user comment/review is to test how much the website encourages collaboration  with and input from the users. Everybody say they treat users as collaborators, not just buyers, but do they live up to that hype?

Reaching customer support desk is one of the most important factors in improving customer satisfaction. Here, time is the critical factor - when observing the time, the testers should count the time the participant took to complete each transaction (this is especially true for reaching out for the support desk, where real customers likely have low tolerance). A commercial website is about helping customers get what they want as fast as possible.

Google site: How it has evolved

I have a new website built on Google sites service: free (as usual), simple, stable, and also flexible enough to allow creating surprisingly decent website. One such "example" website is Steegle - a website offering tips for building Google sites on which itself is built. The author not only contributes frequently to Google Support forum and offers free tips but also provides paid service for advanced consulting. Many features in my own websites were taken from Steegle's tips.

My website, 5-minute courses, offers insights into Asian (mainly Japanese) cultures, especially their "peculiar" aspects which baffle people who are not familiar with them.

 

The course is offered as a small presentation, with comment facilitation at the bottom. The right side bar provides site structure, search box, contact information, and RSS link. The top logo also serves as the link to the home page, and the buttons below the logo offers shortcuts to major sections. Below is the original blueprint of the website as I planned in September.

The goal of the original plan was to provide a web-based training system for an imaginary global consulting company with the following traits:

• Visual-based: non-verbal and reflective learning rather than verbal and active learning
  
• Concept-driven: theories and concepts rather than examples and linear steps
  
• Optimized for global team: can be taken anytime, anywhere

Is the current website meeting the original goals (or even exceeding them)? I will examine the effects and see what was accomplished and what is being left.

Features
Some features have stayed the same and some did not. Here are the quick summary of features, realized or not.

• Realized: Course title, Course content, Controls, User comments, Search button, What's new, Course structure
• Not realized: Login information, Ratings
  

A majority of the functions were realized. I did not include the Login function in the website and instead chose to make it more "open" to end users, by allowing anybody with Google account to comment. Originally commenting in Google sites were allowed only to "collaborators" registered by the web master, which could have worked as a pseudo-login system. The user rating function is something that wasn't realized due to lack of technical expertise or knowledge - by default Google sites does not offer such function (yet).

Design aspects

Let's take a look at each of the realized element and see how it was designed and how effective (or ineffective) it might be.

Template design

I took the Solitude: Olive template shown to the right, took the header background off, and left the green color for texts. I also changed all font into Trebuchet. I wanted a bare-bone design with minimum shades of green (nicely realized in the original template). Although the target users are visual-oriented, I eliminated the background color to let them focus more on the content - too much design element, if not placed carefully, might lead to visual noise.

Logo & Menu design

The logo and menu are clickable image texts. They are designed to match the bare-boness of the website and its color. Functional-wise they work fine, but unfortunately the image color does not exactly match the text color (see below: icon color and title color). The logo/icon font does not match the main content either, but that is intended. Other colors and textures were not added to avoid visual noise, as in the background color: future revision should add more design variations without adding noises.

The menu icon was added also to offer a quick high-level site map for the target users who prefer to have an overview of the situation before going into details.

Main content design

The presentation was created in Google Document (Presentation) because it offered (1) easy integration with Google site and (2) Came with embedded playback control and (3) modifying the original presentation gets reflected to the Google site in real time. Presentation / Video sharing sites such as Slideshare offered similar embedding function, but none could offer the (3) real-time option. Below is the close-up of the embedded control playback.
The content itself -- presentations -- are text-based, which seems not fit the visual-oriented target users, but are designed so that each presentation only contains a handful of words. The presentations are structured so that the main argument (concept) is presented first, followed by examples.

Comment design
Google sites is by default equipped by commenting function but it is not open to anyone who visits the website: the web administrator need to give permission as "collaborators" of the web. This might work well in educational usage (commenting can only be done by students) or internal training as proposed in the original plan. This time, to (1) encourage more people to comment and to (2) test an "unorthodox" comment field mechanism introduced in the aforementioned Steegle website, a more open (anybody with Google account can comment) field is embedded as a widget. As shown in the snapshot below, when a user is logged in to his or her Google account, the profile image automatically shows up, creating an instant identity field. This feature should work especially well with the target users -- visual-oriented and international -- where identifying people by photos is considered much easier than names in texts (example: it is very hard for people from U.S. to identify the sex of the Chinese name Bai Ling).

Search design

Google site by default offers a search box. I have modified the option to add a normal Google search option in addition to the default site search (see right). This is a decision that made the original design more complicated (thus less "visual-oriented") in return of increased convenience.




What's new design

Google site by default offers a page template that can be used for posting news and updates. The What's new page offers a quick overview of what is going on in the website -- it suits the target customers who prefer to grasp the "high concept" before going into the details.









(Bonus) About author design

The About corner, the author's page was also meant to experiment with page design. Instead of the default two-column, I used three-column design with equal width, and embedded a Google map showing my address. The gorilla mask was added to add a fun element for visual-oriented target users.