I came across my old grad school portfolio and saw I had included an article I wrote for one of my classes (circa 11998-1999) on the topic of Web Site Usability Studies. Amazing that 11-12 years later a lot of what I wrote back then is still relevant today. While a full-blown usability study is costly, sometimes it can reveal some things that web analytics can miss. So for your reading enjoyment I am reprinting the original article I wrote. Enjoy!
Web Site Usability Studies
How to do Them, Why to do Them, and things to Consider Before doing Them
Ecommerce. The Internet. The Information Superhighway. The Web.
It’s everywhere you go these days, and everyone is talking about the web. You see web site addresses, commonly known as URLs, on advertisements in magazines, newspapers, even on television. Almost everyone has a web site, including your competition.
How often, though, do you visit a URL and you just can’t see to find the information you came for? You stumble around trying to find which picture means what, what might be a hyperlink and what is just a cool graphic. In the end you find yourself leaving the site in frustration.
Now imagine that you’re the person who built the site, and you get complaint after complaint that people cannot find the information they need. What can you do to find out what the problems are? How do you fix them? Simple. Conduct a usability study of your web site.
What is Usability?
In its simplest terms, usability is the measure of how usable a computer interface is by its potential users. Another way to look at it is, how easy is the interface to initially learn and to remember how to use, how efficient is it to use, how many errors it produces, and the users overall satisfaction in interacting with the interface. This is according to Jakob Nielsen, as stated in his 1994 book “Usability Engineering.”
The same rules that apply to a software or application interface generally apply to web sites as well. Users need to be able to navigate a web site to be able to get to the information they need — quickly. For that to happen, the site’s interface has to be usable.
So you aren’t sure if your site is usable and you want to conduct a usability study? Today everyone and their Uncle Bob seems to be a usability expert. If you delve into these self-proclaimed experts’ methodology, however, you will typically find their studies lacking any reliability or validity.
A properly conducted usability study is done in a very scientific manner, to be able to get the most reliable results. The more reliable your results are, the better analysis you can do of what is right or wrong with your site’s usability. The study may even point toward possible solutions.
The first and most important issue to consider is why conduct a usability study. Simple — it goes straight to your company’s bottom line. If your customers have problems using your company’s web site and aren’t able to track down the information they are looking for, they will very likely go to your competition’s web site. If they find the information more quickly and easily there, your company may have just lost a customer.
The web is congested with companies. Even small mom and pop stores have web sites. It is almost expected by customers to find your company on the web. If your web site isn’t usable to your customers, it’s almost certain they will take their business elsewhere.
The following circumstances are good indicators of when you need to conduct a usability study:
- Are you developing your company’s very first site?
- Has your company had a web site up for several years?
- Are you part of a team given the task of redesigning a pre-existing web site?
For a new site, or the redesign of an old site, conduct a study before launch. If your company has a site and does not plan to do a redesign, look at customer feedback. If you are getting a lot of negative feedback, or questions asking for specific information because customers cannot find it on the web site, conducting a usability study will help to analyze possible problems and point to possible solutions as well.
Your web site can have many purposes. Ecommerce. Information. Education. If can even be a marketing or lead generation tool to bring in new sales. This is another key issue to consider before conducting a usability study, especially if you are launching a new site. Also evaluating, or re-evaluating, your web site’s target audience is essential.
Even if you have an established site, it is a good idea to re-evaluate your web sites purpose and target audience before conducting a usability study. This will help focus your study and ensure that you pull subjects from the target population.
Planning the Study
The first step in planning your usability study is to define exactly whom you are targeting. Are they high-tech gurus or are they average Janes? Are they college educated or high school drop-outs? What other demographics do you feel is important to consider for your company?Once you have defined who your typical target audience member is, you can better select your subjects for the study.
Usability studies are not highly statistically valid studies, and therefore you will not be pulling a random sample of people from the general population. You must understand what audience you want to target, and hat defines that audience, to be able to conduct a study that accurately reflects the usability of your web site.
Before conducting the study, plan it carefully to make sure it’s conducted properly. Usability studies can be costly if done properly, so good planning to determine the budge for the project ahead of time, and determining the cost of how you want to conduct the study is an essential early step.
The ideal setting for a study is a professionally setup usability lab. A lab would include a room for the subject to sit in, equipped with a computer, multiple video cameras, a microphone, and a 2-way mirror. In an ideal world you would also include eye-tracking equipment as well, but that generally becomes too cost prohibitive. Labs like this are costly to set up. One option would be to have a professional usability consulting agency, who would have such a lab already set up, to conduct the actual study. If this option also turns out to be too costly, just setting up a subject in a quiet room with a microphone and recording device, observing their actions during the study and keeping an accurate log may also be sufficient.
You must next determine how many subjects you wish to study. According to Nielson, approximately five to ten subjects are ideal for such a study, and in fact the responses of more than three should consistently find the same usability problems, if any exist, with your web site. Also when calling in subjects for such a study, some sort of any incentive may be necessary. Definitely refreshments, and possibly a monetary inducement, should be offered. If you work for a larger company, another possible way to get subjects for a study could be to offer an entry into a drawing for a prize, such as a digital camera.
The final consideration while planning the usability study is the actual questionnaire to be used for the study.
This questionnaire should consist of three sections. The first section should be a small survey to determine the subjects’ preexisting knowledge of your company, its products, and anything else you wish to find out before they run through the main part of the test. The second section should have five to ten scenarios that the subjects have to complete by navigating through your web site. The final section should be a debriefing survey to determine what they might have learned from the scenario portion, to get demographic information, and to provide space for some open-ended comments about the web site in general.
Once you have defined your target audience, worked out your budget, planned all the details of your study and written your questionnaire, you are ready to move on to the implementation phase of the study.
Conducting the Study
Actually conducting the study could take a few days or a few weeks depending on how many subjects you’ve decided to test, as well as their schedules, availability if the equipment you may be using, and so on.
For a study like this, only test one subject at a time, unless you have multiple usability labs set up, and people to staff them. If you are offering an incentive, this should be taken care of first before beginning the actual study. Then let your subject sit down, and explain to them what you expect them to do for the first part of the study. When they have completed the pre-survey, guide the subject to the computer they will be using for the scenario portion of the study. Each subject must get the exact same instructions, worded the same way, to help preserve the reliability of the study.
If the subjects take more than two or three minutes to try and complete a scenario, instruct them to move on to the next scenario. Subjects may get frustrated and possibly upset if they are allowed or required to try and solve every scenario to completion beyond two to three minutes.
Once the subjects have completed the scenarios, lead them away from the computer to complete the post-survey section. Once this has been completed, dismiss each subject.
Interpreting and Reporting the Results
Once you have completed all testing in your study, sit down and analyze the data. Depending on what sorts of statistical analysis you want to run, you can either lay out your results in a simple spreadsheet program, or use a professional statistical software package. The method you choose depends largely on how you wrote your survey and how detailed a report you wish to give.
Once you have run all of your statistical analysis, you need to analyze the results. Look for any difference in pre-existing knowledge between your subjects, and if that has any correlation to the results of the scenario section. Also look for correlations with the post-survey. Do demographics correlate in any interesting way?
The scenario section can’t be run through a statistical analysis, nor can any open-ended questions or comment areas, so the must be analyzed manually. Do any of the scenarios point to any obvious problems with the web site. What do the open-ended comments tell you? Do they point to any consistent problems? Do subjects indicate alternatives that could point to a solution?
Once you have answered these and many other questions that will come up in the analysis, write a report to share with your web design team and possibly executive management. The report should also include any possible solutions to fix problems discovered about the web site.