Saul Greenberg

back to HCI Lecture Modules

Qualitative Methods

Evaluating interfaces with users' involvement form the basis for usability studies. A good evaluation process means that designers will catch major problems (and successes!) early on, with lesser problems being ironed out as the interface is being refined. Fortunately, there are many low cost evaluation methods for discovering usability problems. Most of these are based on observing people as they use your system to perform particular tasks.

While design is complex, a few basic principles can go a long way. This module summarizes design principles from Don Norman's book: The Design of Everyday Things. It motivates by showing problematic and useless designs, and then turns to basic design principles.

Powerpoint lectures

Topics Covered

  • Evaluating interfaces with Users
    • Why bother?
    • Natural vs laboratory approaches
  • Qualitative evaluation methods
    • Introspection
    • Direct observation: think aloud and co-discovery learning
    • Query techniques: interviews and questionnaires
    • Continuous evaluation: user feedback and field studies
  • Ethics

Suggested Readings

  1. Usability testing. Chapter 6 in Nielsen, J. Usability Engineering, Academic Press. (1993)
  2. Usability assessment methods beyond testing. Chapter 7 in Nielsen, J. Usability Engineering, Academic Press. (1993)
  3. User Observation: Guidelines for Apple Developers, by Gomoll, Kathleen & Nicol, Anne., Apple Inc., January, (1990)

In-Class Activities

I have found that performing usability studies in class is an excellent use of time. I do several of them, each emphasizing slightly different methods of performing a usability study.

  1. Initial conceptual model formation plus think aloud of a paper prototype. I place a stylized picture of a Cannon fax machine on the overhead (included in the lecture) and ask them to explain the meaning of labels and controls. I then asked them to pretend they are sending a one page fax, and have them use think aloud to say what controls they are selecting and why.
  2. Think aloud on a physical artifact. I have a student volunteer do a think-aloud exercise as they try to display a slide on an overhead projector rigged with a burnt bulb---it usually takes the student 20 minutes to discover the problem and realize that the projector has a spare bulb that they can switch to. I also have them try to change the bulb, and most have difficulties figuring out how to open the projector to reach the bulb. The class, who are are taking notes, then critically analyze the design of the overhead projector (relating back to Norman's design principles of everyday things) and suggest improvements. As most recommendations are simple changes to the plastic overhead case, a "better" projector could probably be built for the exact same price. The class often wonders why the manufacturer (3M) had never bothered doing this simple exercise!
  3. Constructive interaction/co-discovery learning on an existing computer system. A pair of students are given a PC (the display is linked to a large screen so the class can see what is going on). One student is usually familiar with conventional GUI file systems, while another is a novice to it. The novice is the driver, while the 'expert' is the coach. Students are asked to do a few tasks on well known systems; one year we had people try to figure out the image placement system in Microsoft Word. While the scenario is somewhat rigged to bring out the interfaces worst features, it is surprising how much difficulty students have doing even the most basic things. I also have done this for the IBM RealPhone Prototype
  4. Case Study - Get Real with the IBM RealPhone. Depending on the class, I either do a usability study of the IBM Real Phone (see below) in class, or have the students do it as an exercise, or as a fusion between the two.

Case Study - Get Real with the IBM RealPhone

IBM Real Objects was an experimental concept / system that recreates real world physical designs as interface designs. We use their IBM Real Phone system as a platform to conduct a usability study. See the presentation above for an overview. These are the materials you will need to actually perform the usability study yourself.

  • The Real Phone software and instructions, as an installable .exe
  • Download the full handout, which contains:
    • evaluator's instructions and tasks
    • consent form
    • Pre-test, During-test and Post-test questions
    • methods for running a basic observational study
    • discussion of guidelines for user observation
    • IBM Real Phone Instruction Sheet and their test-drive instructions
    • Isys' Hall of Shame critique of RealPhone (compare your results with theirs)

Additional Readings

  1. Usability Engineering, Nielsen, J. (1993) p273-4, Academic Press.
    • while it covers much more than usability evaluation, it gives a great well-rounded picture of this area
  2. How to design usable systems. by Gould, J. In Readings in Human Computer Interaction: Towards the Year 2000 (2nd Edition). Baecker, R., Grudin, J., Buxton, W., and Greenberg, S. (1995). Morgan-Kaufmann.
    • a seminal introduction to design and usability testing and the many practical ways it can be done
  3. Risks of Quantitative Studies. Blog post by Nielsen, J. March 1, 2004
  4. A Practical Guide to Usability Testing. Revised Edition. by Dumas, J.S. and Redish, J.C. (1999).

Videos

  1. Time Lapse of Manual Image Clustering (Grounded Theory) by Dustin Freeman
    • shows how he clustered ~275 images from a study over the course of 1 hour and 48 minutes. Sped up here 60x.