Saul Greenberg

Readings in HCI Evaluation Methodologies

These readings were used in the course CPSC 681 - Research Methods in HCI and as a general resource.

Overview

Evaluation is essential in interaction design, where it is used for many purposes and in different development stages of the interface lifecycle. No one evaluation method suffices. Thus the professional HCI person will know about a broad variety of evaluation methods and how to apply them effectively. The readings below provide an introduction to these methods, including case studies on how they are used in practice.

Recommended Classic Textbooks

Chapter excerpts from these texts are also listed later in this list

  1. A Practical Guide to Usability Testing. Revised Edition. Dumas, J.S. and Redish, J.C. (1999)
  2. Usability Engineering, Nielsen, J. (1993) Academic Press.
  3. Usability Inspection Methods (1994) J. Nielsen and R. Mack (eds), Wiley and Sons.

Overview of Methodologies

The following papers introduces (but do not detail) many methodologies. They also discuss and compare methodologies from various perspectives: to influence design, to evaluate products, to encourage end-user feedback, and to compare when and where each is useful. Because you do not yet know the details of these methodologies, the discussions in the papers may be hard to follow at times. Because these papers are just intended to give you a feel for the area, I suggest you give them a quick read now. As your readings progresses, you will find that you will be able to go back to them for a more careful read.

  1. How to design usable systems.
    Gould, J. Baecker, R., Grudin, J., Buxton, W., and Greenberg, S. Readings in Human Computer Interaction: Towards the Year 2000 (2nd Edition). (1995). Morgan-Kaufmann. pp. 93-121.
    • This delightful paper, originally written in 1988, presents an overview of many informal methods and how they are applied to the usability design process.
  2. Chapter 11: Evaluation Techniques.
    Dix, A., Finlay, J., Abowd, G., and Beale, R. Human Computer Interaction, 2nd Edition. Prentice Hall. pp. 363-400, (1998)
    • This useful overview chapter briefly surveys a variety of methodologies.
  3. Methodology matters: Doing research in the behavioural and social sciences.
    McGrath, J. in Baecker, R., Grudin, J., Buxton, W., and Greenberg, S. Readings in Human Computer Interaction: Towards the Year 2000 (2nd Edition). (1995). Morgan-Kaufmann. p152-169
    • See also a student-created summary by Stephanie Smale
    • This excellent paper discusses and compares fundamental concepts in evaluation methods. Initially, it may seem like a 'heavy' read. But its worth re-reading as you gain knowledge about various methods.

Supplemental Readings

  1. Applying the Behavioral, Cognitive, and Social Sciences to Products.
    Norman, D. 2001. mirrored from jnd.org.
    • This essay contrasts the academic vs business approach to product development.

Ethics

  1. Chapter 6: Usability testing.
    Nielsen, J. Usability Engineering, Academic Press, (1993) pp p165-205,
    • Section 6.4 Ethical aspects of tests with human subjects (p181-185) provides a gentle and pragmatic introduction to ethics in HCI.
  2. Ethics Forms, Research Services, University of Calgary
  3. The CORE Ethics Tutorial - Registration, which must be done to run any studies at the University of Calgary. Register an account for the tutorial, after which you will be able to enter the actual tutorial site.
  4. Ethics Web Site (CFREB), Research Services, University of Calgary
    • essential reading for those performing studies at the University of Calgary, as ethics approval is required for all studies.
    • contains documents explaining the ethics process, and templates for ethics applications and consent forms
    • see also Research Participants' Bill of Rights, by Tim Rogers in the Dept. of Psychology, which is a concise statement of what participant's should expect as fair treatment

Controlled Experiments Involving Users

  1. Chapter 11: Evaluation Techniques.
    Dix, A., Finlay, J., Abowd, G., and Beale, R. Human Computer Interaction, 2nd Edition. Prentice Hall. pp. 363-400, (1998)363-400
    • Section 11.3.1: Laboratory Studies (p365)
    • Section 11.5.1: Empirical methods (p375-385)
  2. User Interface Design,
    Eberts, R.E., Prentice Hall 1994
  3. Jacob Nielsen's Alertbox

Supplemental Readings

  1. Methodology matters: Doing research in the behavioural and social sciences.
    McGrath, J. in Baecker, R., Grudin, J., Buxton, W., and Greenberg, S. Readings in Human Computer Interaction: Towards the Year 2000 (2nd Edition). (1995). Morgan-Kaufmann. p152-169
    • see sections on Experimental Strategies (p157-8) and Study Design, Comparison Techniques, and Validity (p159 - 164)
  2. Comparison of menu displays for ordered lists.
    Greenberg, S. and Witten, I. In Proc Canadian Information Processing Society National Conference, Calgary, Alberta, May (1984)
    • This paper describes the first experiment I ever did, and is an example of how statistical methods can be interpreted and applied to comparing particular system features. I use this example in the lectures.

Bad Studies, Bad Statistics, Bas Science

  • Battling bad science by Ben Goldacre is a Ted talk describing how experiments and statistics can be misused]]
  • How to lie with statistics by Darrel Huff is a classic book from the '50s that illustrates common intentional and unintentional errors can lead to erroneous statistics interpretations.

Usability Testing

  1. Chapter 6: Usability testing.
    Nielsen, J. Usability Engineering, Academic Press, (1993) pp p165-205,
    • An excellent overview to various usability testing methodologies.
  2. User Observation: Guidelines for Apple Developers,
    Gomoll, Kathleen & Nicol, Anne. Apple Inc., January (1990)
    • An excellent short guide for performing usability studies.
  3. A Practical Guide to Usability Testing.
    Dumas, J.S. and Redish, J.C., Revised Edition. (1999)

Supplemental Readings

  1. Getting the whole team into usability testing.
    Ehrlich, K., Butler, M. and Pernce, K. IEEE Software, p89-90. (1994)
    • Tips and reasons for including developers as part of the evaluation team.
  2. Constructive interaction: A method for studying user-computer-user interaction.
    O'Malley, C., Draper, S. and Riley, M. From Proceedings of Interact '84, p1-5. (1984)
  3. Don't Make Me Think: A Common Sense Approach to Web Usability.
    Krug, S. 1st Edition. Pearson Education, Inc and New riders. (2005) These excerpts are from the 1st 2000 edition.

Inspection Methods - Overview

  1. Chapter 1: Executive summary.
    Nielsen, J. and R. Mack (eds), Usability Inspection Methods, p1-23, Wiley and Sons. (1994)
    • An executive summary and discussion of inspection methods.

Usability Heuristics and Heuristic Evaluation

  1. Improving a human-computer dialog.
    Molich, R. and Nielsen, J., Comm ACM, 33(3), 1990
    • This article lists the heuristics and presents a working example, solutions, and alternatives.
  2. Chapter 5: Usability heuristics.
    Nielsen, J. , Usability Engineering, p115-163, Academic Press. (1993)
    • Describes the heuristics in detail plus how and why it can be used to evaluate interfaces.
  3. Chapter 2: Heuristic evaluation.
    Nielsen, J. and R. Mack (eds), Usability Inspection Methods,p25-62, Wiley and Sons. (1994)
    • A more in depth discussion of how heuristic evaluation works and its reliability
  4. Exercise 8: Heuristic Evaluation of a Paper Mock-Up.
    Nielsen, J.,Usability Engineering, p273-4, Academic Press. (1993)
    • An example, with answers, of a heuristic evaluation of a a paper prototype.
  5. Enhancing the explanatory power of usability heuristics.
    Nielsen, J. Proceedings of the CHI'94 Conference on Human Factors in Computing Systems, p152-158. (1994)
    • takes usability guidelines developed by different sources and sees which ones contribute most the the explanation of actual usability problems drawn from a database.

Task-Centered Walkthroughs

  1. Working through task-centered system design,
    Greenberg, S. in Diaper, D. and Stanton, N. (Eds) The Handbook of Task Analysis for Human-Computer Interaction. Lawrence Erlbaum Associates. 2003.
    • A how-to approach to task-centered system design.
  2. Chapter 3: The pluralistic usability walkthrough: Coordinated Empathies.
    Nielsen, J. and R. Mack (eds) ]], Usability Inspection Methods, p25-62, Wiley and Sons. (1994)
    • Describe the steps in the pluralistic walkthough process.

Supplemental Readings

  1. Task-Centered User Interface Design: A Practical Introduction.
    Lewis, C. and Rieman, J. (1993). Available as Shareware.
    • This book introduced task-centered system design. Its all valuable.
    • Read the chapter Getting to Know Users and their Tasks., also published in R. Baecker, J. Grudin, W. Buxton and S. Greenberg (eds) Readings in Human Computer Interaction: Towards the Year 2000, p122-127, Morgan-Kaufmann.
  2. Chapter 5: The Cognitive Walkthrough Method: A Practitioner's Guide.
    Wharton, C., Rieman, J. Lewis, C. and Polson, In Nielsen, J. and R. Mack (eds) ]], Usability Inspection Methods, p25-62, Wiley and Sons. (1994)
    • This more complex but more complete cognitive walkthrough method inspired the discount task-centered walkthrough method.
  3. Cognitive Walkthroughs (Excerpt),
    Preece, J. et. al., (Human Computer Interaction, Addison-Wesley, page 679-684, (1994)
    • A checklist on how to do cognitive walkthroughs.

Personas

  1. The Inmates are Running the Asylum. Cooper, A. (1999) Sams (Macmillan). Excerpt: Chapter 9: Designing for Pleasure.
    • Introduces goal-centered system design and how one can use Personas to influence design. Chapter 11 also relates to this topic.
  2. About Face 3. Alan Cooper, Robert Reimann & David Cronin. Wiley Publishing, 2007.
  3. The Persona Lifecycle. John Pruitt and Tamara Adlin. Morgan Kaufmann (Elsevier), 2006
    • Everything you wanted to know about personas, and perhaps a bit more. The hardcover version of this book should be in my library

Prototyping

  1. Chapter 4.8 Prototyping.
    Nielsen, J.,Usability Engineering, p273-4, Academic Press. (1993)
  2. Prototyping for tiny fingers.
    Rettig, M., Communications of the ACM 37(4), ACM Press. 1994.
  3. Low vs. high-fidelity prototyping debate.
    Rudd, J., Stern, K. and Isensee, S. interactions 3(1), 76-85, ACM Press. January 1996.
  4. Pencils before pixels: a primer in hand-generated sketching.
    Baskinger, M., interactions 15(2), 28-36, ACM Press. March 2008.

Supplemental Readings

  1. Sketching User Experiences: The Workbook
    Saul Greenberg, Sheelagh Carpendale, Nicolai Marquardt and Bill Buxton, Morgan Kaufmann Press, 2012
    • My book and web site on sketching and prototyping.
  2. Sketching User Experiences: Getting the Design Right and the Right Design
    Bill Buxton, Morgan Kaufmann Press. 2007
    • Bill's excellent book motivating sketching and prototyping as part of the design process. A must read.

Pictive

  1. Pictive: An exploration in participatory design.
    Muller, M. (1991) In Proceedings of the ACM Conference on Human Factors in Computing Systems, p225-231, ACM Press.
    • A method for low fidelity prototype development that builds it on the fly with postit notes and other office supplies
  2. Retrospective on a year of participatory design using the PICTIVE technique.
    Muller, Michael J. Proceedings of the SIGCHI conference on Human factors in computing systems, ACM Press June 1992
    • Experiences with Pictive and recommendations.

Wizard of Oz

  1. Composing letters with a simulated listening typewriter.
    Gould, J., Conti, J., and Hovanyecz, T. Communications of the ACM, 26(4), 295-308. ACM Press. (1981)
    • A description of the Wizard of Oz prototyping/testing method.
  2. Prototyping an intelligent agent through Wizard of Oz
    Maulsby, D., Greenberg, S. and Mander, R. Proceedings of the ACM CHI'93 Conference on Human Factors in Computing Systems, Amsterdam, The Netherlands, p277-284, ACM Press. (1993).
    • Describes a sophisticated Wizard of Oz experiment, plus gives advice on how to apply WOZ
  3. Jeff Kelley's personal website describes the development of WOZ. He originally named this method. See also his original experiment that used it:

Introduction to Qualitative Field Studies

  1. Lecture on Qualitative Data Analysis (ppt)
    Carman Neustaedter. Also see the video of this lecture.
    • introduces qualitative vs. quantitative studies, and describes several analysis methods such as open coding and affinity diagramming.
  2. Jacob Nielsen's Alertbox

Open Coding of Qualitative Data

  1. Chapter 8. Open Coding.
    Strauss, A. and Corbin, J. Basics of Qualitative Research, 2nd Edition. Sage Publications. (1998).
    • How to do the open coding analysis method of qualitative data
  2. Saldana, J.(2012)
    The Coding Manual for Qualitative Researchers, Sage Publishing

Ethnography

  1. Where the Action Is. The Foundations of Embodied Interaction.
    Dourish, P. (2001) MIT Press, Cambridge MA.
    • Chapter 3: Social Computing p55-97.
    • This chapter introduces anthropology, ethnography and ethnomethodology and how it relates to Social Computing. The book itself is a must read for HCI researchers.
  2. Moving out of the control room: Ethnography in System Design.
    Hughes, J., King, V., Rodden, T. and Andersen, H. Proc ACM CSCW 1996.
  3. Ethnographically-informed systems design for air traffic control
    Bentley, R., Hughes, J.A., Randall, D., Rodden, T., Sawyer, P., Shapiro, D. and Summerville, I. (1992) in Proc CSCW'92, p123-129, November.
  4. Ethnography and Systems Development: Bounding the Intersection.
    Randall, D. (1996) Tutorial notes presented at CSCW'96.
    • Excerpts: Sections 3, 4,5,7

Contextual Inquiry

  1. Contextual Inquiry: A Participatory Approach.
    Holtzblatt, K. and Jones, S. (1996) In D. Schuler and A. Namioka (eds) Participatory Design: Principles and Practices, p177-210.
  2. Contextual Design: Principles and Practice.
    Holtzblatt, K. and Beyer, H. (1993) In D. Wixon and J. Ramey (eds) Field Methods Casebook for Software Design, p303-333.

Interviews, Questionnaires, Focus Groups and Logging

  1. Chapter 7: Usability Assessment Methods Beyond Testing
    Nielsen, J. Usability Engineering, Academic Press, (1993) p273-4.
    • Introduces observations, questionnaires and interviews, focus groups, logging, user feedback, and how to choose between them.
  2. Guidelines for Conducting a Focus Group Eliot & Associates. 2005. Retrieved Sept 10, 2013s.
  3. Designing useful and usable questionnaires: you can't just throw a questionnaire together. interactions 14, 3 (May 2007), 48-ff. DOI=10.1145/1242421.1242453 http://doi.acm.org/10.1145/1242421.1242453

Comparing Methods

  1. Usability Evaluation Considered Harmful (Some of the Time).
    Greenberg, S. and Buxton, B. (2008) In Proceedings of the ACM Conference on Human Factors in Computing Systems - ACM CHI'08. (Florence, Italy), ACM Press, pages 111-120, April 5-10.
  2. Mapping the method muddle: Guidance in using methods for user interface design.
    Olson, J. and Moran, T. (1996) In M. Rudisill, C. Lewis, P. Polson and T. McKay (eds) Human-Computer Interface Design: Success Stories, Emerging Methods, and Real-World Context, p269-300, Morgan-Kaufmann.
    • The authors associate a variety of methodological approaches to specific interface design activities.
  3. Human Computer Interaction
    Preece, J. et. al., (1994) Addison-Wesley
  4. User interface evaluation in the real world: A comparison of four techniques.
    Jeffries, R., Miller, J., Wharton, C. and Uyeda, K. (1991) In ACM Conference on Human Factors in Computing Systems. p119-124, ACM Press.
  5. Comparison of empirical testing and walkthrough methods in user interface evaluation.
    Karat, C., Campbell, R. and Fiegel, T. (1992) In ACM Conference on Human Factors in Computing Systems. 397-4044, ACM Press.
  6. Lets Get Real: A Position Paper on the Role of Cognitive Psychology in the Design of Humanly Useful and Usable Systems,
    Landauer, T. (1991) In J. Carroll (ed) Designing Interaction, Cambridge University Press.
  7. Selecting Empirical Methods for Software Engineering Research
    Easterbrook, S. and J. Singer and M.A. Storey and D. Damian, In Forrest Shull, Janice Singer and Dag I. K. Sjoberg (Eds) Guide to Advanced Empirical Software Engineering, Springer, 285-311, 2007.
  8. Why Most Published Research Findings Are False.
    Ioannidis, J. P. A. (2005) PLoS Medicine, August; 2(8): e124. link
  9. Filling much-needed holes.
    Norman, D.A. (2008) ACM Interactions, Jan/Feb 2008
  10. Technology First, Needs Last: The Research Product Gulf.
    Norman, D.A. (2010) ACM Interactions, Mar/Apr 2010
  11. When to Use Which User-Experience Research Methods.
    Christian Rohrer (2014) Nielsen Norman Group, October

Last updated February, 2013