Controlled Experiments
I introduce quantitative methods for evaluating interfaces with users, concentrating on experimental design, controlled experimentation, simple statistics covering t-tests, correlation, regression, and single/multi-factor anova design
Powerpoint lectures
- Part 1. Controlled Experiments: Introduction introduces controlled experiments
- Part 2. Controlled Experiments: Basic Statistical Tests basic experimental design up to t-tests and correlation
- Part 3. Controlled Experiments: Anova introduces analysis of variance
- Part 4. Controlled Experiments: Keyboard Study is an introduction to the mouse-typing on a keyboard study provided below.
Topics Covered
- Quantitative evaluation methods
- User performance data collection
- Controlled experiments
- The experimental method
- Planning an experiment
- The value of statistics
- T-test
- Significance levels, types of error
- Correlation and regression
- Analysis of variance - single and multi-factor
Suggested Readings
- Eberts, R.E.
User Interface Design, Prentice Hall 1994 - Readings in Human Computer Interaction: Towards the Year 2000 (2nd Edition). by Baecker, R., Grudin, J., Buxton, W., and Greenberg, S., Morgan-Kaufmann, San Francisco, CA, 1995
- Chapter 2: Design and Evaluation - the section on Controlled experiments.
- Methodology matters: Doing research in the behavioural and social sciences. by McGrath, J. p152-169 - covers many fundamental issues in empirical research methodologies.
Hands-on Exercise - Evaluation of Mouse Typing Keyboards
This exercise involves a controlled study of typing with a mouse (mouse-typing) on-screen keyboards with different layouts. I run a subject in-class through a controlled experiment to give people an idea of what a subject session is like. Students then run each other as subjects (either in class or out of class - it takes about 15 minutes). They hand in the data, which is collected as group data set. Students then analyze and interpret the data via various statistical methods (alternately, the analysis is done in class).
Additional Readings
- Battling bad science by Ben Goldacre is a Ted talk describing how experiments and statistics can be misused]]
- How to lie with statistics by Darrel Huff is a classic book from the '50s that illustrates common intentional and unintentional errors can lead to erroneous statistics interpretations.
Additional Videos
There are lots of youtube video showing you how to do basic stats. For example:
- Single Factor Anova and post hoc Tukey and Sheffe test showing basic math
- Another Single Factor Anova in Excel, using a post hoc T-Test.
Statistical Packages
There are many statistical packages available, some for free, some which cost
- Excel's Data Analysis add-in' includes very easy to use basic statistical functions, e.g., t-tests, single factor anova, correlations, and regression. However, it does not include a multi-factor Anova function. A google search will tell you how to include this add-in and how to use the particular function of interest.
- is one of the most popular and powerful stats package out there, and is freely available for U of C students. The catch is that it will take you some time to learn how to use it. Still, the time invested in it is worth it if you plan to use stats regularly.
- Gary Perlman's |STAT is a free very simple to use command line statistics package. I particularly like its anova function, where it figures out your experimental design (i.e., within vs between subjects) based on your data format - see the anova manual page on how to do it. However, the only STAT version easily downloadable is one that runs on DOS, which doesn't work as is on 64 bit machines. Thus to make it work, you have to download and run the DOSBox X86 emulator, where you will run |STAT within it.
- To run a STAT function, first make a folder (e.g., C:/mystuff) that contains the STAT package and your data files.
- Start DOSBOX. Issue the command mount Y: C:/mystuff . This will mount that folder as Y:
- Then type Y: to change into Y:. At this point, you can now run any STAT function.
- R is an increasingly popular and powerful free stat package that can be used as a programming language. I haven't used it myself, and I heard that Anovas are hard to do in it. But look at this page for some information, where basic stats are summarized alongside R.
Major sources used to prepare lecture material
- The readings above provide general background.
- Other sources I used include general psychology statistics books, and chapters in HCI textbooks dealing with controlled experiments (e.g., see Human Computer Interaction textbook by Dix, Finlay, Abowd and Beale).
- The research planning chart included in the slides came from an old CHI tutorial, but I've since lost track of who actually created it.