back to HCI Lecture Modules

Controlled Study: Mouse Typing on Various Keyboards

Learning Overview

Learning Goals. This assignment is a hands-on exercise on running a controlled study. Its immediate purpose is to give you experience conducting a controlled study, in generating data, in analyzing that data via statistical methods, in interpretting the results, and critiquing the experiment.

Learning Scenario. You will assume your team is working for a company that is considering various on-screen keyboard designs, where a person types with their mouse (given today's technology, this should really be done with a touch interface, but for ease of this exercise we will use a mouse). You will use each other as subjects, where you will each run at least one person. You will contribute the data to a pool of data provided by all class members. You will independently analyze the collective results. Details are below, with software and additional material provided in the appendices listed below.

What would normally be delivered. If this were done 'for real', you would be expected to deliver a technical report that includes the background and methodology (most which is already provided), your particular observations and results, the collective observations and results, a description of your statistical analysis and findings, an interpretation of these results, some design recommendations of what keyboard should be used and the perceived benefits/drawbacks, and a critique of the study method. However, because we are doing this as a class exercise, we will discuss all these.

Related Material

• Controlled Experiments: Keyboard Study introductory powerpoint lecture to the study
• Appendix A: Background to existing keyboard layouts
• Appendix B: Instructions to Subjects
• Appendix C: Consent form
• Appendix D: Questionnaires
• Appendix E: Text for copy typing
• Appendix F: Subject Work Sheet
• Appendix G: Running the Software and Understanding the Data File
• Appendix H: Randomized Subject Order
• Executable (and source) for running the study as a zip file

Introduction

The Problem

Your company produces a kiosk-based system, where all input is expected to be done with a mouse. Typically, clients will buy this system for walk-up and use situations. Clients will install their own applications on this system. Customers will typically walk up and use the application in the kiosk. For most client applications, customers will use the mouse to do some simple selections (e.g., menu selections). However, various applications also ask clients to enter some text, ranging from a few characters to several lines of text. Consequently, your company has decided to create an on-screen keyboard that allows people to mouse-type.

Some constraints are already known:

• typing of numbers and text will be done by displaying a simulated keyboard on the screen, and by selecting keys with the mouse (i.e., mouse-typing);
• no hardware keyboard or keypad is present;
• end-customers may vary considerably in standard typing abilities;
• people need to be able to enter their text as quickly as possible (as otherwise line-ups may occur)
• errors in typed text should be minimized (as entered information often goes into a database and can be critical)
• While hardware display size can be increased to accommodate a larger on-screen keyboard, it is considerably more expensive to produce

Several on-screen keyboard layout designs are currently being considered.

• Keyboard type:
  • Qwerty keyboard (as typists know it)
  • Alphabetic keyboard (as may be easier for non-typists)
  • Random keyboard (to get a sense of 'worse case' conditions)
• Keyboard Size
  • Small
  • Large

Essentially, your company disagrees as to which is best. The interaction design group believes that the qwerty keyboard is best overall: they are familiar to experienced typists, and inexperienced typists will likely be equally slow regardless of the layout. The marketing department believes that the alphabetic keyboard is better for inexperienced typists, and that experienced typists can do fine on them because mouse-typing is different from touch typing. The company president is neutral, but is interested in knowing how either keyboard compares to a worse-case situation, i.e., where keys are randomly placed on a keyboard (that is, does layout even matter in mouse typing?), and if touch-typing speed is even relevant. The hardware group believes that people will be much faster on large keyboards than small keyboards, and is arguing that the small screen product should thus be abandoned. The sales group says that size doesn't matter, and that it would be a mistake to stop small screen production as their clients slightly prefer small screens as they are cheaper.

You are in charge of finding out which one is 'best', and in making a recommendation. You have decided to run a controlled experiment comparing typing speed and error rate on these keyboards, and will use the results to make recommendations about which layout should be used in the product.

The Keyboards

The various keyboards are displayed below. All keyboards can be displayed in a large and small size (not shown).

Qwerty (Small)

Alphabetic (Small)

Random (Small) - the actual keys shown vary

Description of the Experiment

Hypothesis.

You will be collecting data and comparing through statistics the mouse-typing speeds on the various layouts and sizes. The null hypothesis are listed below. Note that two forms are provided for Hypothesis 1, where the second form includes people's typing abilities.

Null Hypotheses 1a (simpler): There is no difference between people's mouse-typing ability as measured in speed (characters per second) and error rate (ratio of extra characters typed over the minimum required) when using a qwerty, alphabetic, or random layouts on two different sizes of a simulated keyboard.

Null Hypotheses 1b (complete: There is no difference between people's mouse-typing ability as measured in speed (characters per second) and error rate (ratio of extra characters typed over the minimum required) when using a qwerty, alphabetic, or random layouts on two different sizes of a simulated keyboard, regardless of people's touch typing abilities as measured in characters per second.

Null Hypotheses 2: People's typing speeds when touch-typing is not correlated with mouse-typing speeds, where both are measured in characters per second.

Subjects.

Normally, subjects with a broad range of typing skills would be selected from the public at large to fit the demographic profile. In this case, your subjects are chosen from members of your class, all University of Calgary students.

Materials.

The various appendices listed above describe how to download and use the custom software that presents a soft keyboard. Appendices also include instructions for downloading software, instructions to subjects, text for copy typing, consent forms, questionnaires, and individual work sheets.

Method.

Each subject goes through the following steps, detailed further in Appendix 1.

1. Prolog:
   • The subject is asked to sign a consent form (Appendix 3).
   • The experiment is described to them (Appendix 1).
   • They answer a pre-test questionnaire (Appendix 4).
   • They are given additional instructions as they perform the task (Appendix 1).
2. Trial 1-2: Touch-typing speed with physical keyboard
   • Subjects are given two English paragraphs to touch type using the physical keyboard (Appendix 2).
   • When subjects are ready, the experimenter presses the Start button, and the subject types the paragraph.
   • Immediately after the subject finishes the task, the experimenter presses the Stop button
   • The process repeats itself with a new paragraph.
3. Trials 3-14: Typing speed with the six keyboards
   • The experimenter selects one of the keyboards and sizes (order to be determined by your Instructor) and describes it to the subject.
   • The subject is given an English sentence to mouse type (Appendix 2). After reading it, the subject presses the Start button and enters the sentence. When complete, the subject presses the Stop button. Note: make sure they press this button, as timing stops when it is pressed.
   • The process is repeated on the same keyboard layout with two more sentences.
   • After the keyboard is tested, ask subjects what they thought about the layout.
   • The process is repeated with the other keyboard layout.
4. Post-session
   • Ask subjects what they thought about all keyboards, and have them state which one they preferred (Appendix
   • Ask them for any other comments.
   • Debrief them. Explain your motivation if they wish (for example, expectations about keyboard layouts)

Results: the Data

Each subject will be identified by a unique number. You should collect your numeric and qualitative results on the worksheets provided in the Appendix. Your numeric data should be recorded in EXACTLY the format described on the worksheet. The data collected from all class mates will then be returned to you.

For hypothesis 1, calculate an ANOVA contrasting the typing speed of layouts and sizes. Do the same with the error rate.

For hypothesis 2, do a correlation / regression analysis, showing the line of best fit that contrasts subjects' typing speed on the conventional typing test (the X-axis) with their typing speed on each of the other "soft" keyboards (the Y-axis).

Your write up should include a table of summary statistics and descriptions of these result.

Analysis / Conclusion

An experimental report typically follows the style specified in the accompanying document "How to structure reports on experiments in human-computer interaction". This assignment specification already gives material for the introduction, description of the experiment, and expectations in the results section. It would also include a related work section, and it would be up to you to investigate other keyboard studies. It is up to you to interpret what the results mean. You should discuss this with other experimenters, and think about it. Think about the impact your findings would have to the application constraints described in this scenario, the general use of soft keyboards, the implications to other practitioners, your critical reflections, and a research agenda of future study and issues. Use both the qualitative and quantitative findings of your study, as well as prior research on keyboards.

Your conclusion should make a recommendation and should justify your results. Include your rough data and worksheets in an appendix. This should include the work sheets where you recorded pre- and post-experiment answers to questions, subject's comments, etc. You would not normally include this in a paper, but you should always maintain this as an archive that you can use if you have to refer back to your data and its analysis.