Saul Greenberg

back to the course CPSC 681

CPSC 681 - Project


Overview

You will do a project involving an evaluation, where the outcome of your evaluation informs interface design. Because of the constraints of the course, this is usually in the form of a pilot study.

The purpose of this project is to give you experience planning, running, debugging, analyzing and interpreting the results of an evaluative study. The actual study domain may be chosen from:

  • an evaluation of an aspect of your current research
  • a side project that you are interested in
  • a replication of an existing study, where you vary some aspect of it in an interesting way

Other projects may be proposed that differ from this (see example alternatives at the end of this page). If so, talk to me (and your supervisor!) about it as soon as possible, as my approval is required along with an agreement about deliverables.


Project Oversight

This is an independent project. While I will review your project as the course proceeds, you are otherwise responsible for all aspects of it.

  • Project work is done outside of class time.
  • You are responsible for carrying out all aspects of the project (e.g., defining the project, logistics, finding subjects if necessary, booking rooms and equipment, preparing software, getting data analysis tools, etc.).
  • While I will provide high-level guidance through some of your work if needed, I will not hold your hand.

Team Composition

Individual or two-person projects are allowed. Depending on the class size, you may be required to do the topic in groups of two rather than individually. If working as a team:

  • the quality and quantity of work done must reflect a two person vs a one person effort
  • while teams can be self-selecting, I have the final say
  • you must include an initial document that describes how the workload will be managed . I will ask you to update this document as the term progresses to show evidence that the workload is equal and collaborative.
  • With permission from the instructor, you can include people not in the course if it is part of a broader research project, e.g., faculty / senior student mentors .

Applying methodological techniques to evaluate situations in human computer interaction

The problem must be one relevant to human-computer interaction and of interest to you and your team. In practice, the best problem statements are ones that identify a small but interesting problem worth solving that can be realistically solved in the time frame of the course. Be careful of overly-ambitious problems.

Ideally, the problem will be related to one of:

  • your thesis work,
  • a replication of a published study,
  • systems being worked on at your job site (if you are an employee, but be careful of Intellectual Property),
  • an on-going research project (or related side-project) in your research lab or
  • something that I suggest

Problems selected must be amenable to one of the methodological approaches discussed in class, and must be doable given the time-frame of the course and the resources available to you. Deliverables and milestones are described below. I will give you a schedule stating approximately when they are due.

1. Initial project proposal.

Deliver a 1/2 to 1 page concise problem statement in the form of an informative title, author list, and abstract as defined by the ACM CHI paper format style at ACM CHI Archive Format. I will not read submissions that deviate from this format. See also How to Structure Reports on Experiments in Human Computer Interaction. If you are considering several problems to choose from, you may write up to a maximum of 3 problem statements. I will review these proposals and discuss them with you. Initial sign-off the project proposal by me is required. If there are issues in your proposal, you will be asked to generate new versions until sign off is given.

I strongly recommend you talking to your supervisor about possible ideas and what you propose to do, especially if you want your project to relate to your research or thesis.

The proposal must include:

  • an informative title
  • name(s) of the people invovled in the project
  • some background context
  • a problem statement
  • motivation as to why the problem is worth solving
  • the methodology you will bring to bear on this problem
  • why that methodology is appropriate
  • expected outcomes and contribution.
  • a short list of keywords

Example:

Can People Safely Answer Calls on Smart Phones Interfaces? A Comparison of a Gaze-required vs. a Gaze-free Interfaces
Joe Bloggs
Department of Computer Science, University of Calgary, Calgary, AB, Canada
bloggs@ucalgary.ca
Abstract: People now carry touch-based smart phones with them, where they are expected to respond to incoming calls no matter what their situation. Many smart phones now require people to look at the display and operate a graphical control to answer the call. The advantage is that such mechanisms may protect against people accidentally answering a call by inadvertant touches. Yet interfaces that divert a person's attention from the display can be dangerous, such as when walking or driving. To mitigate this problem, we developed a call-answering interface that can be operated without looking, and which still protects against accidental answers. Our hypothesis is that user distraction in a gaze-required standard interface vs. our experimental gaze-free interface is decreased in both stress-free and stressful situations, as measured by time looking at the display and by time to answer a call. Using a within-subjects Anova factorial design, we will test 20 subjects across both interfaces in a stress-free and a stressful setting. We will also interview them - using an open-ended questionnaire - after each condition to gather their subjective reactions to the interface. This methodology is appropriate for: capturing critical user performance measures in a controlled setting; capturing subjective preferences; and for performing a comparison of the two competing interfaces. We expect to contribute an understanding of the differences in distraction due to gaze-required vs. gaze-free call-answering interfaces.
Keywords: Smart phone interfaces, visual attention, distraction.

2. Developed project proposal

Deliver a 1 - 2 page concise problem statement using the points described above (but in more detail), in the form of an informative title, author list, abstract, and paper introduction as found in the ACM CHI Archive Format. See also How to Structure Reports on Experiments in Human Computer Interaction. Essentially, this is an expanded version of your initial project proposal.

This proposal will serve as a contract of what you will do. Sign off is again required by me.

3. Ethics application.

Research involving evaluations with human subjects require an ethics application that must be reviewed and approved by an ethics committee. Because this assignment is for pedagogical learning vs. actual research, I will not require you to submit this application to the ethics board. However, without ethics approval, you cannot use your pilot results for research purposes. That is, this is strictly a pedagogical learning exercise.

If you do want to use your study for research purposes, you may submit it to the ethics board. However, approval delays will likely conflict with course schedule demands, so you should be prepared to do at least part of your project as a learning exercise until approval is granted. In all cases, you will need to comply with the University Ethics Procedure.

Deliver an ethics application, using the forms and format as provided and required by the University.

  • All sections must be filled in, including attachments as required by the ethics process.
  • In addition, an appendix must be added to the application that illustrates an example of how you would analyze any data returned from the study (you may use made-up 'expected' data)
  • The section describing the study and methodology must provide specific details of:
    • the overall detailed study plan
    • specific details of the particular methodological technique you will use
    • details of how you would analyze any data returned from that study

4. Conduct an initial pilot study and writeup

Experiments, like computer programs, are full of bugs. Testing your experiment will help reveal those problems. Typical problems include flaws and inefficiencies in the methodology, and problems analyzing the data. You should first run yourselves, several class mates, and/or colleagues through your procedure. Analyse the data produced. This is a great time to catch the bugs and get rid of them. It is also a good time to get a sense of what the real results will be like, and perhaps to consider what changes should be made to the problem statement and methodological approach.

Run your study with one or two people (e.g., classmates). The goal here is to both debug your study, to see how amenable your data is to analysis, to see what kind of results you are getting, and ultimately to revise your study as needed. You will probably have to:

  • book rooms if needed
  • prepare any necessary paperwork, equipment and/or software (including consent forms)
  • solicit and schedule subjects
  • run subjects and collect data/observations
  • analyze the data and interpret the results

Deliver a 3-7 page paper in ACM CHI Archive Format (see also How to Structure Reports on Experiments in Human Computer Interaction). The paper combines your (revised) developed project proposal, the materials in your ethics application, plus other sections as follows:

  • title, authors, abstract, etc
  • introduction (motivation, background, and detailed problem statement),
  • detailed methodology used for pilot study
  • data and results observed
  • initial analysis of results
  • an analysis and interpretation of the pilot results, including expected trends
  • a critical appraisal of the limitations of the study and any problems in the analysis
  • a revision of your study plan as needed, based on the above

5. Continue your pilot using your revised plan

Iterate your study and analysis by conducting a more substantial pilot study based on your revised study plan. This is a milestone rather than a deliverable. Most students end up doing this a limited pilot i.e., a scaled down study of a few subjects (again typically your classmates, colleagues and friends unless ethics approval was granted).

6. Final Writeup

7. Final Presentation

  • Deliver a professional conference-style presentation.
  • Presentation time is strict, and includes a question period.
  • Powerpoint or other presentation materials must be handed in as instructed in class

Alternate projects

Depending on your circumstances, you may want to propose a project that differs from what was described above. Some examples are listed below.

Define and introduce a usability engineering process to your interface development team

This is typically appropriate for students who are working full time as a software developer in industry. You will:

  • define a realistic usability engineering / educational process that is amenable to your organization, budget, and work
  • define how the team would learn and/or use it
  • develop a design rationale and a design process for the usability engineering process
  • deploy the usability engineering process to evaluate a (small) project that arises from a real situation
  • critically appraise what worked and what didn't, and suggesting modifications
  • present your work in a professional manner:
    • write a technical report
    • give a conference-style presentation

Craft a software tool that supports a methodological process

Some methodologies can be simplified by writing software to support the process. Examples may include crowd-sourcing to analyzed data, the creation of particular data collection or data analysis tools, or the creation of a tool that supports some aspect of sifting through qualitative data. If you choose this type of project, you will:

  • define an interesting portion of a methodology or analysis that is amenable to software support
  • define the audience and when and where it would be useful
  • develop a design rationale and a design process to bring to bear on that methodology
  • iteratively design and implement the system
  • conduct a simple pilot study using your software tool to see how well it performs in practice
  • critically appraise the strengths limitations of the system
  • present your work in a professional manner:
    • write a technical report
    • give a conference-style presentation