CPSC 681 - Project
Overview
You will do a project involving an evaluation, where the outcome of your evaluation informs interface design. Because of the constraints of the course, this is usually in the form of a pilot study.
The purpose of this project is to give you experience planning, running, debugging, analyzing and interpreting the results of an evaluative study. The actual study domain may be chosen from:
- an evaluation of an aspect of your current research
- a side project that you are interested in
- a replication of an existing study, where you vary some aspect of it in an interesting way
Other projects may be proposed that differ from this (see example alternatives at the end of this page). If so, talk to me (and your supervisor!) about it as soon as possible, as my approval is required along with an agreement about deliverables.
Project Oversight
This is an independent project. While I will review your project as the course proceeds, you are otherwise responsible for all aspects of it.
- Project work is done outside of class time.
- You are responsible for carrying out all aspects of the project (e.g., defining the project, logistics, finding subjects if necessary, booking rooms and equipment, preparing software, getting data analysis tools, etc.).
- While I will provide high-level guidance through some of your work if needed, I will not hold your hand.
Team Composition
Individual or two-person projects are allowed. Depending on the class size, you may be required to do the topic in groups of two rather than individually. If working as a team:
- the quality and quantity of work done must reflect a two person vs a one person effort
- while teams can be self-selecting, I have the final say
- you must include an initial document that describes how the workload will be managed . I will ask you to update this document as the term progresses to show evidence that the workload is equal and collaborative.
- With permission from the instructor, you can include people not in the course if it is part of a broader research project, e.g., faculty / senior student mentors .
Applying methodological techniques to evaluate situations in human computer interaction
The problem must be one relevant to human-computer interaction and of interest to you and your team. In practice, the best problem statements are ones that identify a small but interesting problem worth solving that can be realistically solved in the time frame of the course. Be careful of overly-ambitious problems.
Ideally, the problem will be related to one of:
- your thesis work,
- a replication of a published study,
- systems being worked on at your job site (if you are an employee, but be careful of Intellectual Property),
- an on-going research project (or related side-project) in your research lab or
- something that I suggest
Problems selected must be amenable to one of the methodological approaches discussed in class, and must be doable given the time-frame of the course and the resources available to you. Deliverables and milestones are described below. I will give you a schedule stating approximately when they are due.
1. Initial project proposal.
Deliver a 1/2 to 1 page concise problem statement in the form of an informative title, author list, and abstract as defined by the ACM CHI paper format style at ACM CHI Archive Format. I will not read submissions that deviate from this format. See also How to Structure Reports on Experiments in Human Computer Interaction. If you are considering several problems to choose from, you may write up to a maximum of 3 problem statements. I will review these proposals and discuss them with you. Initial sign-off the project proposal by me is required. If there are issues in your proposal, you will be asked to generate new versions until sign off is given.
I strongly recommend you talking to your supervisor about possible ideas and what you propose to do, especially if you want your project to relate to your research or thesis.
The proposal must include:
- an informative title
- name(s) of the people invovled in the project
- some background context
- a problem statement
- motivation as to why the problem is worth solving
- the methodology you will bring to bear on this problem
- why that methodology is appropriate
- expected outcomes and contribution.
- a short list of keywords
Example:
2. Developed project proposal
Deliver a 1 - 2 page concise problem statement using the points described above (but in more detail), in the form of an informative title, author list, abstract, and paper introduction as found in the ACM CHI Archive Format. See also How to Structure Reports on Experiments in Human Computer Interaction. Essentially, this is an expanded version of your initial project proposal.
This proposal will serve as a contract of what you will do. Sign off is again required by me.
3. Ethics application.
Research involving evaluations with human subjects require an ethics application that must be reviewed and approved by an ethics committee. Because this assignment is for pedagogical learning vs. actual research, I will not require you to submit this application to the ethics board. However, without ethics approval, you cannot use your pilot results for research purposes. That is, this is strictly a pedagogical learning exercise.
If you do want to use your study for research purposes, you may submit it to the ethics board. However, approval delays will likely conflict with course schedule demands, so you should be prepared to do at least part of your project as a learning exercise until approval is granted. In all cases, you will need to comply with the University Ethics Procedure.
Deliver an ethics application, using the forms and format as provided and required by the University.
- All sections must be filled in, including attachments as required by the ethics process.
- In addition, an appendix must be added to the application that illustrates an example of how you would analyze any data returned from the study (you may use made-up 'expected' data)
- The section describing the study and methodology must provide specific details of:
- the overall detailed study plan
- specific details of the particular methodological technique you will use
- details of how you would analyze any data returned from that study
4. Conduct an initial pilot study and writeup
Experiments, like computer programs, are full of bugs. Testing your experiment will help reveal those problems. Typical problems include flaws and inefficiencies in the methodology, and problems analyzing the data. You should first run yourselves, several class mates, and/or colleagues through your procedure. Analyse the data produced. This is a great time to catch the bugs and get rid of them. It is also a good time to get a sense of what the real results will be like, and perhaps to consider what changes should be made to the problem statement and methodological approach.
Run your study with one or two people (e.g., classmates). The goal here is to both debug your study, to see how amenable your data is to analysis, to see what kind of results you are getting, and ultimately to revise your study as needed. You will probably have to:
- book rooms if needed
- prepare any necessary paperwork, equipment and/or software (including consent forms)
- solicit and schedule subjects
- run subjects and collect data/observations
- analyze the data and interpret the results
Deliver a 3-7 page paper in ACM CHI Archive Format (see also How to Structure Reports on Experiments in Human Computer Interaction). The paper combines your (revised) developed project proposal, the materials in your ethics application, plus other sections as follows:
- title, authors, abstract, etc
- introduction (motivation, background, and detailed problem statement),
- detailed methodology used for pilot study
- data and results observed
- initial analysis of results
- an analysis and interpretation of the pilot results, including expected trends
- a critical appraisal of the limitations of the study and any problems in the analysis
- a revision of your study plan as needed, based on the above
5. Continue your pilot using your revised plan
Iterate your study and analysis by conducting a more substantial pilot study based on your revised study plan. This is a milestone rather than a deliverable. Most students end up doing this a limited pilot i.e., a scaled down study of a few subjects (again typically your classmates, colleagues and friends unless ethics approval was granted).
6. Final Writeup
- Deliver an 8-10 page paper in ACM CHI Archive Format and style, detailing all aspects of your project, its results, discussion, and future work (See How to Structure Reports on Experiments in Human Computer Interaction).
- Deliver a separate appendix (both electronic and paper) structured as an archive. This must be a well-structured set of materials that details all your research (this will include everything mentioned above). See How to Structure Reports on Experiments in Human Computer Interaction for a description.
7. Final Presentation
- Deliver a professional conference-style presentation.
- Presentation time is strict, and includes a question period.
- Powerpoint or other presentation materials must be handed in as instructed in class
Alternate projects
Depending on your circumstances, you may want to propose a project that differs from what was described above. Some examples are listed below.
Define and introduce a usability engineering process to your interface development team
This is typically appropriate for students who are working full time as a software developer in industry. You will:
- define a realistic usability engineering / educational process that is amenable to your organization, budget, and work
- define how the team would learn and/or use it
- develop a design rationale and a design process for the usability engineering process
- deploy the usability engineering process to evaluate a (small) project that arises from a real situation
- critically appraise what worked and what didn't, and suggesting modifications
- present your work in a professional manner:
- write a technical report
- give a conference-style presentation
Craft a software tool that supports a methodological process
Some methodologies can be simplified by writing software to support the process. Examples may include crowd-sourcing to analyzed data, the creation of particular data collection or data analysis tools, or the creation of a tool that supports some aspect of sifting through qualitative data. If you choose this type of project, you will:
- define an interesting portion of a methodology or analysis that is amenable to software support
- define the audience and when and where it would be useful
- develop a design rationale and a design process to bring to bear on that methodology
- iteratively design and implement the system
- conduct a simple pilot study using your software tool to see how well it performs in practice
- critically appraise the strengths limitations of the system
- present your work in a professional manner:
- write a technical report
- give a conference-style presentation