« back to Publications
Ledo, D., Houben, S., Vermeulen, J., Marquardt, N., Oehlberg, L. and Greenberg, S. (2018)
Evaluation Strategies for HCI Toolkit Research. In Proceedings of the ACM Conference on Human Factors in Computing Systems – ACM CHI’18. (Montreal, Quebec, Canada), ACM Press, 17 pages, Paper #36, April 21-26.
Type: Refereed conference (full paper).
Earlier version: Report 2017-1096-03, September 2017.
Additional materials: recorded presentation on youtube (see link below).
DOI: 10.1145/3173574.3173610
Download as: Pdf
Recorded presentation: CHI 2018 talk on YouTube
Abstract.
Toolkit research plays an important role in the field of HCI, as it can heavily influence both the design and implementation of interactive systems. For publication, the HCI community typically expects toolkit research to include an evaluation component. The problem is that toolkit evaluation is challenging, as it is often unclear what `evaluating’ a toolkit means and what methods are appropriate. To address this problem, we analyzed 68 published toolkit papers. From our analysis, we provide an overview of, reflection on, and discussion of evaluation methods for toolkit contributions. We identify and discuss the value of four toolkit evaluation strategies, including the associated techniques that each employs. We offer a categorization of evaluation strategies for toolkit researchers, along with a discussion of the value, potential limitations, and trade-offs associated with each strategy.
Keywords: Toolkits; user interfaces; prototyping; design; evaluation.
