Chapter 14: Introducing Evaluation in Interaction Design

Evaluation aims to assess both the system’s usability—how easily it is learned and operated—and the overall user experience (UX), encompassing factors such as satisfaction, motivation, and enjoyment. Critical considerations for effective evaluation include understanding the why (designing for target populations, fixing problems before market release), the what (ranging from aesthetic features and workflow to web accessibility and safety), the where (highly controlled usability labs, uncontrolled natural environments or "in-the-wild" settings, or hybrid, technology-embedded "living labs"), and the when (formative evaluation throughout iterative design, versus summative evaluation upon completion). Evaluation methods are categorized into three broad areas: Controlled settings involving users utilize methods like usability testing (combining observation, interviews, and experiments in a lab) and formalized experiments designed to test specific hypotheses under minimal distraction. Natural settings involving users rely on field studies, including ethnographic and in-the-wild studies, which observe natural usage patterns, often sacrificing control for contextual realism and are used to establish requirements or inform technology deployment. Evaluation methods not directly involving users include inspection techniques such as heuristic evaluation (applying rules of thumb) and cognitive walk-throughs (simulating a user’s step-by-step problem-solving process to assess ease of learning). Other non-user methods involve modeling user behavior, such as using Fitts’ law to predict target reach time, and analytics, such as web or learning analytics, to track and optimize usage of existing systems. Evaluation approaches are often mixed to achieve a richer understanding, and rapid, informal opportunistic evaluations are useful for gathering quick feedback on early design concepts. The chapter illustrates evaluation through case studies, including a controlled experiment using physiological responses and questionnaires to measure challenge and engagement in a game, and a field study utilizing the Ethnobot chatbot to collect ethnographic data and experiences from participants moving outdoors. Finally, key practical and ethical considerations are discussed, emphasizing the mandatory requirement of informed consent forms to protect participants’ rights and privacy (enforced by bodies like Institutional Review Boards or IRBs) and the need to address data quality factors such as reliability, validity, ecological validity (the realism of the setting), scope (generalizability), and potential biases like the Hawthorne effect. Evaluation can also be scaled up using crowdsourcing platforms like Mechanical Turk to recruit thousands of participants quickly for human intelligence tasks (HITs), a practice related to citizen science.