text:
Improving the Learnability of Mobile Devices for Older Adults by Rock Anthony Leung B.A.Sc., The University of Waterloo, 1999 M.C.S., The University of New Brunswick, 2006 A THESIS SUBMITTED IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF DOCTOR OF PHILOSOPHY in THE FACULTY OF GRADUATE STUDIES (Computer Science) THE UNIVERSITY OF BRITISH COLUMBIA (Vancouver) July 2011 © Rock Anthony Leung, 2011 ii Abstract Mobile computing devices, such as smart phones, offer benefits that may be especially valuable to older adults (age 65+). However older adults have been shown to have difficulty learning to use these devices, which is a barrier for technology adoption. The main goal of the research reported in this dissertation was to investigate three promising design approaches – increasing the interpretability of graphical icons, incorporating a multi- layered interface, and augmenting the mobile device's display – to determine whether each can improve the learnability of mobile devices for older adults. We involved both older and younger adults in our studies to uncover benefits unique to older adults. In our investigation of graphical icons, we conducted an experiment to determine which icon characteristics affect initial icon interpretability for older adults. We found that icon interpretability can be improved for older adults by reducing the semantic distance between the objects depicted in the icon and the icon's function, and by labelling icons. In our investigation of multi-layered interfaces, we prototyped a two-layer smart phone address book and conducted an experiment to assess its learnability over four learning phases. We found that the multi-layered interface, compared to a non-layered full-functionality control interface, provided greater benefits for older participants than for younger participants in terms of faster task completion times during initial learning, lower perceived interface complexity, and greater interface preference for learning. In our investigation of augmenting a mobile device display with a larger display to help older adults learn new devices, we conducted a comprehensive survey of older adults' learning needs and preferences. Based on the survey findings, we designed and prototyped Help Kiosk, an augmented display system for helping older adults to learn to use a smart phone. An informal evaluation found preliminary evidence that Help Kiosk may be able to assist older adults in performing new mobile phone tasks. Through these three investigations, our research identified and validated design approaches that researchers and developers can use to improve the learnability of mobile devices for older adults, which should increase the chances of technology adoption. iii Preface All work reported in this dissertation was conducted under the co-supervision of Dr. Joanna McGrenere (Department of Computer Science) and Dr. Peter Graf (Department of Psychology). All research with human participants was reviewed and approved by the UBC Research Ethics Board. The numbers and project titles for the associated Certificates of Approval are: • B03-0147: Cognitive demands of an assistive communication device (Amendment: H03- 80147) • H10-00433: Investigating the methods and resources older adults employ to learn to use mobile devices • B05-1017: Evaluation of handheld technology usability (Amendment H05-81017) I was the primary contributor to all aspects of this research and conducted the studies in every case not specified below. For the multi-layered interface experiment (Chapter 4), Leah Findlater, a UBC senior PhD student at the time of the research, contributed feedback on the study design and data analysis. Under my supervision, a graduate student and three undergraduate research assistants helped with the execution of this research. For the experiment on graphical icons (Chapter 3), Xiaojuan Ma, a visiting graduate student, conducted all of the background information interviews and administered two standardized tests. In addition, Jessica Dawson, an undergraduate research assistant independently scored participant data in this experiment using video recordings of study sessions. For the experiment on multi-layered interfaces (Chapter 4), Justine Yang, an undergraduate research assistant, implemented much of the prototype and all of the interactive tutorial used in the experiment, conducted the experiment with half of the participants (I conducted the experiment with the other half), and assisted with data analysis. For the survey study (Chapter 5), Vilia Ingriany, an undergraduate research assistant, helped to recruit participants and assisted in the qualitative analysis. For the Help Kiosk research (Chapter 6), Vilia implemented the Help Kiosk prototype and helped me to conduct the user study. iv Large parts of Chapters 3 and 4 are updated versions of published papers that were written primarily by me with feedback from various co-authors: • An earlier version of Chapter 3 was published as: Leung, R., McGrenere, J., & Graf, P. (2009). Age-related differences in the initial usability of mobile device icons. Behaviour & Information Technology, 14 pages. © Taylor & Francis, 2009. doi:10.1080/01449290903171308 • An earlier version of Chapter 4 was published as: Leung, R., Findlater, L., McGrenere, J., Graf, P., & Yang, J. (2010). Multi-layered interfaces to improve older adults' initial learnability of mobile applications. ACM Transactions on Accessible Computing. (TACCESS) 3(1), Article 1, 30 pages. © ACM, 2010. doi:10.1145/1838562.1838563 v Table of Contents Abstract ........................................................................................................................................... ii Preface ............................................................................................................................................ iii Table of Contents ............................................................................................................................ v List of Tables................................................................................................................................. vii List of Figures .............................................................................................................................. viii Glossary ............................................................................................................................................ x Acknowledgements ........................................................................................................................ xi Dedication ..................................................................................................................................... xii 1. Introduction ................................................................................................................................. 1 1.1 Research goal ..................................................................................................................... 2 1.2 Research strategy ............................................................................................................... 3 1.3 Choosing design approaches to investigate ........................................................................ 5 1.4 Summary of research contributions ................................................................................. 11 1.5 Dissertation outline .......................................................................................................... 12 2. Background and Related Work ............................................................................................... 13 2.1 Helping older adults learn to use new technology ........................................................... 13 2.2 Graphical icons ................................................................................................................ 15 2.3 Multi-layered interfaces ................................................................................................... 17 2.4 Augmented displays ......................................................................................................... 20 2.5 Summary .......................................................................................................................... 24 3. Increasing the Interpretability of Graphical Icons: An Experiment .................................... 25 3.1 Objectives and methodology ............................................................................................ 25 3.2 Preliminary exploratory study .......................................................................................... 26 3.3 Methods ........................................................................................................................... 28 3.4 Results .............................................................................................................................. 37 3.5 Discussion ........................................................................................................................ 40 3.6 Summary .......................................................................................................................... 45 vi 4. Incorporating a Multi-Layered Interface: An Experiment ................................................... 46 4.1 Objectives and methodology ............................................................................................ 46 4.2 Methods ........................................................................................................................... 47 4.3 Results .............................................................................................................................. 57 4.4 Discussion ........................................................................................................................ 68 4.5 Summary .......................................................................................................................... 73 5. Augmenting the Mobile Device's Display: A Survey .............................................................. 74 5.1 Objectives and methodology ............................................................................................ 74 5.2 Methods ........................................................................................................................... 75 5.3 Results .............................................................................................................................. 81 5.4 Discussion ........................................................................................................................ 93 5.5 Summary .......................................................................................................................... 96 6. Augmenting the Mobile Device's Display: Design and an Informal User Study ................. 97 6.1 Objectives and methodology ............................................................................................ 97 6.2 Help Kiosk design ............................................................................................................ 98 6.3 User study methods ........................................................................................................ 103 6.4 User study results ........................................................................................................... 106 6.5 Discussion ...................................................................................................................... 108 6.6 Summary ........................................................................................................................ 109 7. Conclusions .............................................................................................................................. 111 7.1 Primary research contributions ...................................................................................... 111 7.2 Secondary contributions ................................................................................................. 114 7.3 Reflections on overall research strategy and findings .................................................... 114 7.4 Future work .................................................................................................................... 116 7.5 Concluding comments .................................................................................................... 118 References .................................................................................................................................... 119 Appendices ................................................................................................................................... 126 Appendix A: Graphical icon experiment materials and a portion of the raw results ............. 126 Appendix B: Multi-layered interface experiment materials .................................................. 144 Appendix C: Survey study materials ..................................................................................... 161 Appendix D: Help Kiosk informal user study materials ....................................................... 182 vii List of Tables Table 1.1: Types of scaffolding that can support learning ............................................................... 6 Table 3.1: Definitions of concreteness, semantic distance and label .............................................. 27 Table 3.2: Participant characteristics .............................................................................................. 29 Table 3.3: Example icons given to our three icon characterizers ................................................... 30 Table 3.4: Familiarity score mappings for each of the three label conditions ................................ 33 Table 3.5: Correlation between familiarity and icon interpretability ............................................. 36 Table 4.1: Participant characteristics .............................................................................................. 49 Table 4.2: Options menu functions for the Reduced- and Full-Functionality layers ...................... 52 Table 4.3: Minimum number of steps for basic and advanced task sets ......................................... 62 Table 5.1: Definitions of different levels of expertise with technology ......................................... 76 Table 5.2: Participant characteristics .............................................................................................. 77 Table 5.3: Learning resource qualities and features ....................................................................... 78 Table 5.4: Learning methods .......................................................................................................... 78 Table 5.5: Pair-wise correlations between responses to Q12, Q13, Q14 and Q15 ......................... 87 Table 5.6: Pair-wise correlations between responses to Q12 and Q14, by age group .................... 88 Table 6.1: Participant characteristics ............................................................................................ 103 Table 6.2: Task completion using Help Kiosk and manual .......................................................... 107 viii List of Figures Figure 1.1: Example calendar application icons ............................................................................... 7 Figure 1.2: Example of a multi-layered interface ............................................................................. 8 Figure 1.3: Possible way to augment a mobile device interface with a larger display ................... 10 Figure 3.1: Example of icons that older adults had difficulty interpreting ..................................... 26 Figure 3.2: Example of icons differing in concreteness ................................................................. 27 Figure 3.3: Example of icons differing in semantic distance ......................................................... 27 Figure 3.4: Ratings for, and groupings of, the 60 icons used in experiment .................................. 31 Figure 3.5: Example icons representing the four concreteness+semantic-distance combinations . 31 Figure 3.6: Example of application icons and menu list icons ....................................................... 32 Figure 3.7: Example pages for the three label conditions ............................................................... 32 Figure 3.8: Icon object identification accuracy scores ................................................................... 37 Figure 3.9: Icon interpretation accuracy scores by age group and concreteness ............................ 38 Figure 3.10: Icon interpretation accuracy scores by age group and semantic distance .................. 39 Figure 4.1: Example screenshots from Reduced- and Full-Functionality layers ............................ 51 Figure 4.2: Older adult holding the mobile device used in the study ............................................. 54 Figure 4.3: Extra steps and completion times for attempts in all four ML learning phases ........... 59 Figure 4.4: Number of attempts until mastery in the two task acquisition phases ......................... 60 Figure 4.5: Extra steps and completion times for first three attempts, separated by step type ....... 63 Figure 4.6: Participants' interface preferences for learning and for long-term use ........................ 65 Figure 5.1: Screen capture of the online questionnaire .................................................................. 80 Figure 5.2: Frequency of wanting to learn new tasks and forgetting learned tasks ........................ 82 Figure 5.3: Importance of learning task steps vs. gaining understanding ....................................... 83 Figure 5.4: Importance of various learning resource features ........................................................ 84 Figure 5.5: Learning method preferences ....................................................................................... 85 Figure 5.6: Importance of learning task steps vs. gaining understanding, by expertise ................. 92 ix Figure 6.1: Help Kiosk Find Info and Learn/Do panes ................................................................ 100 Figure 6.2: Text instructions used in Help Kiosk and manual ...................................................... 105 Figure 6.3: Participants' perceived workload while using Help Kiosk and manual ..................... 107 x Glossary Abstract icon. An icon that represents information using graphical features such as shapes and arrows (McDougall, Curry, & de Bruijn, 1999). In the research reported in this dissertation, icons with symbols (e.g., , ), even those that had precise well-established meanings, were considered to be abstract icons. Concreteness. How closely an icon depicts real world objects, places or people (McDougall, de Bruijn, & Curry, 2000). Learnability. The degree to which a system's interface enables novices to attain mastery in performing basic and advanced system functions. Semantic distance. The closeness of the relationship between the objects or concepts depicted in the icon's graphic and the function being represented (McDougall, Curry, & de Bruijn, 1999). In this dissertation, semantic distance refers only to the icon's graphic and not to the icon's text label, if present. Smart phone. A mobile phone that offers advanced computing ability and connectivity, and allows the user to install and run software applications. xi Acknowledgements I am thankful for the many people who contributed towards the research reported in this dissertation and helped me grow as a researcher. I would first like to express my heartfelt gratitude to my supervisors Dr. Joanna McGrenere and Dr. Peter Graf for their guidance and continuous support. I am thankful to Dr. McGrenere for her many insights on user study design, as well as her advice on how to clearly communicate research findings and the overall "story". I also always appreciated Dr. McGrenere's unbelievably quick turnaround in giving feedback on manuscript drafts. I am thankful to Dr. Graf for his invaluable guidance on aging and technology research, and sharing his statistical expertise and his positive attitudes on research. Interdisciplinary research collaborations can be challenging, but Dr. McGrenere and Dr. Graf made the experience both productive and enjoyable. I thank my committee members, Dr. Kellogg Booth and Dr. Jim Little, who provided much helpful input, particularly on this dissertation. I also thank my examining committee members, Dr. Ron Rensink, Dr. Stephen Petrina, and Dr. Vicki Hanson, for their thorough and insightful feedback that improved the final dissertation. I am thankful to many others from the HCI community. Tony Tang, Karyn Moffatt, Leah Findlater, Michael McGuffin, and Mike Wu, who I consider my "big brothers and sisters" in the community, always offered helpful answers to my many questions and were good role models. Xioajuan Ma, Jessica Dawson, Justine Yang, and Vilia Ingriany were very helpful in prototyping user interface designs and conducting user studies. Garth Shoemaker, Ricardo Pedrosa, Steve Yohanan, Dr. Charlotte Tang, Matthew Brehmer, Mike Blackstock, Dr. Rodger Lea, Joseph Luk, and many others (from the MUX, Imager and MAGIC labs and the HCI Grad Research Forum) provided useful feedback on this research and on completing the Ph.D. program. This research was supported financially by the Natural Sciences and Engineering Research Council of Canada (NSERC), the Canadian Institutes of Health Research (CIHR), and Nokia. Nokia and Google provided several smart phones used in the studies. Furthermore, I thank my parents, who were the best pilot participants I could ask for and who were always available to help my young family out during busy times. Finally, I thank my wife, Adrienne, for travelling this graduate school journey beside me with love, patience, and a good sense of humour. xii Dedication Soli Deo Gloria xiii 1) everything that's already in the world when you're born is just normal; 2) anything that gets invented between then and before you turn thirty is incredibly exciting and creative and with any luck you can make a career out of it; 3) anything that gets invented after you're thirty is against the natural order of things and the beginning of the end of civilisation as we know it until it's been around for about ten years when it gradually turns out to be alright really. - Douglas Adams, 1999 1 Chapter 1 1. Introduction Mobile computing devices, such as smart phones, digital cameras, and digital media players, are increasingly pervasive, computationally powerful and feature-rich. They offer benefits that may be especially valuable to older adults (age 65+), a growing segment of the population in most countries. For example, smart phones can provide tools to older adults that users of all ages benefit from, such as software applications for connecting with loved ones, accessing contact information, browsing Internet content, and playing games. Further, mobile devices can help older adults remain more independent and maintain their quality of life as they experience declines in perceptual, motor and cognitive abilities due to natural aging. For example, blood glucose meters help many older adults manage their diabetes, novel media players can help adults with dementia access and enjoy music (Sixsmith, Orpwood, & Torrington, 2010) and innovative memory aids may help older adults remember important information (Inglis et al., 2003). Researchers have found that many older adults want to learn to use existing mobile technologies but have difficulty doing so (Kurniawan, Mahmud, & Nugroho, 2006; Massimi, Baecker, & Wu, 2007). For example, in a 2009 UK survey of 2905 people age 16+, respondents who owned mobile phones were asked how confident they felt about performing a range of typical mobile phone tasks (Ofcom, 2009). The survey revealed that 47% of the older mobile phone owners (age 60+) wanted to use their phone to take a photo and 22% wanted to send a text message but could not confidently do so. In addition, the proportion of older mobile phone users who could perform each of these tasks was much lower than the proportion of all surveyed mobile phone owners who could perform each task (17% and 40% compared to 56% and 81%, respectively). The difficulties that older adults experience in learning to use existing mobile phones may have contributed in part to the lower adoption of mobile phones by this population. Only 49% of surveyed older adults reported owning a mobile phone, compared to 82% of all surveyed adults. 2 The difficulties older adults have with learning to use mobile devices appear to be due in part to older adults' abilities and experiences. The natural decline in older adults' cognitive abilities makes it harder for them, compared to younger adults, to learn new computer skills (Fisk, Rogers, Charness, Czaja, & Sharit, 2009). Older adults also generally have less experience with computer user interfaces (UIs) and more experience with mechanical and electromechanical interfaces, which might make it harder for them to learn exiting computer interfaces (Docampo Rama, de Ridder, & Bouma, 2001). Adding to the challenges in learning to use a device, many older adults have difficulties operating mobile devices due to lower-level perceptual and motor issues such as sensitivity to glare (Marmor, 1998) and difficulties with target acquisition (Moffatt, 2010). In addition, the learning difficulties older adults experience appear to be due in part to the design of existing mobile device UIs. Existing mobile technology, such as mobile phones, are generally not designed for older adults (Kurniawan et al., 2006). In a 2001 study, older adults reported that mobile devices had too many functions for them (NTT DoCoMo, 2001). Another study, published in 2005, compared novice cell phone users between ages 20-35 and 50-64, and found that the group of older adults took over 40% more steps beyond the minimum number of required steps than did the group of younger adults (Ziefle & Bay, 2005). In a study later published in 2007, older adults similarly reported that mobile phones had too many menus, and that the menus were complicated and difficult to understand (Kurniawan, 2007). The problem at hand is that while mobile computer technologies have the potential to help many of today's older adults, existing mobile devices appear to be difficult for this population to learn to use, adopt, and benefit from. There is evidence that this problem may only get worse in the foreseeable future as mobile devices become more feature-rich and complex. We also believe that today's younger adults will not be immune from these difficulties when they have to learn to use future mobile technology as tomorrow's older adults. Mobile device UIs, and the user interactions required to use them, are evolving at a rapid pace; for example Apple's innovative iPhone and multi-touch interactions are now ubiquitous, but were only introduced into the market five years before this writing. Thus, we believe tomorrow's mobile devices will require continual learning from users, regardless of their previous experience. Mobile devices that are easier to learn to use are needed today and in the future, but there has been little research in this area. 1.1 Research goal The main goal of the research reported in this dissertation was to investigate design approaches proposed in the literature to assess whether any of them could help improve the learnability of mobile devices for older adults. We sought to find approaches that focus on redesigning mobile device UIs to make them easier for older adults to learn to use. We sought to answer two main research questions for each design approach we chose to investigate: 3 1. How can the design approach be used to improve a mobile device's learnability for older adults? 2. How much does the design approach improve a mobile device's learnability, for older adults? 1.2 Research strategy In this section, we present the overarching research strategy that we took, followed by details on how we defined learnability and focused on specific older adults groups and smart phones. After laying out the research strategy, we present in the following section the three design approaches we chose to investigate – increasing the interpretability of graphical icons, incorporating a multi- layered interface, and augmenting the mobile device's display – and our motivations for investigating them. 1.2.1 Breadth-based strategy Our research strategy was to investigate a number of different and promising design approaches to assess how they affect mobile device learnability for older adults. We investigated diverse design approaches (i.e., breadth-based strategy) instead of investigating one single approach more thoroughly (i.e., depth-based strategy); we discuss the benefits and drawbacks of this choice in Section 7.3.1. We chose our three design approaches because they would each potentially improve the learnability of mobile devices for older adults in different ways (see Section 1.3.3). For each chosen design approach, we aimed first to understand the specific design space of that approach informed by past research and by conducting an exploratory study if needed. We then sought to evaluate the approach empirically by means of conducting a user study, and to apply the approach to the design of a prototype if necessary for the evaluation. 1.2.2 Learnability defined Learnability is known to be an important component of usability (in addition to efficiency, memorability, errors and satisfaction; Nielsen, 1993) but there has not been a consensus on its definition (Grossman, Fitzmaurice, & Attar, 2009). The working definition of learnability used in our research was: the degree to which a system's interface enables novices to attain mastery in performing basic and advanced system functions. We focus on two types of learnability (Grossman et al.): initial learnability (i.e., supporting initial performance with the system) and extended learnability (i.e., supporting improvements in performance over time or learning to perform more tasks). Our definition's focus on attaining mastery (i.e., reaching a certain pre- defined level of usage proficiency) aligns with definitions used by other researchers (Grossman et al.; Santos & Badre, 1995; Nielsen). 4 1.2.3 Older adults without serious impairments For much of our research, we focused on adults of age 65 and older, who we refer to hereafter as older adults. This age 65 threshold is commonly used in aging and technology research (e.g., Czaja & Lee, 2007; Ofcom, 2008; Kurniawan, 2007; Rogers, Cabrera, Walker, Gilbert, & Fisk, 1996) because the majority of adults experience natural declines in cognitive, sensory and/or motor abilities by this age (Fisk et al., 2009). In addition, 65 is a useful age delimiter because it has been, at least at the time of writing, the normal retirement age in many countries. Thus this delimiter can be useful in separating older adults who live two different lifestyles: those who are retired and those who are still employed. Although the ability to learn to use technology has not been found to be directly associated with one's age but rather with one's working memory, processing speed, and other abilities that decline with age (Echt, Morrell & Park, 1998), age is a key metric for HCI research involving older adults (Dickinson, Arnott, & Prior, 2007). Having said this, it is well known that the older adult population is more diverse than the younger adult population. Describing "average" behaviour becomes less accurate as the age of the group being described increases (Hawthorn, 2000), so we report other user characteristics, in addition to age, to better characterize the groups we studied and to help the interpretation of our findings. We focused on investigating how design approaches specifically benefit older adults. Thus we recruited both younger adults and older adults for our studies to ground the older participants' data and to uncover age-related differences. Without the data from younger adults, it would not have been possible to determine whether findings from our studies were actually the result of age- related factors, and not other factors such as culture, socio-economic status, or computer experience. While we use the phrase age-related differences throughout this dissertation to refer to differences in findings between older and younger participants, we are actually referring to differences related to age-groups (e.g., group of adults between ages 20-49, group of adults of age 65+), rather than age itself. However, to increase readability, we refer to age-group-related differences simply as age-related differences. We targeted older adults who generally did not suffer from major declines in cognitive abilities (e.g., dementia) or physical abilities. Compared to older adults with major declines in abilities, older adults without these declines are more similar to younger adults and are more homogenous as a group, and thus were more suitable for helping us find age-related differences. 1.2.4 Smart phone applications We focused on the learnability of smart phone applications, which are software programs that run on these advanced phones. Even though older adults can benefit from using many types of mobile devices, smart phones are increasingly pervasive and are also some of the most complex of all mobile devices, requiring some degree of learning to operate. We did not improve device 5 learnability by means of redesigning the hardware UI (e.g., physical buttons). We generally used commercial hardware that could run our software prototypes. 1.3 Choosing design approaches to investigate To help us choose specific design approaches to investigate, we first considered existing design strategies to improving the learnability of computer user interfaces. To our knowledge, there are three main strategies that a mobile device UI designer can take. More than one strategy can be applied to redesigning a particular mobile UI. 1.3.1 General design strategies to improving learnability The first strategy is to permanently reduce the functionality (i.e., features) of the mobile device. This strategy reduces the complexity of the device and simplifies the UI, so that users have less to learn and less information to process during the learning process. This strategy appears to have been used to design many "simple", commercially-available phones for older adults (e.g., Samsung, n.d.; NTT DoCoMo, n.d.; Doro, 2010). The second strategy is to improve the design of UI elements, in order to increase the initial usability of the interface so that novices can more easily start to interact with the interface and perform tasks. The usability of UI elements can be improved in various ways, such as making them easier to perceive (e.g., by increasing size, contrast), easier to interpret (e.g., by using familiar language and metaphors), or easier to interact with (e.g., by improving controls and feedback). Much work in interaction design and research has followed this strategy to improve a UI's usability for novice users. It is well known in the human-computer interaction community that poor initial usability will negatively affect an application's overall learnability. Compared to the first strategy, this strategy also involves a permanent change to the UI, but not a reduction of functionality. The third strategy is to incorporate scaffolding into the UI, a key concept from the area of Learner-Centred Design (LCD). In LCD, scaffolds are defined as software features that temporarily support learners while learners mindfully engage in activities that they cannot yet perform (Soloway et al., 1996). As novice users gain expertise and confidence, the scaffolding is meant to be removed (by the user, or the computer system), leaving the learner with a user interface that is more similar to those used by experts (Quintana, Carra, Krajcik, & Soloway, 2002). The temporary nature of the scaffolding support is a key differentiator of this strategy from the other two. Scaffolds are particularly suitable for helping a heterogeneous group of learners to learn at their own pace, which is preferred by older adults (Fisk et al., 2009). Different types of scaffolding have been identified: supportive, reflective, intrinsic and motivational (Jackson, Krajcik, & Soloway, 1998; Lepper, Woolverton, Mumme, & Gurtner, 1993; see Table 1.1 for descriptions). Each type can support learning in different ways. 6 Scaffolding type Description Intrinsic Helps learners by reducing a task's complexity and focusing their attention on key elements to learning the task (e.g., temporarily restricting access to only basic functions). Supportive Helps learners to perform tasks by offering advice and support. This type of scaffolding includes guiding, coaching and modeling. As this scaffolding fades away, the task is the same as before but users are expected to have internalized the concepts which had been scaffolded. Reflective Supports learners by encouraging them to think about the task, for example, by planning how to perform the task or predicting task outcomes. Motivational Supports the learner's motivation (e.g., through praise and reassurance, error messages with extra information to help user recover) so that the learner continues to persevere through the learning process. Table 1.1: Different types of scaffolding that can be used to support a user during the learning process. 1.3.2 Three promising design approaches Based on the general design strategies above, we chose three different design approaches to investigate: increasing the interpretability of graphical icons, incorporating a multi-layered interface, and augmenting the mobile device's display. We decided at the outset of our research to avoid any approach that permanently reduces device functionality, because such an approach may remove features desired by many older adults, and a number of mobile phone manufacturers have already applied this type of approach to their devices. Instead, we chose the above three approaches because all three appeared to be promising for improving the learnability of mobile devices for older adults and they had not previously been well-explored on a mobile device platform or for helping older adults to learn. We also chose these approaches because they could enable older adults to learn by themselves, rather than with other people, to support independent learning. In this section, we introduce each of the three design approaches and describe our motivations for investigating them. Increasing the interpretability of graphical icons We first chose to investigate increasing the interpretability of graphical icons, which is a design approach for improving the design of UI elements. The design of an icon greatly affects its interpretability, its initial usability and the initial usability of the application in which the icon is found. Take for example the two sets of icons shown in Figure 1.1 that come from two handheld computer calendar applications that are both used to view schedule information for one day, one week or one month. The icons on the left have a greater resemblance to traditional paper calendars than do the icons on the right. Users are generally familiar with paper calendars and would likely find it easier to identify the icon objects on the left than those on the right (assuming icons are the 7 same size). Users may find it hard to identify and interpret the icons on the right without additional information or prior experience. Figure 1.1: Example calendar application icons for viewing one day, week and month; icons are from the HP iPaq rx3715 (left) and the Palm Treo 650 (right). Correctly interpreting icons is particularly important in learning to perform tasks on a mobile device. Mobile devices have a wide variety of functionality that is not commonly available on laptop/desktop computers, such as a number of text entry methods and data connectivity options (e.g., Wi-Fi, Bluetooth, infrared and cellular). Many of these functions can only be accessed through unique icons and generic buttons (as opposed to dedicated buttons and other controls). In addition, there are many more operating systems (OS) for mobile devices (e.g., Android, iPhone OS, Symbian, Windows Mobile, PalmOS, Blackberry OS, and embedded Linux) than desktop operating systems (e.g., Windows, MacOS, and Linux), and each system typically uses its own unique set of icons. Finally, initial icon interpretability may be more critical for mobile devices because mobile devices appear to evolve more quickly so users tend to replace them more frequently than laptop or desktop computers (Associated Press, 2011). Despite the prevalence of icons, little work has investigated the influence of graphical icons on older adults' use of traditional computers or mobile devices. It seems likely that the decline in perceptual and cognitive abilities that accompanies normal aging might have some effect on older adults' ability to interpret graphical icons. First, an icon has a number of graphical properties that need to be processed (e.g., visual details such as lines and dots, its resemblance to a real-world object) in order to identify the visual objects being presented in the icon. This step may be more difficult for older adults with decreased perceptual abilities. Second, an icon is a pictographic item that represents one or more objects, a concept or a function (Peirce, 1932), so interpreting a new icon requires understanding the link between that item and the icon's function (Heim, 2007). Hawthorn (2000) questions whether older adults, compared to younger adults, are able to benefit as much from the cues provided by icons for interpreting the icon's function. Interpreting the icon also depends on other factors, such as the context provided by the software application in which the icon is used and the user's familiarity with the icon as well as with its application context (Horton, 1994). Compared with younger adults, older adults are less likely to have experience with contemporary handheld devices and be less familiar with existing icons and applications. Given the natural declines in visual and verbal abilities, and declines in the capacity to learn new associations, we speculated that some icons may be more difficult for older adults to initially use, and that there are opportunities to design icons that are easier for older adults to initially use. 8 Incorporating a multi-layered interface The second design approach we chose to investigate is a scaffolding approach that focuses on improving initial learning of software applications using the multi-layered (ML) interface approach (Shneiderman, 2003). ML interfaces can be designed to support learning such that novices first learn to perform basic tasks by working in a reduced-functionality, simplified layer (i.e., version) of the interface (see Figure 1.2, top screenshot). Once users have mastered this layer or require more advanced functionality, they can transition to increasingly complex layers and learn to perform more advanced tasks (see Figure 1.2, bottom screenshot). Figure 1.2: An example of a multi-layered interface. Layer 1 (top) allows the user to type and to perform a few functions; Layer 2 (bottom) has additional buttons, a ruler, status bar, and different font options. Figures from Shneiderman, B. (2003). Promoting universal usability with multi-layer interface design. Proceedings of the 2003 Conference on Universal Usability, p. 4. © 2003 ACM, Inc. doi: 10.1145/957205.957206. Reprinted by permission. Thus ML interfaces can provide a form of intrinsic scaffolding. Specifically, ML interfaces reduce an application's complexity (e.g., functions, content) during the learning process, thereby helping learners focus on key elements to begin performing tasks. Functionality-reduced layers are likely to place fewer demands on the user's working memory, which may especially benefit older users. Although to our knowledge little research has explored the learning benefits of ML interfaces for older adults and no research has evaluated ML interfaces on mobile devices to 9 improve learnability, ML interfaces might have the potential to help older adults (or users of any age) learn to use mobile technologies. Augmenting the mobile device's display The third design approach we chose to investigate is another scaffolding approach, one that is centred on the idea of providing supportive scaffolding, through an additional larger display, to help older adults during the learning process. Mobile device users already have access to a number of supportive scaffolding resources, such as manuals, information on the Internet, and people in their social circle. However, given the difficulties older adults have with learning to use mobile devices, it is not clear how effective current resources are in helping them and whether older adults actually use or want to use them. These resources are generally not closely integrated with the tool being learned, and interactivity is often limited. Our investigation into using an additional display to provide supportive scaffolding was further motivated by the capabilities of current mobile devices to easily connect to other devices and displays, offering an opportunity to overcome the limitations of a small screen and build new types of novel learning support resources. The mobile device's small screen limits the amount of interactive visual help that the device can provide to the user, a feature that is pervasive in many desktop applications today. On a small screen it is very difficult to simultaneously display both the application to be learned and the related help content that may be required for learning. Augmenting the mobile's small display with a larger display, such as a desktop monitor (see Figure 1.3) or a wall display, helps to overcome this limitation. Researchers have already begun using mobile devices together with large screen displays (e.g., Greenberg, Boyle, & Laberge, 1999), but to our knowledge no research prior to our work explored how an additional display can be used to facilitate the learning of mobile devices. This extra larger display can provide guidance and feedback (e.g., overlays that indicate which button to press next on, encouraging feedback after a subtask is successfully completed) in a variety of media and formats (e.g., text help, demonstration videos, real-time access to domain experts). This design approach appears to have much potential for creating a resource that can help older adults while they learn to use mobile devices. 10 Figure 1.3: We proposed augmenting a mobile device interface (left) with a larger display (right) that provides supportive scaffolding to help the novice user to learn to perform tasks with the mobile application. 1.3.3 Comparing design approaches We investigated these three particular design approaches because they had potential for improving mobile device learnability in different ways. To understand how our three design approaches relate to each other in the design space of improving the learnability of computers, we discuss how they differ in four key ways. Permanency of learning support: The added learning support can be permanent or removable (by the user or computer). Increasing the interpretability of icons applies learnability support permanently. Our other two approaches apply scaffolding, which is inherently removable (though some users may depend on the scaffolding so much that they do not want to remove it). Scope of redesign: Our three approaches also involve interface redesign at different granularities. Increasing the interpretability of icons involves redesign at an individual UI element level. Incorporating multi-layered interfaces involves redesigning groups of UI elements (or a device's entire UI). Augmenting the device's display involves designing the UI of a system consisting of multiple displays. Source of learning support: The support can either be incorporated into the device that is being learned (i.e., built-in support), or into some other resource (i.e., external support). The learnability improvement of increasing the interpretability of icons and incorporating multi- layered interfaces is built into the device UI itself. By comparison, the learnability improvement resulting from augmenting the mobile device's display with an additional display comes from the additional display, a source external to the device. Changes in functionality: Augmenting the device's display with an additional display adds functionality to provide additional learning support. In contrast, the multi-layered interface approach generally removes functionality from one or more layers, to reduce working memory demands during the learning process, but its multiple layers and an added mechanism to switch 11 between layers also increases its overall complexity and functionality. Redesigning graphical icons does not change the device's functionality at all. By choosing three different design approaches, we are better able to explore the design space and understand which approach may be more appropriate to take in a particular case than in another. These different approaches are complementary and may be implemented together (e.g., a device running a multi-layered mobile application can be augmented with an additional display). Although we can compare these approaches at a high-level in the design space, they support learning in such different and complementary ways that formally comparing how they affect learnability (e.g., in a controlled experiment) is not particularly meaningful; thus we do not formally compare the learnability of our three approaches against each other in this dissertation. 1.4 Summary of research contributions The research presented in this dissertation makes four main contributions. We provide a high-level description of each contribution here and elaborate on each in the concluding chapter (Chapter 7). 1. Increasing the interpretability of graphical icons (Chapter 3): Experimental results confirm our hypothesis that older adults had greater difficulties interpreting existing mobile device icons, which offers evidence that the initial interpretability of existing mobile device icons can and should be improved for older adults. We also found that of the three characteristics we studied, semantic distance had the largest effect on increasing initial icon interpretability for older adults. We propose guidelines, based on our experimental results, for designing icons for older adults. 2. Incorporating a multi-layered interface (Chapter 4): We conducted an experiment to study the effects of using a ML mobile application over four distinct phases to learn basic and advanced tasks. We found that our ML interface's Reduced-Functionality layer, compared to our control interface's Full-Functionality layer, helped older participants more than younger participants to perform initial basic task attempts in less time. Older adults also perceived the complexity of the ML interface to be lower than the full- functionality control interface and preferred the ML interface for learning to use the application. 3. Understanding learning needs in order to augment the mobile device's display (Chapter 5): We conducted a large and comprehensive survey to better understand the learning needs and preferences of older adults. Quantitative and qualitative survey data offered many insights into how and why older adults use certain methods to learn to use mobile devices. 4. Augmenting the mobile device's display (Chapter 6): We designed and prototyped an augmented display system, Help Kiosk, for helping older adults learn to use a smart phone. To our knowledge, this is the first system that uses an additional larger display to 12 provide real-time guidance and feedback to help people learn to use a smart phone. We found preliminary evidence through an informal user study that our Help Kiosk prototype can assist older adults in learning to perform new mobile phone tasks. 1.5 Dissertation outline We begin in Chapter 2 by presenting background knowledge on aging and learning to use technology, as well as previous work related to each of our three design approaches. We then present our investigations into increasing the interpretability of graphical icons in Chapter 3 and incorporating a multi-layered interface in a mobile application in Chapter 4. Chapters 5 and 6 document our investigation into augmenting the mobile device's display; Chapter 5 presents our comprehensive survey that informed our design of an augmented display help system, which is presented in Chapter 6. In Chapter 7, we review the contributions of this dissertation, reflect on our research strategy and findings, and suggest a number of potentially fruitful avenues for further research. We include a number of appendices that present the materials used in our research. 13 Chapter 2 2. Background and Related Work In this chapter, we first present background information, summarizing work by both the HCI and psychology communities to understand older adults' challenges in learning to use computers and to help make the learning process easier. In the last three sections of the chapter, we present work related to each of the three design approaches that we chose to investigate. 2.1 Helping older adults learn to use new technology It is well established in the literature that older adults generally have more difficulty than younger ones in learning new skills, particularly in learning to use new technology (Fisk, Rogers, Charness, Czaja, & Sharit, 2009; Kelly & Charness, 1995). We first summarize past research on what hinders older people when they learn to use computer technology. We then discuss published guidelines for designing resources to help older adults learn to use technology. 2.1.1 User characteristics that hinder learning Researchers have attributed older adults' difficulty with learning to use technology to a number of user characteristics, including declines in spatial working memory, slower information processing, lack of relevant technology experience, and a higher negative reaction to errors. We considered these factors throughout our study designs, data analysis and prototype designs. We summarize research on the effects on these characteristics. Researchers have found that cognitive declines in spatial working memory, as well as processing speed, make it more difficult for older adults to learn to use computers (Echt, Morrell, & Park, 1998). Spatial working memory is the online mental capacity for processing and storing visual spatial information, and has been found to decline with age (Echt et al., 1998; Hawthorn, 14 2000). Perceptual speed is how quickly mental operations are performed, and this ability has also been found to decline with age, as well as decrease more for older adults than younger adults as task complexity increases (Fisk et al., 2009). Morrell and colleagues found that spatial working memory capacity, and perceptual speed were closely related to task performance and number of help requests, and were significant predictors of computer skill acquisition in certain instances (Echt et al., 1998; Morrell, Park, Mayhorn, & Kelley, 2000). Older adults generally have less experience with today's computers than younger adults, which means they have more to learn. In addition, older adults are generally more experienced with UIs of older technologies, such as electro-mechanical UIs in which most if not all the system's functionality is accessible simultaneously through mechanical controls (e.g., push buttons, switches, and dials) (Docampo Rama, de Ridder, & Bouma, 2001). In contrast, current mobile device UIs and those of other interactive computing technologies only show a subset of functions at once and often incorporate a navigational hierarchy to access specific task functions. Thus, unlike electro-mechanical UIs, mobile device UIs often have buttons (physical or on-screen) that perform different context-dependent tasks. As a result, the knowledge older adults have gained by using electro-mechanical UIs may not positively transfer to using existing computer UIs and may make it harder for them to learn these UIs (Docampo Rama, de Ridder, and Bouma, 2001). As some researchers have put it, older adults can be thought of as digital immigrants and younger adults who grew up using computers can be thought of as digital natives (Fozard, Bouma, Franco, & van Bronswijk, 2009). Thus, older adults may have to unlearn the user interactions they learned to operate older dissimilar interfaces. In addition, they may have to relearn words that have taken on new definitions (e.g., "mouse", "web") and learn new terms and metaphors (e.g., wrench or gears for "settings"), essentially gaining literacy in a somewhat foreign language. Further, difficulties in learning to use new computer technology often lead to more errors, which have been found to more negatively affect older users (Birdi & Zapf, 1997). Birdi and Zapf looked at how workers between ages 19 to 55 responded to errors, and found that older workers compared to younger ones were more negatively affected by an error and were less likely to try to solve the problem that led to the error on their own. 2.1.2 Designing learning resources for older adults Given the unique needs of older adults in learning to use technology, much research has been carried out to identify effective ways to help them in the learning process. That research has largely focused on designing better training programs and instructional resources. For example, researchers have studied what guidance (Hickman, Rogers, & Fisk, 2007), how much guidance (Morrell et al., 2000), and what media formats (Echt et al., 1998) are most suitable for older adults. In contrast, very little research has focused on how to increase the learnability of computer technology by redesigning the UI, which is our research goal. A notable exception is the use of 15 multi-layered interfaces (Dickinson, Arnott, Newell, & Hill, 2007), which we present later in Section 2.3. Despite the lack of research on improving the learnability of computer technology and mobile devices, research on developing better training programs and learning resources to help older adults learn to use computers offers a wealth of recommendations that are relevant for our research, particularly in designing scaffolding for older adults. Fisk et al. (2009) summarized many of these recommendations, including the following: • Provide self-paced learning, which is preferred by older adults. • Include support to help build the user's confidence, immediate feedback, and motivating exercises that lead to an attainment of mastery within a reasonable period of time, because older adults experience greater frustration and anxiety than do younger adults when learning complex tasks. • Provide multiple exposures of learning material over time, which is seen to be more effective than exposure to material in one session. • Minimize working memory demands. • Provide cues and aids. • Avoid overloading learners with too much information. • Avoid requiring learners to make complex inferences or fill in gaps of missing information. Further, when training is provided by computers, researchers have found it important to focus not only on the pedagogical aspects of the training but also on ensuring that the computer UI is easy for older adults to use (Hawthorn, 2005). 2.2 Graphical icons The initial interpretability of graphical icons, as well as other UI elements used in mobile devices, affects the application's initial usability and learnability. We first discuss whether existing icon design guidelines are applicable for older users. We then summarize past work researching the effect of concreteness and labels on icon interpretability, which is related to our experiment that we present in Chapter 3. 2.2.1 Generalizability of existing design guidelines Past work on designing icons has identified many icon characteristics that affect icon interpretability, such as physical characteristics (e.g., visual detail, colour, size), choice of icon object(s) associated with the intended icon meaning, and how that object is depicted (Heim, 2007). Researchers have also identified various user-related characteristics (e.g., intelligence, experience, culture) and the context in which the icon is found (e.g., mobile device capabilities, task, and software application interface) as factors that influence icon interpretability (Horton, 1994). A 16 number of studies have looked closely at the effects on icon usability of some of these characteristics, such as animation (Baecker, Small, & Mander, 1991), and spacing and size (Lindberg & Nasanen, 2003). There is a wealth of knowledge on icon design but few studies have looked at whether existing guidelines generalize to older adults. Many icon design guidelines have been published, suggesting for example: "Design icons to identify clearly the objects or concepts they represent" (Sun Microsystems, 1999, "Designing Icons" section, para. 3), "Icons should be suggestive of the functionality with which they are associated." (Benson, Clark, & Nickell, 2010, "Design Functionally Suggestive Icons" section, para. 1), and "Icons should be familiar to the user" (Heim, 2007, p. 430). However, these guidelines are not age specific and the implied assumption is that they apply universally to all ages. We do not know whether existing guidelines are appropriate for older adults because, to our knowledge, these guidelines are based on research involving only younger adults. Thus, we were motivated to investigate whether there are age-related differences in using icons, in order to understand whether existing icon design guidelines can be generalized to older adults. 2.2.2 Effect of concreteness and label on icon interpretability Little of the work on designing computer interfaces for older users has looked specifically at initial computer icon usability beyond investigating sensory-related icon characteristics, such as icon size (Siek, Rogers, & Connelly, 2005) and colour and contrast (Hawthorn, 2000), which affect the accessibility (i.e., lower-level usability) of the icon. We sought to take a step towards filling this gap by looking at icon characteristics such as concreteness (how closely an icon depicts real world objects, places or people), semantic distance (closeness of the relationship between the objects depicted in the graphic and the function being represented) and presence of labels that are related to learning the icon's function (see Section 3.2 for some of the motivation behind this choice). Related to our work, McDougall, de Bruijn, and Curry (2000) conducted a series of experiments to investigate the effects of icon concreteness and visual complexity on tasks involving visual search and matching icons with labels. Visual complexity refers to the amount of visual detail or intricacy, such as lines and shading, in the icon. Although concrete icons often have more visual detail than abstract ones, McDougall, Curry and de Brujin (1999) found concreteness and visual complexity to be two distinct dimensions. The icons used in these experiments were taken from a corpus of 239 icons that included graphics from road signs and electronic symbols, as well as computer icons. McDougall et al. (2000) conducted one particular experiment with 20 participants (recruited from a local university; age was not reported nor a factor in the study). After being given a label, participants were asked to select the associated icon from a set of 9 icons. Participants were faster and more accurate in initial attempts with concrete icons than with abstract icons. 17 Past work has also looked at the effect of labels on initial icon usability. Wiedenbeck (1999) conducted an experiment in which 60 undergraduate students with little computer experience were asked to operate a desktop email program using buttons that had icons, labels, or a redundant combination of icons and labels. Usability was measured by correctness of the tasks performed, time to perform tasks, and number of times the help facility was accessed. It was found that participants performed better with text labels than unlabelled icons during initial use. Participants performed similarly with labelled icons and labels alone on each of the three usability measures, but significantly better than with icons alone. Further, participants reported finding the labelled icons the easiest to use. To summarize the work related to the investigation of our first design approach, little prior research had explored how to design graphical icons for older users. We explored age-related differences in the effects of concreteness and labels, plus semantic distance, on initial icon interpretability. In addition to focusing on older adults, our study differed from the studies by McDougall et al. (2000) and Wiedenbeck (1999) because we showed our participants existing icons from mobile devices, and presented other contextual information such as a screenshot and a description of the application in which the icon was used. 2.3 Multi-layered interfaces Multi-layered (ML) interfaces have been seen as a suitable approach to helping novice users to learn to use software interfaces (Shneiderman, 2003). 2.3.1 Multi-layered interfaces to support learning Novices can learn with multi-layered interfaces by starting with an initial layer that has a limited function set, and progressing to more full-functioned layers when needed or when novices are comfortable using the initial set. Multi-layered interfaces are often used in games, in which new players begin with simple challenges to get used to the interface control and the game concept, but then progress to increasingly harder challenges that use more of the control interface (Shneiderman, 2003). A number of studies have found that ML interfaces can improve the learnability of desktop software (Carroll & Carrithers, 1984; Catrambone & Carroll, 1986; Findlater & McGrenere, 2007), and that they can improve experienced users' satisfaction in using feature-rich software (McGrenere, Baecker, & Booth, 2002). However, to our knowledge no research has evaluated ML interfaces on mobile devices to improve learnability, and little research has studied the learning benefits of ML interfaces for older adults. Carroll and his colleagues' work on evaluating the original Training Wheels (TW) interface studied the benefits of learning with a ML interface that was incorporated into a desktop word processor application (Carroll & Carrithers, 1984). In the TW's reduced-functionality layer, advanced functions were blocked (i.e., the names of these functions were visible but the functions 18 were disabled) and some features were disabled to prevent users from making common errors, such as printing before creating a document (Carroll & Carrithers). Twelve temporary office workers (whose ages were not reported but were presumably not older adults) were recruited. Participants were asked to perform a "basic" word processing task (creating and printing a letter) within a two-hour time limit; half of the participants performed the task on a reduced-functionality TW version of a commercial word processor while the other half performed the task on the full- functionality version of the application. The study showed that participants were faster and less error-prone when learning to perform the given task on the reduced-functionality interface compared to the full-functionality version of the application. Results from the Carroll and Carrithers' study (1984) were replicated in a follow-up study (Catrambone & Carroll, 1986), which also evaluated the transition from the TW interface to the full-functionality interface. For this follow-up study, 16 temporary office workers (ages not reported) with little computer experience were recruited; half used the reduced-functionality word processor to complete a series of tasks while the other half used the full-functionality version. Participants using the reduced-functionality interface were faster and less error-prone when learning to perform the basic letter creation task compared to the full-functionality version of the application. After attempting a basic task on either interface version, all participants were asked to perform a similar task using the full-functionality interface. This study found that both groups of novices performed similarly in terms of task completion times on the full-functionality interface, suggesting that learning on the reduced-functionality layer did not make it harder to perform the same tasks on the full-functionality layer. This study also asked both groups to perform advanced tasks on the full-functionality layer and found that novices who learned to perform basic tasks on the reduced-functionality layer were faster than those who learned on the full-functionality layer. This performance benefit was attributed to the TW participants spending less time making mistakes. 2.3.2 Multi-layered interfaces to help older adults Almost no research has looked at incorporating ML interfaces into applications to help older adults learn to use computer technology. Two exceptions are a desktop software tutorial called FileTutor (Hawthorn, 2005) and a ML web portal (Dickinson et al., 2007). FileTutor was designed to help older adults learn to use the Windows Explorer file management application (Hawthorn, 2005). FileTutor allows learners to perform a number of exercises on an embedded reduced- functionality version of Windows Explorer in which advanced functions were hidden. It should be noted that Hawthorn labelled this application as a "training wheels" version of Windows Explorer even though advanced functionality was hidden and not blocked, demonstrating that terminology related to reduced functionality interfaces is not yet standardized. Although this reduced- functionality interface has not been formally evaluated (e.g., no pre-testing, no comparison with a control interface), a "quasi-experimental validation" (Hawthorn, p. 2) with 25 older adults (ages 19 60-88) who had previously failed a file management learning module demonstrated that FileTutor could help them learn to perform many file management tasks on Windows Explorer with fewer help requests. Another notable system is the ML web portal for a desktop computer designed by Dickinson et al. (2007) to help older adults navigate and search the Internet. This web portal design had multiple layers: two functionality layers (one with basic functions only, and a second with basic and advanced functions), and three content layers (highly accessible content controlled by a development team, content from the development team and third parties who conformed to similar standards, and any Internet content that did not necessarily conform to standards). This ML web portal was evaluated in comparison to a commercial web portal of a major Internet service provider. Eleven older adults (ages 63-87) with some basic computer experience were recruited for the study. The study involved two sessions: participants were asked to perform basic tasks on both the ML and control interfaces in the first session and a combination of basic and advanced tasks on the two interfaces in the second session. Participants made fewer errors over the two sessions using the ML web portal and commented positively on the simplicity of its interface. Although participants completed more given tasks in the first session using the ML interface than with the control interface, participants completed a similar number of tasks in the second session on each of the two interfaces. Dickinson et al. also tried to assess learnability through measures of unassisted repetitions and progressions, but did not find differences in these measures between the two interfaces. The lack of significant results despite a strong preference for the ML portal may be due to the advanced tasks for the two portals not being comparable in difficulty (i.e., there were non- isomorphic tasks between interface conditions). The study also recruited participants who had a wide range of abilities, so 11 participants may not have been a large enough sample to find statistically significant results. 2.3.3 Designing layers A common issue in designing multi-layered interfaces is deciding how to present functions that are disabled in the initial reduced-functionality layers. Two approaches have been used in the literature: blocking and hiding these functions. The blocking approach allows disabled features to remain visible, but a user's attempt to use a blocked function results in no activity, or is greeted with a system message saying that the function is disabled. The original Training Wheels interface for a word-processing desktop application incorporates the blocking approach (Carroll & Carrithers, 1984; Catrambone & Carroll, 1986). In contrast, the hiding approach removes the disabled functions from view leaving only the available functions visible. Findlater and McGrenere (2007) conducted an experiment to compare the blocking and hiding approaches. The experiment had three interface conditions: a reduced-functionality interface that blocked disabled functions, another reduced-functionality interface that hid disabled functions, and a full-functionality interface. Blocking was implemented by positioning "x"s by the disabled 20 functions. Thirty participants (ages 19-55), students and community members, were recruited for this study. Findlater and McGrenere found that the reduced-functionality interface that hid the disabled advanced features helped participants complete tasks faster than did the full-functionality interface. In addition, participants preferred the interface that hid the disabled advanced features over the full-functionality interface. Findlater and McGrenere also found that the interface that blocked disabled functions did not have significant performance benefits over the other two interfaces and was the least preferred of the three. The study by Findlater and McGrenere (2007) provides evidence that the hiding approach is better than the blocking approach in learning to perform tasks on a reduced-functionality layer. A hiding approach also seems to be more suitable for mobile applications because it does not visually present disabled functions that would take up limited screen real estate. Further, hiding functions rather than blocking them reduces the number of items shown on the screen, which arguably decreases the complexity of the interface and thus could be more beneficial for older adults. To summarize the work related to the investigation of our second design approach, the foregoing studies provide evidence that ML interfaces can improve the learnability of new technology in terms of errors, task completion time and number of tasks completed successfully. There is also some preliminary evidence that ML interfaces can help older adults to learn new technology. However, it is not clear how well results of past studies involving desktop applications generalize to mobile applications; performing mobile application tasks through the device's small screen and limited set of buttons is very different than performing desktop applications tasks. In addition, many open questions remain, such as how to select the functions that go into each layer, and how to design the interface to easily transition from one layer to another. Our study extends this past work by exploring the learning effects of ML interfaces on mobile applications and involving multiple age groups to identify age-related differences in these effects. 2.4 Augmented displays Our investigation into the augmented display design approach focused on how one would design a system using a larger display to help older adults learn to use mobile devices. No past research had looked at this approach, so we first turned to the literature on designing learning resources for older adults, and to understanding older adults' needs and preferences in learning to use mobile phones and other technology. We then consulted the literature for insights on designing learning and help resources, such as manuals, online help systems, and augmented reality systems used for learning. 21 2.4.1 Designing a novel learning resource for older adults As mentioned earlier, there are many available recommendations for designing training programs and instructional resources (Fisk et al., 2009), but researchers have expressed a need for more assistance in applying them. Czaja and Lee expressed this need well when they wrote that "[existing] guidelines do not indicate what type of training technique is best suited for a particular task, technology or application." (2007, p. 346). We found this lack of clear direction particularly evident when we began to design our augmented display system. For example, should the system focus on helping older adults to learn how to perform tasks, to build a correct mental model of the UI, or both? Further, should the system primarily support trial & error, a common strategy for learning mobile devices, or provide training, a strategy studied more in the literature? A design cannot practically include all features that have been found to help older adults acquire new computer skills because such a system would likely overwhelm the older learner's cognitive resources. Thus, we wanted to first understand the learning needs and the preferences of older adults in order to understand the functional requirements of our system. 2.4.2 Older adults' learning needs and preferences Several research teams have recently investigated which resources older adults prefer when learning how to use technology. Selwyn, Gorard, Furlong and Madden (2003) conducted a survey study that examined older adults' access to information and communications technology (ICT) such as computers, video game machines, and televisions, and analyzed data from 352 adults of age 60 and older. Mitzner et al. (2008) conducted a focus group study with 113 older adults (ages 65-85) to investigate their training needs and preferences for technologies used in the home (but did not specify which). Kurniawan (2006) conducted a focus group with 7 older women (median age: 67.5) to explore how the group learned to use a new mobile phone. Findings from these studies have been mixed on which learning methods older adults prefer. Selwyn et al. (2003) found a strong preference for trial & error and that friends, family and work colleagues were rarely consulted for ICT support. One limitation of this study is that it did not include use of manuals. Kurniawan (2006) found that manuals were used most but only after trial & error were unsuccessful. Mitzner et al. (2008) found that older adults had a strong preference for using manuals over trial & error. Further, a limitation of this body of work (Mitzner et al.; Kurniawan; Selwyn et al.) is that researchers did not compare their older participants' responses with data from younger adults, making it difficult to determine whether preferences were due to age or perhaps to other factors, such as social support. The mixed findings from these studies, and the lack of direction on how to apply existing guidelines were key motivators for conducting our survey study on the learning needs and preferences of older adults Our survey (presented in Chapter 5) looked at older adults' learning method preferences in a new way. First the survey captured data from both younger and older adults so that we could 22 analyze age-related differences. Second, it focused on learning to use mobile devices rather than on other technologies. Third, the survey included many open-ended questions that focused on older adults' reasons behind their preferences. The survey included questions on all learning methods that were identified by these related studies. 2.4.3 Designing manuals and online help Past work on designing instruction manuals, online help, and other resources offers some design principles (e.g., Carroll, Smith-Kerker, Ford, & Mazur-Rimetz, 1987; Duffy, Palmer, & Mehlenbacher, 1992; Rieman, 1996), but this work has generally focused on learning to use desktop computer applications and supporting workers and other younger adults. Thus, it was unclear to us whether these principles would be relevant to today's mobile phones, or whether they would appropriately address the unique needs of older adults. Much work on helping the general public learn to use desktop computers and their applications has focused on improving instruction manuals. One of the key research developments was the Minimal Manual (Carroll et al., 1987). Prior to the Minimal Manual, instruction manuals generally prescribed users to carry out sequences of drill and practice exercises. In contrast, the Minimal Manual was specifically designed to help learners perform actual tasks and recognize and recover from errors. Today's manuals follow many of the Minimal Manual design principles. However, the improved learnability of recent computer UIs and the availability of online help, which is embedded directly in software applications, appear to have reduced users' need for a manual (Novick & Ward, 2006). As a consequence, manuals are often not included with today's computing devices and software products. However, many older adults still prefer and benefit from manuals (Kurniawan, 2006; Mitzner et. al, 2008). Our survey (Chapter 5) identified those aspects of the instruction manual that support older adults in learning to use mobile devices to determine how much our augmented display system should be based on the manual. Research has also focused on developing better online help systems. Online help systems deliver performance-oriented, procedural or reference information through computer software (Duffy et al., 1992). Online help systems support task performance rather than the more conceptual learning that is often provided in courses, tutorials, and training materials. Many design guidelines have emerged from this work, including a proposal that online help content needs to target the task, be easy to access, and lead to efficient transfer from the help system to the task (Duffy et al.). However, this work is primarily based on desktop computer applications and user testing with university students and workers (age<65). Past online help guidelines may not generalize to mobile devices because their small screens limit the amount of help that can be shown along with the main application. In addition, few studies have looked at whether online help systems need to be designed differently for older adults, whose cognitive abilities decline naturally with age making it harder for them to learn to use new technology like online help (Fisk et al., 2009). 23 A notable exception to the lack of work on designing online help for older adults is the FileTutor system (Hawthorn, 2005), first mentioned in Section 2.3, which runs entirely on a desktop computer and has been found to help older adults learn to use a file management application. This self-paced tutorial was divided into two halves where users first learned basic concepts about files, directories, drives, and computer technology. This first half was found to be crucial to the tutorial because many participants did not have basic computing knowledge (e.g., knowing what the term "select" meant, dragging and resizing windows). In the second half of the tutorial, participants used a reduced-functionality version of Windows Explorer to carry out file management tasks. Although FileTutor was not formally evaluated, making it difficult to know how well the research findings generalize, we consider FileTutor's reported content and interface design decisions in designing our augmented display help system. 2.4.4 Augmented reality as a help system Little work has examined augmenting a mobile interface with other computer technology to help a novice learn to use the mobile application. However, two documented research systems provided ideas for the design of our augmented display system. From the area of Augmented Reality, CounterIntelligence (Bonanni, Lee, & Selker, 2005) is a working kitchen augmented with sensors and projected displays that are used to help users work more efficiently in the kitchen. CounterIntelligence can project recipes and instructions on cabinets and other surfaces to indicate in which drawer a kitchen utensil is located by illuminating the drawer handle, and to provide other cooking support. Origami Desk (Ju et al., 2002) is a kiosk that also uses sensors and projected displays to help users to fold pieces of paper into one of two origami shapes. This kiosk provides demonstration video clips to illustrate performing various folds, animations projected on the origami paper to show where the folds lie, and fold sensing to give users positive feedback when they successfully complete a step. Although neither system was formally evaluated, both CounterIntelligence and Origami Desk were found to provide users with the supportive scaffolding needed to accomplish new tasks. Although the augmented display system that we designed (Chapter 6) does not involve augmented reality, our system's feature for suggesting which specific UI elements the user should use for a given step (i.e., Live View) was inspired by CounterIntelligence and Origami Desk. To summarize the work related to the investigation of our third design approach, no research has explored how to augment a mobile interface with a larger display (or other computer technology) to provide real-time interactive guidance and feedback to help a novice learn to use the mobile application. However, there has been some evidence that such scaffolding can help older adults to learn new computer skills. We built on past work by evaluating how well this approach helps older adults and exploring how to apply this approach to a mobile device. We considered many research and design challenges related to this approach, which include: What technology would be suitable for augmenting mobile device interfaces? What kind of instructions 24 should be provided? How can the mobile application and the additional technology be integrated so that both can be easily controlled by older adults? How well will older adults be able to perform tasks that they learned with the supportive scaffolding after the scaffolding is removed? Our goal was to better understand older adults' needs and preferences in learning to use mobile devices. Our survey study (Chapter 5) helped us sort through and apply existing guidelines. 2.5 Summary The survey of related work presented in this chapter highlights the lack of published work on improving the learnability of mobile devices for older adults. Most work on learnability has focused on desktop UIs and not mobile device UIs. In addition, and more generally, very little work has focused on improving the learnability of computer technology specifically for older adults, beyond designing better training programs and instructional resources. There is a clear need for research that includes both older and younger adults in order to identify learnability benefits unique to older adults. Past work provides some evidence that our design approaches have the potential to help older adults learn to use new technology, and offers insights into specifics of each design approach that might best improve the learnability of mobile devices. This dissertation provides additional evidence and suggestions for future research, as well as practical recommendations based on our research findings, which can be used by developers of new systems. 25 Chapter 3 3. Increasing the Interpretability of Graphical Icons: An Experiment In this chapter, we present our investigation into our first design approach: increasing the interpretability of graphical icons. We first identify the objectives of our investigation and then describe our research methodology. We next present an exploratory study we conducted to identify icon characteristics that most affect initial icon interpretability for older adults. We then describe an experiment we designed and conducted to evaluate the age-related effects of concreteness, semantic distance and labels on initial icon interpretability, followed by a summary of the key findings from the experiment. Finally, we discuss the implications of our findings and propose guidelines for increasing the interpretability of icons for older adults that are based on the findings. 3.1 Objectives and methodology Our goal for this research was to empirically determine which icon characteristics had an age- related effect on initial icon interpretability, with the overall aim of understanding how to design icons that are more usable by older adults. We sought to identify which icon characteristics help or hinder interpretability, and to determine experimentally whether the effects of those characteristics differ across age groups. With a view to technology adoption, we focused on the initial interpretability of existing mobile device icons, specifically on icons employed on smart phones, handheld computers and personal digital assistants (PDAs). To ground our understanding of age-related differences in icon interpretability and to identify icon characteristics that might cause more problems for older adults, we began with a qualitative exploratory study that had 10 participants of varying ages. After identifying a number of icon characteristics that appeared to have a greater effect on initial interpretability for older adults, we conducted an experiment with 36 participants, half older adults and the other half younger adults, 26 to investigate in-depth the specific influence of each of these factors and how these influences differ across age groups. 3.2 Preliminary exploratory study We first carried out a qualitative exploratory study to identify icon characteristics that might have an effect on older adults' interpretation of icons. As stated earlier, we focused on investigating characteristics beyond sensory-related ones (e.g., icon size, colour, contrast, and visual complexity). We recruited 10 participants from three age groups (20s, 60s and 70s). All participants reported good or corrected vision, some computer experience, some cell phone experience and little to no experience with more advanced mobile devices. Participants were first required to look at unlabelled icons, to identify the object(s) they saw in each icon and to indicate which device function they assumed to be associated with that icon. Icons were retrieved from screenshots of two commercial mobile devices, an HP iPaq rx3715 and a Palm Treo 650, and they were shown on a laptop computer enlarged approximately two times. In addition, participants were asked to complete a series of tasks on each device, such as finding icons for a particular application (e.g., camera program) or function (e.g., the help function). Finally, participants were shown pairs of icons that were used to identify the same function on the iPaq and on the Treo, and they were asked to choose from each pair an icon that they found more usable and to explain their choice. There were two important findings from this exploration. First, the older participants were less accurate in identifying icon objects and in interpreting icon meanings than were the younger participants, confirming our initial speculation. To illustrate, the older participants were unable to correctly identify the Rolodex card in the Contacts icon or interpret the meaning of the Tasks icon depicted in Figure 3.1. Figure 3.1: Icons – "Contacts" (left) and "Tasks" (right) – from the Palm Treo 650 which were found to be more difficult to use by older adults. Second, we identified, based on participants' responses, three icon characteristics that appeared to have an effect on icon initial interpretability by older adults (see Table 3.1 for definitions). When required to choose between paired icons and to explain their choices, participants tended to prefer and focus on icons depicting something familiar. Participants also preferred concrete icons – icons that depicted object(s) resembling real-world items, places or 27 people (McDougall, de Brujin, & Curry, 2000) – compared to abstract icons. In Figure 3.2, the icons in the top row are more concrete than icons on the bottom row. Further, participants preferred semantically close icons, icons with a close link between the depicted object and associated device function, compared to semantically far icons. In Figure 3.3, the icons on the left for zooming out and in (i.e., three trees and one tree, respectively) are semantically further than those on the right (i.e., "1x", "2x"). Although such a preference for familiar, concrete and semantically close icons is not surprising, it seemed that this preference was more pronounced in older participants. Icon characteristic Definition Concreteness How closely an icon depicts real world objects, places or people (McDougall, de Bruijn, & Curry, 2000). Semantic Distance The closeness of the relationship between the objects or concepts depicted in the graphic and the function being represented (McDougall, Curry, & de Bruijn, 1999). Label Presence or absence of a text label for the icon. Table 3.1: Definitions of icon characteristics: concreteness, semantic distance and label. Figure 3.2: Icons can differ in concreteness. These icons are used in calendar applications for viewing a day, week and month. The icons from the HP iPaq rx3715 (top) are more concrete than the icons from the Palm Treo 650 (bottom). Figure 3.3: Icons can differ in semantic distance. The icons in the white outlines are used in camera applications for zooming in and out. The icons from the HP iPaq rx3715 (left) are semantically further than those from the Palm Treo 650 (right). 28 3.3 Methods On the basis of the findings of our exploratory study, we sought to understand more precisely through an experiment the extent to which concreteness and semantic distance affect icon interpretability by younger versus older adults. In addition, we wanted to understand the effect of labels on initial icon interpretability. We tested the following four hypotheses. All hypotheses had the same three dependent variables for assessing icon interpretability. Our two primary dependent variables were accuracy in identifying icon object(s) and accuracy in interpreting icon meaning, while our third dependent variable, secondary to the other two, was participants' confidence in their interpretations. H1. Overall interpretability. Older adults will have more difficulties than younger adults with mobile device icons. H2. Concreteness. Compared to younger adults, older adults will have more difficulties with abstract icons than with concrete icons. H3. Semantic distance. Compared to younger adults, older adults will have more difficulties with semantically far icons than with semantically close icons. H4. Label. Compared to younger adults, older adults will have more difficulties with unlabelled icons than with labelled icons. 3.3.1 Participants We recruited two groups of 18 participants, adults between ages 20-37 years old and adults of age 65+ years old, which are referred to hereafter as our younger and older participants. Table 3.2 gives an overview of the descriptive data for age, gender, education and verbal fluency for the two age groups. One of the younger participants was replaced because her responses suggested that she did not understand the experimental tasks. Participants were recruited through advertisements (see Appendix A.1) placed in the Vancouver Courier, a free local newspaper, and online classifieds (http://vancouver.en.craigslist.ca/). Interested potential participants were pre-screened over the phone to ascertain that they had at least some computer experience (e.g., regular use of a computer and experience with Internet browsers, email and word processing), functional eyesight, no colour blindness and fluency in English (see Appendix A.3). Interested participants were excluded if they had had any experience using handheld computers, PDAs or advanced smart phone functions (e.g., messaging, browsing web pages and taking photos), because we wanted our participants to be as unfamiliar as possible with the icons we used in the experiment. To confirm the screening results, eyesight and verbal fluency were tested at the study session using a reduced Snellen eye test and the FAS test (Benton & Hamsher, 1978), respectively. The results from these tests showed that all participants had normal, age-appropriate eye-sight and verbal fluency levels. No significant age group differences in verbal fluency were found. 29 Younger participants Older participants N 18 18 Age M (SD) 30.7 (5.7) 71.5 (5.7) Gender # male # female 3 15 7 11 Highest education attained # high school # some college # bachelor's degree # master's degree # doctoral degree 2 5 10 1 0 4 6 5 1 2 Verbal fluency M (SD) 42.9 (9.3) 39.3 (9.9) M=mean, SD=standard deviation FAS, verbal fluency: higher score=more fluent Table 3.2: Participant characteristics of the two age groups (N=36). 3.3.2 Materials In this section we first identify our study's experimental conditions. We then describe our process for selecting the specific icons used in the experiment, and how we presented them in the experiment sessions. We next describe the measures we recorded during the sessions. Conditions Participants were asked to interpret icons and labels from a number of experimental conditions. There were two levels for the independent variable concreteness (concrete and abstract), and two levels for semantic distance (semantically close and semantically far). We explain later in the Icon sets section how we found icons for each of the four concreteness+semantic-distance combinations. There were also three levels for the independent variable label (icon-only: icon shown without label; icon+label: icon shown with its associated label; and label-only: icon's label shown without the icon). Icon sets Three sets of 20 icons were selected for the experiment; one set was needed for each of the three label conditions presented in an experimental session. Icons were selected from 149 icons that were used on mobile devices that were popular in 2008 when the experiment took place (i.e., Sony Ericsson W610i and W850i, Blackberry 7730, Nokia N95, HP iPaq rx3715, Palm Treo 650, Apple iPhone). The icons were obtained from high-quality screen captures posted on the Internet. Three graduate students specializing in Human-Computer Interaction research independently rated the concreteness and semantic distance of each of the 149 icons. These raters were instructed to assume that they had never seen the icons before, but were familiar with the capabilities of the mobile devices from which the icons came from. They were given definitions for "concreteness" and "semantic distance" (Table 3.1) and the functions associated with each icon. They were asked to rate each of the icons on a five-point scale. These raters were first shown a set of icons as examples of various rating values (see Table 3.3). These raters scored the set of icons 30 independently, and their rating scores for each of the icons were averaged to produce one concreteness score and one semantic distance score for each of the 149 icons. Characteristic Rating Example icons Concreteness 5=definitely concrete for "chemistry" 1=definitely abstract for "air vent – right and left outlets" Semantic distance 5=very strongly related (i.e., semantically close) for "printing document" 1=not closely related (i.e., semantically far) for "hazard ahead" Table 3.3: Example icons given to our three icon characterizers prior to them being asked to rate 149 existing mobile device icons. Using these averaged rating scores, 60 icons were chosen to represent the four concreteness+semantic-distance combinations, which are depicted in Figure 3.4. We chose icons that had the lowest or highest concreteness and semantic distance ratings to represent abstract/concrete and semantically far/close icons. Figure 3.5 shows a sample of these icons and Appendix A.5 shows the complete set. We arranged the icons into three sets, with each set consisting of six concrete+close, four abstract+close, four concrete+far and six abstract+far icons (there were fewer choices for icons that could be categorized as either abstract+close or concrete+far, hence the unequal number within each set). In addition, for each set of 20 icons, we controlled for the number of icons from application user interfaces (UIs; i.e. icons on buttons to operate an application) and from a menu list (e.g., list of applications that could be launched from the main menu, list of preference settings); see Figure 3.6 for examples of these two types of icons. We did this because icons may be interpreted differently in these two contexts. Icons used in applications are often used to execute a function in the application; the meanings of these icons can be thought of as verbs. In contrast, icons used in settings menus and application launchers often represent applications, categories, or settings; the meanings of these icons can be thought of as nouns. Each set of 20 icons had 5 icons from application UIs and 15 icons from menu lists. To provide context, the application interface icons were presented in a screen capture that displayed all other icons used in the interface, and the menu list icons were usually presented with 31 at least one other icon from the menu. An image of the mobile device on which the icon was used was also shown to provide context (see Figure 3.7). 1 2 3 4 5 1 2 3 4 5 Concreteness Se m an tic d is ta nc e Concrete+Close (18 icons)Abstract+Close (12 icons) concreteabstract close far Concrete+Far (12 icons)Abstract+Far (18 icons) Figure 3.4: Mean concreteness and semantic distance ratings for, and groupings of, the 60 icons used in experiment; data point size reflects the number of icons characterized with a particular rating. Abstract+Close Concrete+Close Tasks Calculator Phone Alarm Send Exit Print Mail Abstract+Far Concrete+Far Preferences Zip Converter Settings Profiles Web Options Figure 3.5: Example icons representing the four concreteness+semantic-distance combinations. Document Editor 32 Figure 3.6: Icons from application UIs, such as a camera application (left), and from menu lists, such as an application launcher (right), were used in the experiment. Icons are from the HP iPaq rx3715. Figure 3.7: Example pages for icon+label (left), icon-only (centre), and label-only conditions (right); icons are from the Sony Ericsson W610i's program menu. For the icon+label and label-only conditions, we used the wording of existing labels unless they included a product name (e.g., "HP Image Zone", "Quickoffice") or abbreviated words (e.g., "Prefs"). To avoid such labels, we replaced five labels from the 60 icons with a more generic, unabbreviated word or phrase (e.g., Image Viewer, Document Editor and Preferences). Images of the icons were obtained from screenshots. Most icons were enlarged to approximately 2.54cm x 2.54cm (300dpi) and smaller icons were enlarged to approximately 1.27cm x 1.27cm (300dpi). This was done to minimize the effects of icon size, which could be especially pronounced due to individual differences in eye sight. Icons were enlarged without anti- aliasing. Icons were presented to users on paper instead of on a computer screen to minimize any possibility of glare, to which many older adults are especially sensitive (Marmor, 1998). Furthermore, icons were presented on paper instead of actual mobile devices so that our results 33 would not be influenced by participant's varying abilities to operate mobile devices. This removed one more source of cognitive load that probably would have been heavier for the older participants, which would have made the overall icon interpretation task more difficult for them. To help us systematically present icons during the study sessions, icons sets were labelled (A, B, and C) and icons were further grouped within sets by mobile device brand and by menu/application (e.g., one page showed camera application icons for the HP iPaq rx3715 and another page showed program menu icons from the HP iPaq rx3715). Thus, each of the three sets of icons was presented over six pages (one to eight icons/page). Pages were created for each of the three label conditions (e.g., see Figure 3.7). Measures To test all four hypotheses, data were collected, one score per icon/label, for the following dependent variables: • Accuracy in identifying icon object(s) (values: 0 or 1, no scores recorded for the label- only condition) • Accuracy in interpreting icon/label meaning (values: 0 or 1) • Confidence in interpreting icon/label meaning (self-reported on a scale of 1-4; 1="not sure at all", 4="very sure") • Icon/label familiarity (self-reported responses to questions described in the Procedure section and mapped to values from 1-10; see Table 3.4 for mapping) In order to focus on initial interpretability, we tried to recruit participants who were unfamiliar with the icons presented in the experiment. Realistically, however, it was not feasible to find people who were completely unfamiliar with all of our icons or anything similar, so participants' familiarity with icons/labels was collected to assess, and possibly control, its influence on the interpretability scores. Icon-only and icon+label conditions Score Description 1 Not see this icon before 2 Maybe have seen this icon before 3 Seen a very similar icon before but have not used 4 Seen this exact icon before but have not used 5 Seen a very similar icon before and have maybe used 6 Seen this exact icon before and have maybe used 7 Seen a very similar icon before and have used on computer or different device 8 Seen this exact icon before and have used on computer or different device 9 Seen a very similar icon before and have used on a mobile device like this 10 Seen this exact icon before and have used a mobile device like this Label-only condition Score Description 1 Not seen this label before on any technology 6 Maybe have seen and used this label before on some technology 8 Seen this exact label before and have used on computer or different device 10 Seen this exact label before and have used a mobile device like this Table 3.4: Familiarity score mappings for each of the three label conditions. 34 3.3.3 Experimental design A 4-factor mixed design was used: 2 age groups (ages 20-39, age 65+; between subjects) by 2 concreteness levels (concrete, abstract; within-subjects) by 2 semantic distance levels (close, far; within-subjects) by 3 label conditions (icon-only, icon+label, label-only; within-subject factors). As mentioned above, no accuracy scores for identifying icon objects were recorded for the label- only condition, so the design for analyzing accuracy in identifying icon objects only had two conditions for the label factor: icon-only and icon+label. Presentation order of the label conditions was fully counterbalanced whereas the presentation order of the three icon sets (described in the Procedure section) followed a Latin-square design. 3.3.4 Procedure All study sessions took place in a usability lab on the university campus, and were recorded using a video camera with audio. Each participant took part in a study session individually. After a participant gave consent, approximately 5 minutes was spent familiarizing participants with the functional capabilities of existing mobile devices (e.g., wireless connectivity, taking and viewing digital photos and contact information management). Participants were asked to list the capabilities that they knew of, and were informed afterwards by the experimenter of other existing capabilities. Participants were also given a reference sheet, which listed general capabilities of mobile devices, which they could use if desired. The experimenter only discussed the capabilities of mobile devices with participants, and did not demonstrate these capabilities on an actual device. For the rest of the study session, participants were shown the three sets of icons and were asked a series of questions for each icon/label that was presented (60 total). Icons were shown in three blocks, one block per icon set paired with a different label condition. The specific purpose/function of each application and menu list was described to participants, as the associated icons were presented, to help participants interpret the icons in context (e.g., for the application shown in Figure 3.6, left screenshot: "This camera application is used to capture photos. This is the viewfinder and these are the buttons used to operate this application"; for the menu shown in Figure 3.6, rightmost icons: "This menu lists a number of programs that a user can run on the device"). In the icon-only and icon+label conditions, participants were asked: • To identify the icon objects ("What is shown in the icon?"); • To interpret the icon's function ("How might the icon be used?" or "What would happen if you 'clicked' on the icon?"); • How familiar they were with the icon ("Have you seen this particular icon before? If so, have you used this icon before?"); and, • How confident they were with their interpretation ("How sure are you of the icon's function?"). 35 In the label-only condition, participants were asked; • To describe the function associated with the label; • How familiar they were with the label ("Have you seen and used this exact label before, say on a computer, mobile device, etc.?"); and, • How confident they were about their interpretation ("How sure are you of the label's function?"). We also conducted a brief interview on participants' education level and occupational status (see Appendix A.4) and administered the Snellen and FAS tests. Study sessions took on average approximately one and a half hours for younger participants and two and a half hours for older participants. 3.3.5 Data analysis and treatment Scoring Participants' accuracy in identifying icon objects and in interpreting icon meanings were scored during the study session by the experimenter, and then separately by an independent rater using the session video recordings. The agreement percentages for the two sets of icon object identification and icon interpretation scores were 88% and 92%, respectively, corresponding to Cohen's kappa scores of .47 (p<.001) and .81 (p<.001). All scores on which the two raters disagreed were discussed and resolved by consensus. To assess whether the icon object(s) were correctly identified and whether the icon meaning was correctly interpreted, the raters focused more on the expressed concepts and ideas, rather than on whether the right technical words or phrases were used. For example, for the icon used to exit the camera program, responding with either "to get out of program" or "to close program" would be scored as correct. For icons showing abstract objects, participants could identify the objects by naming the shapes (e.g., dots and arrows) and graphical features (e.g., lines). Statistical tests We used analysis of variance statistical tests (ANOVAs) to test our hypotheses. Whenever a statistically significant interaction was found, we followed up with post-hoc pairwise comparisons, using Bonferroni correction to protect against Type I error. In addition, Greenhouse-Geisser corrections were used when sphericity was an issue. Using this correction can result in degrees of freedom that are not whole numbers. We also report the partial eta-squared (ηp2) statistic, a measure of effect size, which is often more informative than statistical significance in applied human–computer interaction research (Landauer, 1997). To interpret this statistic, .01, .06 and .14 are considered small, medium and large effect sizes, respectively (Cohen, 1988). 36 Controlling for confidence As expected, participants' confidence was significantly higher when they correctly interpreted an icon's meaning (M=3.14, SD=0.7) than when they gave an incorrect interpretation (M=2.05, SD=0.9) (paired t285=16.8, p<.001). When reporting interpretation confidence scores, we only used the scores for icons whose meanings were correctly interpreted because we were less interested in participants' confidence when they incorrectly interpreted an icon. Some participants did not correctly interpret any icons for a particular experimental condition and thus did not have confidence scores for each experimental condition; this is reflected in the lower degrees of freedom in the related ANOVA results. Controlling for familiarity We asked participants to report their familiarity with the icons/labels in order to assess the influence of familiarity on icon interpretability scores. An ANOVA of the familiarity scores showed that our younger participants gave significantly higher familiarity scores (M=4.8 out of 10, SD=3.2) than our older participants (M=2.6 out of 10, SD=2.6) (F1,34=21.5, p<.001, ηp2=.39). Because of this difference, we examined the influence of familiarity on (i.e. correlation with) each of our primary dependent interpretability measures. We found that familiarity was significantly correlated with each of our three interpretability measures, but the amount of variance that could be attributed to familiarity (r2) was relatively small, 16% or less, on the three dependent measures (see Table 3.5). Nevertheless, because familiarity had a significant effect on our interpretability scores, we sought to control for its effect by treating it as a covariate. Because familiarity was different across the different conditions, we were not able to factor out the influence of familiarity by means of a traditional analysis of co-variance (ANCOVA). Instead, we removed the influence of familiarity from each variable by means of a regression analysis, saved the residuals and examined those by means of an ANOVA (see Appendix A.6 for more details). We found that even with familiarity controlled, almost all the significant effects of the other independent variables remained (see Appendix A.7 for ANOVA tables for the original scores and of the residual scores.). Thus we present the ANOVA results based on our unadjusted interpretability scores, and note when significant effects were different in the analyses where the scores were controlled for familiarity. r r2 p N Identification .27 .07 <.001 288 Interpretation .40 .16 <.001 432 Confidence .35 .12 <.001 398 Table 3.5: Correlation between perceived familiarity and icon interpretability measures, r (Pearson correlation) and r2. 37 3.4 Results We present our findings here, focusing on the results that support or did not support our four hypotheses for this experiment. 3.4.1 Older participants generally had more difficulty interpreting existing icons As predicted by H1, compared to our younger participants our older participants were significantly less accurate in identifying icon objects and interpreting icon meanings. Specifically, younger participants correctly identified the objects in 96% of the icons (SD=10%) while older participants correctly identified the objects in only 86% of the icons (SD=19%; significant main effect of age, F1,34=29.4, p<.001, ηp2=.46). Younger participants also correctly interpreted 75% of the icons (SD=28%) while older participants correctly interpreted only 60% of the icons (SD=32%; significant main effect of age, F1,34=27.1, p<.001, ηp2=.44). Both older and younger participants felt quite confident in their interpretation when the interpretation was correct (older: M=2.95 out of 4, SD=0.68; younger: M=3.14 out of 4, SD=0.66). Interpretation confidence scores did not differ significantly between the two age groups (F1,11=0.9, p=.36, ηp2=.08). 3.4.2 Concreteness helped older participants identify objects in semantically-far icons, but did not help interpretation We found significant age-related effects of concreteness and semantic distance on icon identification accuracy. Specifically, a significant three-way interaction of age, concreteness and semantic distance was found (F1,34=8.44, p=.006, ηp2=.20). As shown in the two charts in Figure 3.8, older participants performed worse on abstract-far icons than on concrete-far icons (p<.001), while concreteness did not affect younger participants performance. This age-related difference in icon identification accuracy supports H2. younger participants 0% 20% 40% 60% 80% 100% concrete abstract Concreteness Id en tif ic at io n ac cu ra cy older participants 0% 20% 40% 60% 80% 100% concrete abstract Concreteness Id en tif ic at io n ac cu ra cy close far Figure 3.8: Mean icon object identification accuracy scores by age group, concreteness and semantic distance (N=36). Error bars represent 95% confidence intervals. Older participants had more difficulty than did younger participants in identifying objects in abstract-far icons. 38 We did not find an age-related effect of concreteness on icon interpretation accuracy (F1,34=0.53, p=.47, ηp2=.02), as shown in Figure 3.9, which is counter to H2. However, we did find that concreteness had a significant effect on interpreting unlabelled icons regardless of age (significant two-way interaction between concreteness and label, F1684=3.3, p=.04, ηp2=.09). Both age groups interpreted significantly more unlabelled abstract icons (M=49%, SD=33%) than unlabelled concrete icons (M=40%, SD=30%, p=.002). We did not expect abstract icons to be easier to interpret than concrete ones. We discuss this further in Section 3.5.2. 0% 20% 40% 60% 80% 100% concrete abstract Concreteness In te rp re ta tio n ac cu ra cy 20-39 65+ Figure 3.9: Mean icon interpretation accuracy scores by age group and concreteness (N=36). Error bars represent 95% confidence intervals. When analyzing the interpretation confidence scores, a significant four-way interaction of age, label, semantic distance and concreteness was found (F2,22=3.5, p=.047, ηp2=.24). To facilitate interpretation of this interaction, we carried out follow-up three- and two-way ANOVAs, but did not find any significant age-related effects of concreteness, semantic distance, or label on interpretation confidence. Concreteness did not affect interpretation confidence for any age group. 3.4.3 Semantically close icons were easier for our older participants to interpret As stated in the previous section, we found significant age-related effects of concreteness and semantic distance on icon identification accuracy. Further analysis of the three-way interaction of age, concreteness and semantic distance showed that older participants performed worse on abstract-far icons than on abstract-close icons (p<.001), as shown in Figure 3.8. In contrast, semantic distance did not affect younger participants' performance. This age-related difference in icon identification accuracy supports hypothesis H3. We also found a significant age-related effect of semantic distance on icon interpretation (significant two-way interaction between age and semantic distance, F1,34=11.9, p=.002, ηp2=.26), as hypothesized in H3. Both age groups performed significantly worse on semantically far than on semantically close icons (p<.001 for both groups), as shown in Figure 3.10. However, the 39 difference in accuracy scores was significantly greater for older participants than for younger ones (outside the 95% confidence interval for difference in scores), suggesting that older adults have much more difficulty interpreting semantically far icons. We did not find any significant age-related effects of semantic distance on interpretation confidence, as stated in the previous section, which is counter to H3. However, semantic distance did have a significant effect on interpretation confidence for all participants (F1,11=70.4, p<.001, ηp 2=.87). Participants were significantly more confident in their interpretation of semantically close icons (M=3.3 out of 4, SD=0.5), than their interpretation of semantically far icons (M=2.7 out of 4, SD=0.8). 0% 20% 40% 60% 80% 100% close far Semantic Distance In te rp re ta tio n ac cu ra cy 20-39 65+ Figure 3.10: Mean icon interpretation accuracy scores by age group and semantic distance (N=36). Error bars represent 95% confidence intervals. 3.4.4 Labels help both age groups to interpret icons No significant age-related effects of label on icon object identification were found, offering no support for H4. However, both age groups correctly identified more objects in labelled icons (M=94%, SD=13%) than in unlabelled icons (M=88%, SD=18%; significant main effect of label, F1,34=15.4, p<.001, ηp2=.31). We also did not find a significant age-related effect of label on icon interpretation, offering no support for H4. A significant three-way interaction of age, label, and semantic distance was found on icon interpretation scores (F2,68=3.9, p=.025, ηp2=.10), but this interaction was not significant when familiarity was controlled (F2,68=1.1, p=.36, ηp2=.03). This is our only reported result where a significant effect was no longer significant after we controlled for familiarity. Although no age- related effect was found, both age groups correctly interpreted significantly more labelled icons (M=78%, SD=23%) than unlabelled icons (M=45%, SD=32%; significant main effect of label, F2,68=192.9, p<.001, ηp2=.85). Further, we did not find any significant age-related effects of label on interpretation confidence, as previously stated, which also offers no support for H4. However, both age groups 40 felt more confident in their interpretation of labelled icons (M=3.3 out of 4, SD=0.6) compared to unlabelled ones (M=2.6, SD=0.8; significant main effect of label, F2,34=26.1, p<.001, ηp2=.70). 3.4.5 Summary of results Our laboratory study revealed the following critical findings concerning our hypotheses. We consider a hypothesis to be supported if age-related effects were found on icon object identification accuracy and interpretation accuracy, our primary dependent variables. H1. Overall interpretability: supported. Older participants generally had more difficulty than younger participants identifying objects in icons. Older participants also had more difficulties correctly interpreting icon meanings. However, compared to younger participants, older participants felt just as confident in their interpretations when they interpreted icons correctly. H2. Concreteness: partially supported. Older participants were better able to identify objects in concrete-far icons than in abstract-far icons, whereas differences in concreteness did not affect younger participants' object identification scores. We did not, however, find any age-related effect of concreteness on interpretation accuracy. Unexpectedly, all participants were able to correctly interpret more unlabelled abstract icons than unlabelled concrete ones. Further, concreteness did not affect interpretation confidence for either age group. H3. Semantic distance: supported. Older participants were better able to identify objects in abstract-close icons than in abstract-far icons, whereas differences in semantic distance did not affect younger participants' object identification scores. Both age groups were better able to interpret semantically close icons than semantically far icons, but this difference in interpretation accuracy was larger for older participants than for younger ones. While semantic distance did have an effect on interpretation confidence for both age groups, no age-related effect was found. H4. Label: not supported. Labels helped both older and younger participants to better identify the abstract icons, to better interpret icon meanings, and to be more confident in their interpretations. However, there was no age-related effect of label on any of these three measures. 3.5 Discussion In this section, we discuss our key findings and how they might help us understand older adults' difficulties with using existing mobile devices. We also discuss our findings in light of past work that reported different findings. We then discuss implications for designing icons for older adults and propose design guidelines. 41 3.5.1 Existing icons harder for older adults to use We found empirical evidence that older adults do have significantly more difficulty than younger adults with the initial interpretability of a variety of existing mobile device icons. Having similar mobile device experience and at least some computer experience, our older participants could only correctly interpret 38% of our unlabelled icons and 71% of our labelled ones, whereas our younger participants did substantially better, interpreting 52% and 86%, respectively. Although both age groups were found to have trouble with similar types of icons, our results suggest that older adults may get stuck more often because they are not able to interpret many icons in an existing mobile device interface, whereas a younger person may be able to "get by." Difficulties with using icons, which leads to difficulties using the entire interface, may partly explain why older adults find mobile devices difficult to use and why they have been relatively slower to adopt mobile devices. This suggests that there is a strong need to redesign some existing icons in order to make mobile device interfaces easier for older adults to use, especially for devices that can help improve their quality of life. We found that abstract icons, semantically far icons and unlabelled icons were especially difficult for our older participants to use. With this as motivation, we next discuss our findings and their implications for designing better icons for older adults. 3.5.2 Icon object concreteness had little effect on icon interpretation We found evidence that icons depicting concrete objects help older adults to identify more icon objects in semantically far icons, but this did not have an age-related effect on icon interpretation. In fact, when looking at the two age groups together, participants correctly interpreted significantly more abstract icons than concrete icons (as shown in Figure 3.9). Given the strong effect of concreteness on increasing initial icon interpretability reported in McDougall et al. (2000), we were interested in understanding why our concrete icons were not always easier to interpret than our abstract ones. We believe that this discrepancy may be due in part to the difference between the way icons were rated on concreteness in the study by McDougall et al. (2000) and in the study we present in this chapter. According to our definition of concreteness, concrete icons included representations of real-world objects, places or people, but icons with symbols, even those that had precise well- established meanings (e.g., math symbols and musical notes), were classified as abstract icons. The icons used in the study by McDougall et al. (2000) also depicted commonly used symbols, but they were often rated as being more concrete than abstract. As a result, familiarity with that study's set of icons (unlabelled) was reported to be highly correlated with concreteness (r=.78) (McDougall et al., 1999), whereas familiarity with our set of icons (unlabelled) was found to be much less correlated with concreteness (r=.15). This shows that the way one defines icon concreteness affects how concreteness relates to familiarity and its effect on icon interpretability. 42 More generally, this shows that defining icon characteristics is non-trivial and that slight variations in definitions can result in large differences in study outcomes. Further research aimed at achieving standardised definitions for icon characteristics is required to address this issue. 3.5.3 Semantically close icons are much easier for older adults to use We found that semantically far icons were generally much harder for our older participants to use than semantically close icons. As presented earlier, our older participants correctly interpreted 80% of our semantically close icons, compared with only 41% of our semantically far icons. Further, our older participants had significantly more difficulty than did our younger participants in interpreting semantically far icons. On the basis of our findings, semantic distance appears to have a larger effect than concreteness on initial icon interpretability. Why would older adults have more difficulty with semantically far icons (and icons in general)? We argue that their difficulties may be related to older adults' difficulty in forming and using mental models, which has been reported in the literature (Freudenthal, 1998; Ziefle & Bay, 2004). Users often rely on their understanding of the system and how it operates (i.e. their mental model) to help "cross" the semantic distance from the icon object to its function. In other words, icons, especially semantically far icons, generally require an accurate mental model of the system that the user can apply in interpreting the icon. For example, one needs to know that a device can perform calculations in order to know how to interpret a calculator icon. One also needs to know that a device can and is sometimes used to compress files, in order to correctly interpret the "Zip" icon in Figure 3.5 showing a clamp. Although we spent time during the experiment reviewing common mobile device capabilities and chose participants with similar levels of computer and mobile device experience, our older participants may have had more difficulty remembering many of the new functions that we introduced to them, or had difficulty applying their mental model of the device to interpreting the icons. A comment by one of our older participants that one needed "umpteen degrees to keep up with technology" suggests that he did have difficulty developing accurate mental models of new technology. 3.5.4 Guidelines for reducing semantic distance We have found evidence that semantically far icons are particularly difficult for older adults to learn to use, and that they should be redesigned. One method to reduce semantic distance is to choose icon objects with semantically close associations to the icon function. We describe here other ways for reducing semantic distance. Use familiar metaphors We observed in this research that familiarity with an icon often plays a large part in being able to use it. Existing icon design guidelines suggest using images and metaphors that are familiar to the target user. However, an icon's familiarity depends on an individual's experience. Commonly 43 used computer metaphors (e.g., disk for saving and wrench for device options) may not always be known to older adults, who generally have less experience with computers. Instead, when designing icons for older users we suggest using everyday metaphors with which they are familiar. If it is not feasible to only use familiar metaphors, one should ensure that the metaphors used in the interface are taught to the user, perhaps through documentation (reference card) or by someone (e.g., caregiver or customer support). One of our older participants commented that she was very interested in learning the common metaphors used in computer icons. Some may argue that older adults' lack of familiarity with commonly-used computer metaphors will no longer be an issue in the future when the upcoming generations, who generally have relatively more computer experience, become older adults themselves. Hawthorn (2000) counters that there are still many younger adults who have jobs or home situations where they do not interact much with computers. In addition, he points out that even if older working adults have opportunities to keep up to date with new computer technology, they may be less able to adjust to the changes. Further, once retired, older adults may have less need to keep up to date with new technology. Computer metaphors will continue to evolve and we therefore expect future generations of older adults to continue to have more difficulty than younger adults staying current with these metaphors. Label icons We found that labels greatly help both younger and older adults to initially use icons. Icon object identification and interpretation performance in the icon-only condition was generally worse than the two label conditions (i.e. icon+label and label-only), whereas performance in the two label conditions was similar, which is consistent with Wiedenbeck's findings (1999). Although we did not find that labels helped our older participants significantly more than our younger ones to interpret icons, we expect that labels provide greater benefits to the general older adult population (which likely has on average less computer experience than the older participants in our study). In fact, three of our older participants commented that they usually interpreted the label before the icon. On the basis of our findings, we suggest that all mobile device icons should be labelled for older users, at least initially. It would be interesting to explore whether older adults continue to rely on labels after continued use of the icons with labels. Although the presence of labels greatly improves icon interpretability, it may not always be feasible to label all mobile device icons because of the device's limited screen real-estate. However, there are a number of alternative labelling techniques that may be suitable. For example, some mobile interfaces show only one label at a time (usually for the selected icon) in a designated area in the interface, or they have "tool tip" labels that pop up above the highlighted/selected icon. Past research has found that older adults have poorer visual spatial abilities (Czaja, 1997), and thus research is needed to see which labelling technique works best with this population. 44 Allow user to select icons Our results highlight the fact that an icon that is easy for one person to use may be difficult for another. We propose that a mobile interface provide not just one icon for the different functions in the interface, but allow users to choose, from a set of icons for each function, an icon that they each feel is most suitable. A mobile device interface could provide, for example, a variety of icons associated with the "device options" function portraying different objects such as a wrench, control knobs, form radio buttons or a graphic of the device itself. The semantic distance between an icon and its function can be different for each individual because it depends in part on the user's familiarity with the objects depicted and the metaphor used in the icon. A feature to allow users to select icons is more suitable for a mobile device interface, compared with a desktop computer interface, because only a relatively small set of icons would need to be selected. The process of choosing suitable icons for an application could be supported by a software wizard to minimise the time and effort required from the user. A wizard can provide clear and more detailed instructions on how to change UI settings, which Hanson and Crayne (2005) found to be important for supporting older users. The selection of icons could also be performed by a loved one or caregiver. A number of existing commercial desktop and mobile assistive technologies for people with communication disorders, many who are older, currently help users to personalise the interface's icons (e.g., Pocket Communicator [Gus Communications, 2009], Lingraphica [Lingraphicare, 2009]). We expect that giving older adults the option to choose their own icons would improve both initial icon interpretability and usability over time, and that younger users might benefit from this feature as well. 3.5.5 Limitations The presence of some ceiling effects (i.e., correctly identifying or interpreting 100% of the icons) in younger participants' results may have reduced our experiment's power to find some age- related effects (e.g., Section 3.4.2). Our younger participants had perfect identification/ interpretation scores 6% of the time while our older participants never scored perfectly. However, these ceiling effects do not negatively affect the validity of the significant effects that were found. In our experiment we showed participants enlarged icons and text labels on paper. This might have affected the ecological validity of the results. Enlarging the icons and text, and presenting them on paper allowed us to ensure that they were as readable as possible and to reduce the influence of individual and age-group differences in eyesight, which we felt was an important first step in understanding icon characteristics. Future work will need to look at the extent to which our findings change, if at all, when different icon and label font sizes are considered and when icons are shown on a mobile device. When incorporating icons into a mobile interface, the designer should consider the results of past research on effect of icon and text size on older users (e.g., Siek, Rogers, & Connelly, 2005). 45 3.6 Summary In this study, we investigated age-related effects of three icon characteristics on icon interpretability, with the overall goal of learning how to better design icons for older adults and developing icon design guidelines that take into account a user's age. Through a qualitative exploratory study, we observed older adults having more difficulty using unfamiliar mobile device icons, and identified, based on their comments, some icon characteristics that helped or hindered their icon interpretability. A follow-up experiment was conducted to better understand the effects of those characteristics, specifically of concreteness, semantic distance and the presence of labels. We found that these characteristics did affect initial icon interpretability, but some more than others. From the results of the experiment, we determined that older adults were able to interpret significantly fewer existing mobile device icons than younger adults, particularly semantically far icons. We also found that older adults found it more difficult to identify objects in abstract and semantically far icons. Further, we found that unlabelled icons were more difficult for both age groups to use. We concluded that many icons need to be redesigned in order for older adults to be able to use the icons and their interfaces. We presented a number of reasons why good icon and interface usability is particularly important on mobile devices used by this population. We suggested, based on our results, that icons incorporate concrete objects or commonly used symbols. More importantly, we suggested reducing semantic distance by choosing icon objects semantically close to the icon function, using familiar metaphors, using labels and allowing users to choose an icon from a set of potentially suitable icons. Although our empirical results are consistent with many existing icon design guidelines, the results highlight related guidelines that, when followed, should improve an older user's initial interpretability of icons on mobile devices and other computer interfaces. Existing and future mobile devices offer older adults many opportunities to remain active and increase their independence. By making mobile device icons easier for older adults to use, we expect that the overall device will be more usable and will have a better chance of being adopted. 46 Chapter 4 4. Incorporating a Multi-Layered Interface: An Experiment In this chapter, we present our investigation into a second design approach: incorporating a multi- layered interface design in a mobile device user interface. After specifying our investigation objectives and describing our research methodology, we present a controlled experiment that we conducted to compare a multi-layered (ML) mobile address book application to a non-layered full- functionality application across four stages of learning. We describe the many design decisions we made in prototyping a ML mobile application for our experiment. After describing the experiment methods, we present the results, starting with the performance measures, followed by perceived learning experience and preference. We then discuss the implications of our findings and suggest situations in which ML interfaces for mobile applications could provide the most benefit. 4.1 Objectives and methodology The goal of this investigation was to assess older and younger adults' performance using a ML mobile application across four learning phases: Basic Task Acquisition on an initial reduced- functionality layer, Retention of acquired skills (participant performs basic tasks again after a break), Transition to full interface (participant performs basic tasks, but on a second layer, which is a more complex full-functionality layer), and Advanced Task Acquisition on this second layer. Catrambone and Carroll (1986) studied a similar series of phases for using a ML desktop application, but their study did not include a retention phase. As noted in Chapter 2, previous research has already shown that ML interfaces can be beneficial for younger adults. We were particularly interested in studying whether those benefits would be the same or different for older adults. Assessing performance across four phases allowed us to develop a more complete understanding of how ML interfaces affect learning, from initial usage to more experienced usage, and in the acquisition of tasks on different layers. 47 We tested the following four hypotheses, one related to each learning stage. The control is a non-layered full-functionality interface (see the Conditions subsection in Section 4.2.2 for more details). Mastery and extra steps are defined in the Measures subsection in Section 4.2.2. H1. Basic Task Acquisition. The ML interface's Reduced-Functionality layer (compared to the control's Full-Functionality layer) will better help users to master the basic task set, in terms of fewer extra steps, shorter task completion times, and fewer attempts to reach mastery. H2. Retention. The ML interface's Reduced-Functionality layer (compared to the control's Full-Functionality layer) will help users to better perform the basic task set mastered 30 minutes previously, in terms of fewer extra steps and shorter task completion times. H3. Transition. Transitioning from the Reduced-Functionality layer to the Full-Functionality layer will negatively affect basic task set performance (compared to using the control interface and remaining in the Full-Functionality layer) in terms of more extra steps and longer task completion times. H4. Advanced Task Acquisition. Learning on the ML interface's Reduced-Functionality layer (compared to the control's Full-Functionality layer) will better help users to master the advanced task set on the Full-Functionality layer in terms of fewer extra steps, shorter task completion times, and fewer attempts to reach mastery. We also tested a fifth hypothesis that focused on the age-related effects of learning on our ML mobile application. Testing this hypothesis was very important to understanding whether our ML interface offered unique learnability benefits to older adults. H5. Benefit to Older Adults. The performance benefits provided by the ML interface will be greater for older adults than for younger ones in the Basic Task Acquisition, Retention and Advanced Task Acquisition phases. 4.2 Methods In this section, we start by describing our participants. We then present the ML mobile device application used in the experiment, beginning with the choice of mobile application (an address book) and its functions, followed by the design of the two interface layers. Afterward, we describe the tasks that participants were asked to perform and the overall experimental procedure. 4.2.1 Participants We recruited 16 older adults (ages 65-81) and 16 younger adults (ages 21-36); there were 32 participants in total. Participants were recruited through posters at UBC, at libraries and at senior centers, as well as via postings on an online classifieds site (http://vancouver.en.craigslist.ca/). Participants were pre-screened over the phone to ensure that they had limited computer experience, no or very little experience with handheld computers, and no experience with smart 48 phone capabilities beyond making phone calls. Participants were also pre-screened to be free of visual and physical impairments that might prevent them from operating a mobile device. Visual acuity was tested during the study using the Snellen pocket eye chart; all participants were found to have normal or corrected-to-normal eyesight. Four older participants, two assigned to the ML interface condition and two assigned to the control group, could not finish the study within the allotted time and had to be replaced (we discuss the details and implications of this in Section 4.4.5). Participants received an honorarium for participating. Participants from both age groups were assigned to one of two interface groups: ML or control interface (described in Conditions subsection in Section 4.2.2). Table 4.1 gives an overview of the descriptive data for age, gender, education, and cognitive abilities for all four experimental groups. A Mann-Whitney U test revealed no significant differences with respect to age and number of years education between the two older adult interface groups (age: U=30.5, p=.88; education: U=21, p=.24) nor between the two younger adult interface groups (age: U=29, p=.75; education: U=28.5, p=.70). However, between the younger and older groups, Mann- Whitney U tests revealed significant differences with respect to years of education (U=50.5, p=.003; the younger participants had more years of education) and years of mobile experience (U=39.5, p<.001; the younger participants had more years of mobile experience), but no difference with respect to years of computer experience (U=99.5, p=.22). Because we did not pre-screen participants on cognitive abilities, all participants completed a test battery so that we could characterize the groups and check for additional interface-group differences, especially with respect to sensory, perceptual and cognitive factors that are known to change with age and thus might affect performance. We assessed participants' verbal working memory using the Reverse Digit Span Test (RDST) (Wechsler, 1981), their visual-spatial working memory using the Corsi Block test (Milner, 1971), their perceptual and motor speed using the Digit Symbol test (Wechsler, 1981), and their verbal abilities using the North American Adult Reading Test (NAART) (Blair & Spreen, 1989). We also assessed participants' attitudes towards computers and the Internet with the Technology Profile Inventory (TPI) (DeYoung & Spence, 2004). A multivariate analysis of variance (MANOVA) revealed no significant difference between the interface groups regarding these measures for older participants (F4,11=0.93, p=.48) as well as for younger participants (F4,11=0.51, p=.73). However, a MANOVA did reveal a significant difference between the two age groups in regards to measured cognitive abilities (F4,25=25, p<.001). Follow-up analysis of variance statistical tests (ANOVAs) revealed that younger participants significantly outperformed older participants in tests that measured visual-spatial working memory (F1,28=36.1, p<.001) and perceptual and motor speed (F1,28=79.3, p<.001), as might be expected. These follow-up ANOVAs also revealed a significant interaction between age group and interface on NAART verbal ability scores (F1,28=10.3, p=.003) where older participants in the ML interface condition outperformed those in the control condition, and conversely the 49 younger participants in the control condition outperformed those in the ML condition. Older participants also had significantly higher verbal ability scores than younger participants did (significant main effect of age, F1,28=9.1, p=.005), as expected, but no significant differences were between interface groups on these scores were found. This unexpected interaction may need to be considered when interpreting the results. ML interface M (SD) Control interface M (SD) Younger participants N 8 8 Age 24.4 (5.1) 22.4 (1.1) Gender 3 male, 5 female 4 male, 4 female # years education 15.9 (1.5) 16.3 (1.2) Verbal working memory 6.3 (1.9) 7.6 (2.3) Visuo-spatial working memory 7.3 (1.8) 7.6 (1.3) Perceptual and motor speed 74.1 (8.2) 77.5 (11.2) Attitudes towards computer technology 112 (13.0) 116 (13.5) Verbal abilities 31.1 (11.2) 42.5 (9.8) Older participants N 8 8 Age 71.6 (5.5) 70.8 (4.2) Gender 8 female 4 male, 4 female # years education 11.8 (3.7) 14.8 (3.4) Verbal working memory 7.0 (2.6) 5.8 (1.2) Visuo-spatial working memory 5.0 (1.2) 5.6 (1.7) Perceptual and motor speed 50.0 (6.4) 47.6 (7.8) Attitudes towards computer technology 110 (15.5) 110 (17.2) Verbal abilities 51.9 (8.0) 41.9 (8.4) M=mean, SD=standard deviation RDST, verbal working memory: higher score=better memory, max. score: 12 Corsi Block test, visuo-spatial working memory: higher score=better memory, max score: 12 Digit Symbol test, perceptual and motor speed: higher score=faster TPI, attitudes towards computers: higher score=more positive attitudes, max score: 150 NAART, verbal abilities: higher score=better abilities, max score: 61 Table 4.1: Participant characteristics (N=32). 4.2.2 Materials In this section we describe the design of the ML mobile application, the tasks participants were asked to perform, and the measures we recorded during the experiment sessions. Conditions One interface version for each of the two experimental conditions : To explore the effects of learning on a ML interface, we compared it to a traditional, non-layered interface, which we used as our experimental control condition. The ML interface had two layers: an initial reduced- functionality layer that presented only a small subset of functions and a full-functionality layer that included all functions available in the application. 50 We had the following two interface conditions. 1. ML interface: participants learned the basic task set (described later in Task sets subsection) on a reduced-functionality layer and then learned the advanced task set on a full-functionality layer. 2. Control (non-layered) interface: participants learned both basic and advanced task sets only on a full-functionality layer. Choice of application The experimental application was based on existing commercial mobile address books. The application consisted of two main screens: a contact list screen ( : We chose a mobile phone address book (also known as a phone book, contact list, or contacts application) as the experimental application for this study because both the older and younger participants were likely to be familiar with this application domain, having used a paper or electronic address book. We made this choice because older adult mobile phone users have previously expressed interest in learning to use the address books on their phones but have often had difficulty with functions such as adding a new contact (Ofcom, 2006). Figure 4.1, top two rows) and a contact details screen (Figure 4.1, bottom two rows). The contact list screen allowed users to page through their contacts, with 6 names presented at one time. Selecting a name from the list led to a screen of contact details for that individual; users returned to the contact list by pressing the "Back to list" right soft key on the contact details screen. The address book was pre-populated with 24 contacts, all actors and actresses whose names were expected to be relatively familiar to our participants. The two application screens each provided an "Options menu" (referred to hereafter simply as menu) to execute a variety of task functions. In our control interface, these two menus offered a total of 24 functions, which were chosen based on a survey of common features in 9 existing mobile address book applications (from Nokia, Motorola, Sony Ericsson, Sanyo, Blackberry, and Apple phones). Functions for editing an individual contact's information (e.g., edit, add custom ringtone) were placed in the Contact Details menu, while functions for using the contact information to perform a task using a contact's information (e.g., call, send text message) were placed in the Contact List menu. Other miscellaneous functions (e.g., help, settings) were also placed in the Contact List menu. The menus, which were located in the bottom left corner of the screen, were accessed using the device's left soft key. All functions related to the experimental tasks (see Task sets subsection) were implemented. The other functions were either implemented or displayed a "function not simulated" message when the user tried to execute them. The contents of the two menus are shown in Table 4.2. 51 Contact List screen Contact List screen Contact Details screen Contact Details screen Figure 4.1: Example screenshots from our two interface layers: Reduced-Functionality layer (left column) and Full-Functionality layer (right column). The top two rows show the Contact List screens and the bottom two rows show the Contact Details screens. The second and fourth rows show screens with their Option Menus open. The "v" symbol in a list indicates that a subsequent page exists. Reduced-Functionality Layer Full-Functionality Layer 52 Multi-layer design Figure 4.1 : We created two distinct interface layers that primarily differed in the number of available functions (see ). Following existing ML design conventions, the Reduced- Functionality layer, which a user would first learn on, only contained relatively basic functions, whereas the Full-Functionality layer contained more advanced functions in addition to the basic functions. Functions were classified as being basic or advanced based on an informal survey of eight mobile phone users (four users between ages 20-39 and four users of age 50+). Basic functions were ones that most of our surveyed mobile phone users reported learning first and identified as necessary for using the application. In contrast, advanced functions were seen by those surveyed as secondary functions that a user would learn to use after the basic functions. The Reduced- Functionality layer contained five functions (including all basic ones), three in the Contact List and two in the Contact Details menus. Functions that were disabled in this layer were hidden, rather than blocked, because this approached seemed most suitable for our mobile application (see Section 2.3.3). The Full-Functionality layer contained 24 functions, split evenly across the two menus. Because a maximum of six functions could be shown at once, the menus in the Full- Functionality layer each contained two pages of functions. Table 4.2 shows the lists of functions for each layer and the second row of images in Figure 4.1 show the Contact List screen menu for both layers. List of functions in the two menus Reduced-Functionality Layer Full-Functionality Layer Contact List screen Call View Contact New Contact1 (page 1) Call Use Number Send Message2 Send Contact Mark/Unmark View Contact (page 2) New Contact1 SIM Phone Book Synchronization Help Settings General Info Contact Details screen Edit Contact1 Delete Contact1 (page 1) Duplicate Contact Edit Contact1 Set Voice Dial2 Add Picture Set Speed Dial Categories (page 2) Add Custom Ringtone2 Set as Default Copy to SIM Copy from SIM PTT Options Delete Contact1 1 Used in basic task set. 2 Used in advanced task set. Table 4.2: Options menu functions for the Reduced- and Full-Functionality layers. Related functions were grouped together (e.g., Duplicate Contact was positioned close to Edit Contact). This intermixed basic and advanced functions in the Full-Functionality interface and is representative of many mobile and desktop computer interface menus. As a result of this design decision, using the Full-Functionality layer sometimes required the user to go to the second page of a menu to find a basic task function. The main drawback of this approach is that the menu 53 positions of the basic functions were not consistent across layers. An alternative approach would be to group functions by layer, with basic functions at the top of the menu followed by advanced function below. By grouping functions by layers, basic functions would remain in the same location regardless of the interface layer, which would likely help users transition between layers. However, grouping functions by layer often separates related functions, removing contextual information that helps users to interpret concise function names. For example, the function Use Number shown in Table 4.2 is closely related to Call, which helps the user correctly interpret "number" as phone number. However, Use Number may be harder to correctly interpret if placed between non-related functions such as New Contact and Send Message, which makes it less clear what "number" means. We chose the grouping-by-related-functions approach over the grouping- by-layer approach because it is representative of many user interface menus and has been used in a number of other research studies on multi-layered interfaces (Findlater & McGrenere, 2007; McGrenere, Baecker, & Booth, 2002; Shneiderman, 2003). The only operational difference in performing the tasks on either layer was the number of steps required to navigate through the options menu to find a particular function. We define a step to be one device button press performed by the user. Other aspects of the tasks, such as scrolling through the contact list and entering text, required the same number of steps for both interface layers. The Full-Functionality layer's contact detail screen also had more visual complexity in the form of additional contact details (see the two screens in Figure 4.1, third row). Thus, any differences due to the interface conditions could be related to the number of menu items, switching between menu pages, or visual complexity. Our experimental design and evaluation prototype did not allow us to separately analyze the effect of each of these three ML design characteristics, but rather allowed us to evaluate the learnability of a multi-layered interface for a mobile application as a whole. Equipment The experiment equipment included a Nokia E61i device, shown in Figure 4.2. The E61i was chosen for its relatively large screen size (resolution: 320x240 pixels, width: 5.7cm, height: 4.3cm) and QWERTY keyboard (button size: 0.5cm x 0.6cm); the device's overall dimensions were 7.0cm (width) x 11.7cm (height) x 1.4cm (depth). The E61i can run Flash Lite applications (unlike many other devices with a similar form factor), which allowed us to quickly develop the address book prototype in Flash Lite 2.1. In our pilot studies, no participants reported difficulties reading the mobile application text off the device. A number of older participants did report some difficulty pressing the buttons but were comfortable using the eraser end of a pencil, which we provided during our study, to press each individual button. Four of the 16 older participants chose to use the pencil during the study, which appeared to help them to avoid pressing the buttons surrounding the target button and to increase typing accuracy. No differences in input speed were observed between those who did versus those who did not use an eraser. 54 Figure 4.2: An older adult holding the mobile device (Nokia E61i) used in this study. Tasks Participants were asked to perform sets of basic and advanced tasks throughout the experiment. The basic set consisted of three tasks: adding a new contact into the address book, editing the information of a previously entered contact, and deleting a contact from the address book. The advanced set also consisted of three tasks: adding voice dialing to a contact (to phone the contact by speaking their name), sending a text message to a contact, and adding a custom ringtone to a contact (so the phone would produce a distinctive ringtone whenever that contact called). Participants repeatedly performed the same set of three tasks for a particular phase, but task- related information (e.g., contact name, phone number, text message) varied from attempt to attempt. Example wordings of the tasks given in the experiment session were "Add Meryl Streep's name and her home phone number 267-946-9907," and "Change Kevin Spacey's cell phone number to 468-243-2301." The minimum number of steps required to complete each task was determined to provide a reference point for evaluating mastery, and for calculating the number of extra steps (see Measures subsection). The average number of required steps for performing basic task sets on the Reduced- Functionality and Full-Functionality layers was 75.2 steps and 90.2 steps, respectively. The average number of required steps for performing advanced task sets on the Full-Functionality layer was 87.0 steps (see Table 4.3 for the required number of steps broken down by step type). Depending on the experimental condition and progress through the four learning phases of the experiment, participants performed the tasks in either the Reduced- or Full-Functionality layer. Participants were not able to switch between layers on their own. Measures Quantitative performance measures to assess the learnability of the two interfaces included the total number of attempts before mastery, the number of steps (i.e., button presses) to perform a task, and task completion times. We also calculated the number of extra steps (i.e., errors) for a 55 task, which was the total number of steps taken by a participant to complete a task minus the minimum number of steps required to complete that task. For the purpose of this study, mastery was defined as being able to perform a task set twice in a row, requiring fewer extra steps than 20% of the minimum number of steps required for each task or task set. Based on our pilot study data, 20% additional steps (i.e., extra steps) seemed to be a reasonable mastery threshold that required users to perform a task without too many extra steps but would allow for some flexibility. To increase our confidence that a user had mastered the task set and was not simply lucky, we required the user to perform the task set under the mastery threshold twice in a row. Subjective quantitative measures, gathered by a questionnaire with six-point Likert scale items, focused on the perceived ease of learning, confidence in application use, perceived application complexity, and perceived workload. 4.2.3 Experimental design A 3-factor mixed design was used: 2 age groups (younger, older; between-subjects) by 2 interface conditions (ML, control; between-subjects) by required attempts (this value differed by phase as described next; within-subjects). For the Basic Task Acquisition and Advanced Task Acquisition phases, we only analyzed the initial three attempts. Participants were asked to perform at least three attempts. Although some participants mastered the given task set only after 10 or more attempts, others mastered the set within the three attempts and thus did not provide performance data for Attempts 4-10 for us to analyze. We analyzed the first two attempts performed in the Retention phase and the first four attempts performed in the Transition phases. An equal number of participants from each age group were assigned to each interface condition. We analyzed the subjective questionnaire data using a 2 (age group) x 2 (interface) x 2 (phase: Basic Task Acquisition vs. Advanced Task Acquisition) mixed design. 4.2.4 Procedure All study sessions took place in a usability lab on the university campus. Two experimenters conducted the study sessions, each experimenter running half the study sessions. Each participant took part in a study session individually. Device Tutorial. All participants started the study session by completing an interactive tutorial, given on the device, to learn how to enter text with the keyboard and to use the soft keys and direction pad for navigation. Participants were given as much time as desired to familiarize themselves with the device buttons and could repeat the tutorial exercises. Completing the tutorial required around 5 minutes for younger participants and around 10 minutes for older participants; no participants chose to repeat the tutorial. Participants were asked to use the device in any position they found comfortable (e.g., hold in hands, lay on table). 56 Basic Task Acquisition Phase. The set of basic tasks (i.e., add, edit, and delete a contact) was first described orally to participants (see Appendix B.4). Participants were then asked to "learn to perform these tasks with as few extra steps as possible" (speed was not mentioned). Participants carried out a series of attempts (3 minimum) until they had mastered the tasks (there was no cutoff on the number of attempts). Each attempt consisted of performing the set of three tasks, with new contact information provided for each attempt. Participants in the ML condition completed this phase in the Reduced-Functionality layer, while those in the control condition used the Full- Functionality layer. All participants performed the tasks in the same order. Tasks were given in written form on paper in large font (36-point Arial; see Appendix B.6). No further instructions on how to use the address book were given. Help was only offered when participants were stuck for more than two minutes. They filled out Questionnaire 1 (see Appendix B.8) after participants mastered the basic task set. 30-Minute Break from Interface. Participants filled out a background questionnaire (see Appendix B.7), completed the cognitive assessments listed in Section 4.2.1 and carried out a distractor task (assembling a jigsaw puzzle) for the remaining time. These tasks were intended to prevent rehearsal of what participants had just learned in order to permit an assessment of short- term retention. Retention Phase. Participants were asked to perform the basic tasks twice more (i.e., two attempts) on their assigned interface layer (Reduced-Functionality or Full-Functionality), this time "as accurately and quickly" as they could (see Appendix B.4). Advanced Task Acquisition and Transition Phases (attempts interleaved). All participants used the Full-Functionality layer in these two phases. The advanced set of tasks (voice dialing, text message, custom ringtone tasks) was described to participants (see Appendix B.4). Participants were then asked to "learn to perform these tasks with as few extra steps as possible." No further instructions on how to perform the advanced tasks were given, and help was offered only when participants were stuck for more than two minutes. Participants carried out a series of attempts until they achieved mastery or until they had performed a total of 10 attempts (unlike in the Basic Task Acquisition phase, which had no cutoff). Participants then filled out Questionnaire 2 (see Appendix B.9). To determine the effect on performance of transitioning from a reduced-functionality to a full- functionality layer, participants were asked to perform the basic task set four times (i.e., four attempts). We interleaved Transition phase attempts (of basic tasks) and the Advanced Task Acquisition phase attempts (of advanced tasks) (i.e., A(dvanced Task Acquisition) T(ransition) A T A T A T A A A . . . ). We used this task set order instead of having all Transition attempts before Advanced Task Acquisition attempts so that participants in the ML condition would have minimal familiarity with the Full-Functionality layer for both types of attempts. This simulates the 57 expected use of a ML interface, where users would transition to the more complex layer when they want to perform more advanced tasks, but would continue to use basic functions. Post-task Interview. We conducted a semi-structured interview with each participant. Study sessions could run to a maximum length of four hours, to allow participants ample time to complete the study but also to prevent participants and the experimenter from getting overly fatigued. Younger participants generally took two hours to complete the study, while older participants took three to four hours. 4.2.5 Data analysis and treatment We used ANOVAs to test our hypotheses. We report effects that were significant (p<.05) or that represent trends (.05<=p<.10). Whenever a statistically significant interaction was found, we followed up with post-hoc pairwise comparisons, using a Bonferroni correction to protect against Type I error. In addition, Greenhouse-Geisser corrections were used when sphericity was an issue; using this correction can result in degrees of freedom that are not whole numbers. We also report the partial eta-squared (ηp2) statistic, a measure of effect size; to interpret this statistic, .01, .06, and .14 are considered small, medium, and large effect sizes, respectively (Cohen, 1988). A small percentage of the participant interactions with the mobile application was not logged due to a technical issue, resulting in the loss of extra step and completion time data for 1.6% of tasks. This small percentage of missing data was spread broadly over participants in each of our experimental conditions. Fully discarding incomplete participant data (i.e., listwise deletion) would have resulted in a loss of 12% of performance data. Therefore, to make use of data from all participants, we imputed the missing data using scores from the attempt immediately preceding the affected attempt. Although it is known that this single imputation procedure can increase errors in significance testing, we chose this imputation procedure because it is simple to implement and more suitable in cases where only a very small portion of the data is missing (McKnight, McKnight, Sidani, & Figueredo, 2007; Scheffer, 2002). To validate our data treatment approach, we ran ANOVAs both with missing data imputed and again with participants' data fully discarded whenever data were missing, and found that our method for dealing with missing data did not alter the pattern of effects. We thus report results from our ANOVAs on complete participant data sets with missing data imputed, but note with an "*" effects and trend results that were found to be no longer statistically significant (p>.05 and p>.10, respectively) when incomplete participants' data were discarded. 4.3 Results We first present the key performance results (completion times, number of extra steps, number of attempts before mastery), followed by perceived learning experience and preference. 58 4.3.1 Overview of data across the four learning phases We summarize participants' performance data (i.e., extra steps, task completion times) in Figure 4.3 to illustrate how each experimental group performed in the four phases. Although most participants took fewer than 10 attempts to master each of the two task sets, a number of participants took 10 attempts before mastering the basic or advanced tasks; thus we present performance data up to the 10th attempt for the Basic and Advanced Task Acquisition phases. For those participants who achieved mastery in fewer than 10 attempts, we used the completion time and extra step values from their last attempt as a conservative estimate of what their performance would have been on further attempts (these estimates were not used in our ANOVAs). The visual summaries of the overall data shown in Figure 4.3 reveal a number of interface- and age-related differences, all of which were either statistically significant or showed an informative trend in the analysis results (reported in the following sections). Specifically, younger participants using the ML interface initially took on average fewer extra steps in the Basic Task Acquisition and Retention phases than did the younger participants in the control group. Younger participants' task completion times were not affected by interface condition in any of the phases. In contrast, older participants using the ML interface initially took less time, in addition to fewer extra steps, in the Basic Task Acquisition and Retention phases than did the older participants in the control group. In the Advanced Task Acquisition phase, younger participants performed similarly regardless of interface condition and, unexpectedly, the same was found for older participants. As we did expect, older participants consistently took more time over all four phases than the younger participants did to complete task sets. The sections that follow present the statistical analyses of the performance data for each of the four phases. 4.3.2 ML interface helped initial basic task acquisition, particularly for older participants Compared to those in the control condition, both younger and older participants in the ML condition performed fewer extra steps in the first three attempts of mastering the basic task set (main effect of interface*, F1,28=5.7, p=.024, ηp2=.17), supporting H1. A trend result in the data suggest that the ML interface may have provided a greater performance benefit for older adults in terms of fewer extra steps (three-way interaction of interface, age, and attempt*, F1.2,34.2=2.8, p=.0996, ηp2=.09). Inspection of the data showed that older participants in the ML condition took significantly fewer extra steps than those in the control condition for Attempts 1 and 2 (Attempt 1: 47 vs. 138 extra steps; Attempt 2: 17 vs. 42 extra steps; shown in Figure 4.3, callout a). Younger participants in the ML condition also took significantly fewer extra steps than those in the control condition but only in Attempt 2 (16 vs. 43 extra steps). No significant differences were found between interface conditions for the other attempts. These findings offer support for H5, but they need to be interpreted cautiously because they are based on a trend result and require more work to substantiate. 59 Basic Task Set Advanced Task Set 0 20 40 60 80 100 120 140 1 2 3 4 5 6 7 8 9 10 1 2 1 2 3 4 Attempt # E xt ra s te ps 0 20 40 60 80 100 120 140 1 2 3 4 5 6 7 8 9 10 Attempt # E xt ra s te ps younger, ML younger, control older, ML older, control 0 100 200 300 400 500 600 1 2 3 4 5 6 7 8 9 10 1 2 1 2 3 4 Attempt # C om pl et io n tim e (s ) 0 100 200 300 400 500 600 1 2 3 4 5 6 7 8 9 10 attempt # C om pl et io n tim e (s ) Figure 4.3: Mean number of extra steps and completion times for attempts in all four ML learning phases (N=32). Callout areas a, b, and c are referred to in the text. a a b c Basic Task Acquisition Retention Transition Advanced Task Acquisition Basic Task Acquisition Retention Transition Advanced Task Acquisition 60 Analysis of task completion times showed that older participants benefited more from the ML interface than did younger participants. We found a 3-way interaction of interface, age and attempt on task completion time (F1.3,36.1=6.7, p=.009, ηp2=.19). Post-hoc pairwise comparisons revealed that in the first attempt, older participants in the ML condition took significantly less time than did older participants in the control condition (338s vs. 641s, p=.002; shown in Figure 4.3, callout b). No other significant differences were found for older participants in the second or third attempt. Younger participants in both interface conditions took similar amounts of time in their first three attempts to perform the basic task set. These findings support H5, but do not support H1. As expected older users took significantly more time overall than younger ones did to complete the basic task set (main effect of age, F1,28=34.3, p<.001, ηp2=.55). Although participants using the ML interface mastered basic task sets in fewer attempts on average than those using the control interface (shown in Figure 4.4, left bars), no statistically significant interface- or age-related differences were found in the number of attempts participants took to master the basic task set. This finding provides no support for H1. 0 2 4 6 8 10 Basic Task Acquisition Advanced Task Acquisition To ta l # o f a tte m pt s un til m as te ry younger, ML younger, control older, ML older, control Figure 4.4: Mean number of attempts until mastery in the Basic Task Acquisition (N=32, four left bars) and Advanced Task Acquisition phases (N=26. four right bars). Error bars represent 95% confidence intervals. 4.3.3 ML Interface helped participants retain mastery of tasks The ML interface helped both age groups to retain their mastery of the task sets. Specifically, participants using the ML interface required significantly fewer extra steps than those in the control group to perform the basic task sets 30 minutes after mastering them (main effect of interface, F1,28=25.0, p<.001, ηp2=.47). This finding supports H2. No significant interaction of interface and age was found, offering no support for H5. A trend result in the data suggests that older participants in the ML condition took less time to complete the Retention phase tasks than those in the control condition (interaction of interface and age on completion time*, F1,28=3.8, p=.060, ηp2=.12). Inspection of the data (shown in Figure 4.3, 61 callout c) revealed that older participants in the ML condition took less time than those in the control condition (137s vs. 170s). There was only a minimal difference between interface conditions for younger participants. These findings offer support for H5, but need to be interpreted cautiously because they are based on a trend result. As expected, older users took significantly more time than younger ones did to complete the basic task set (main effect of age, F1,28=57.4, p<.001, ηp2=.67). 4.3.4 Transition in using ML interface led to a negative effect on performance In the Transition phase, we looked at how participants in the ML interface condition performed on the Full-Functionality layer after learning on the Reduced-Functionality layer. We compared the performance of participants in the ML interface to those in the control condition who did not have to transition to another interface layer. As hypothesized, the ML condition had a negative effect on the completion of basic tasks after participants transitioned to the Full-Functionality layer. ML participants took more extra steps and more time to complete basic tasks on the Full-Functionality layer than did the control participants (main effect of interface on extra steps, F1,28=5.8, p=.023, ηp2=.17; main effect of interface on completion time, F1,28=7.3, p=.012, ηp2=.21). These findings support H3. Although Figure 4.3 shows a larger effect of interface for the older participants compared to the younger participants, no significant interaction between interface and age on performance (i.e., extra steps, task completion times) was found. However, we did find that older participants took significantly more extra steps and were slower than younger participants as we expected (main effect of age on extra steps*, F1,28=4.5, p=.043, ηp2=.14; main effect of age on completion time, F1,28=69.2, p<.001, ηp 2=.71). 4.3.5 ML did not help nor hinder advanced task acquisition. Performance during the first three attempts for learning the advanced task sets (on the Full- Functionality layer) was similar for both interface conditions. No significant effects of interface condition were found on extra steps taken or task completion times, offering no support for H4 or H5. However, compared to younger participants older participants did take more extra steps and more time to complete the advanced task sets during the first three attempts (main effect of age on extra steps, F1,28=18.0, p<.001, ηp2=.67; main effect of age on completion time, F1,28=57.3, p<.001, ηp 2=.39). Participants in the two interface conditions mastered the advanced task set in similar number of attempts. One younger participant (in the control condition) and five older participants (two in ML condition, three in control condition) did not master the advanced task set within 10 attempts. Of those who did master the task set within 10 attempts, the ML participants required slightly more attempts to do so (M=5.8, SD=1.9) than did those in the control condition (M=5, SD=1.3; see 62 Figure 4.4, right bars); however, no significant effect of interface was found, offering no support for H4. 4.3.6 ML helped older participants spend less time using menus, but also navigating and entering text We performed additional exploratory analyses to gain insight into which types of steps (i.e., button presses associated with a particular aspect of the interface) contributed most to differences in performance. We classified all possible steps into one of four types: contact navigation: navigating through the contact list and data fields of an individual contact; menu: opening and navigating through the options menu, and executing a function in the menu; function: function- specific steps (e.g., choosing a ringtone, saving changes); and, typing: entering and correcting text. Table 4.3 shows steps and task completion times broken down by these four categories. Minimum number of steps Basic Task Set Advanced Task Set Step type Reduced- Functionality Layer Full- Functionality Layer Full- Functionality Layer Contact Navigation 23.8 23.8 43.6 Menu 9.0 24.0 16.0 Function 6.0 6.0 12.2 Typing 36.4 36.4 15.2 TOTAL 75.2 90.2 87.0 Table 4.3: Mean minimum number of steps required to complete the basic and advanced task sets, classified by step type. Separating performance data (i.e., extra step, task completion times) into the four types provides insight into which parts of the task were more difficult during the Basic Task Acquisition phase (mean data shown in Figure 4.5). As expected based on the differences in Table 4.3, participants using the ML interface's Reduced-Functionality layer took fewer extra menu steps than did those using the control interface's Full-Functionality layer (Figure 4.5, graph b). More interesting, however, was that completion times differed for the two age groups: younger participants spent similar lengths of time on menu steps regardless of interface condition, whereas older participants using the ML interface's Reduced-Functionality layer took less time using menus than those using the control interface's Full-Functionality layer. Thus menu-related extra steps appear to negatively affect task completion time for older participants but not for younger participants. 63 Contact Navigation 0 20 40 60 1 2 3 Attempt # Menu 0 20 40 60 1 2 3 Attempt # Function 0 20 40 60 1 2 3 Attempt # Typing 0 20 40 60 1 2 3 Attempt # younger, ML younger, control older, ML older, control Contact Navigation 0 100 200 300 1 2 3 Attempt # Menu 0 100 200 300 1 2 3 Attempt # Function 0 100 200 300 1 2 3 Attempt # Typing 0 100 200 300 1 2 3 Attempt # Figure 4.5: Mean number of extra steps and completion times for first three attempts in Basic Task Acquisition phase, separated by step type (N=32). a b c d e f h g C om pl et io n tim e (s ) E xt ra s te ps 64 The performance data separated by step type also suggest that, in comparison to the Full- Functionality layer, the Reduced-Functionality layer also helped older participants navigate through contact lists/details and type text with fewer extra steps. These differences were unexpected because contact navigation and typing required the exact same sequence and number of steps in both interface conditions. Older participants using the ML interface's Reduced- Functionality layer, compared to the control's Full-Functionality layer, took fewer extra steps and less time on both contact navigation (Figure 4.5, graphs a and e) and typing (Figure 4.5, graphs d and h). Looking across the four phases, older and younger participants generally performed a similar number of extra function- and typing-related steps but older participants consistently took more time than did the younger ones. This was likely due to differences in older and younger participants' perceptual and motor speeds. 4.3.7 Participants in each interface condition received similar amounts of help A number of older participants were noticeably stuck for more than two minutes while learning to perform tasks. These older participants, an equal number from both interface conditions, received help from the experimenter. Specifically five older participants from the ML interface condition required help a total of 15 times (Basic Task Acquisition Phase: 2 times, Advanced Task Acquisition Phase: 13 times) and five older participants from the control condition needed help a total of 16 times (Basic Task Acquisition Phase: 7 times, Advanced Task Acquisition Phase: 9 times). Participants in the control condition received more help while learning basic tasks but those in the ML condition received more help while learning advanced tasks, which was expected because those were the phases when participants first performed tasks on the more complex Full- Functionality layer. 4.3.8 Older participants perceived ML interfaces to be less complex At the end of the session, participants used a six-point Likert scale to rate how much they agreed with statements related to learning on their assigned interface. Interface- and age-related differences were found on perceived application complexity, ease in remembering function location, and frustration. Although participants generally disagreed that they "felt overwhelmed by the complexity of the address book program," older participants found the ML interface to be significantly less complex than the control interface (a significant 2-way interaction of interface and age: F1,28=4.2, p=.049, ηp2=.13). Older participants rated the ML interface (1.4 out of 6) lower than the control interface (2.3) on complexity (significant difference found in post-hoc pairwise comparisons, p=.011) while younger participants found the complexity of both interfaces to be similar (1.8 and 1.8, respectively). 65 Participants in the control group appeared to find it easier to remember function locations for both basic and advanced functions while those in the ML interface group found remembering function locations more difficult after transitioning to the Full-Functionality layer. Specifically, participants in the ML and control conditions generally agreed after mastering the basic task set that "it was easy to remember where all the [needed] functions . . . were located" (4.6 and 4.3 out of 6, respectively); however, after mastering the advanced set of tasks, participants in the ML condition dropped to neutral while ratings for participants in the control condition remained positive (3.6 and 4.4, respectively; a significant two-way interaction of interface and phase, F1,28=8.9, p=.006, ηp2=.24; significant difference found in post-hoc pairwise comparisons, p=.011). This difference may be due to interference between the two layers that participants in the ML condition had to adapt to. Although all participants disagreed that "performing the basic task set was frustrating," a trend result in the data suggested participants in the ML condition found performing the basic task set less frustrating compared to participants in the control condition (1.8 and 2.4 out of 6, respectively; main effect of interface, F1,28=4.2, p=.0502, ηp2=.13). 4.3.9 Most older participants prefer ML interface for learning, mixed preference for younger participants In the post-task interview, we described both the ML and control address book interfaces to participants and asked participants to indicate their preference. Specifically, participants were asked to choose an address book interface (including one that consisted only of the Reduced- Functionality layer) they would most like to use if they had to: learn to perform the task sets over again, and use the application long-term. They were also asked to explain why they chose a particular interface. Figure 4.6 shows a summary of participant preferences. One older participant in the control condition completed almost the entire study but did not have time to state his interface preferences in the allotted time. We summarize data for only 15 of the older participants. Interface Preference 0 2 4 6 8 ML control ML control ML control ML control younger older younger older # of p ar tic ip an ts Prefer Control Prefer only Reduced- Functionality Layer Prefer ML Figure 4.6: Participants' interface preferences for learning and for long-term use (N=31). for learning for long-term use 66 The majority of older participants (10/15) reported they would prefer to use a ML interface if they had to learn the task sets over again. The main reasons for this preference were that the Reduced- Functionality layer was perceived as being "simpler" and "easier", and that learning on a simpler version allowed them to become comfortable before moving to a more complex one. Interestingly, of those in the control condition, the majority of the older participants (5/7) indicated a preference for using the ML interface for learning, while none of the younger participants did; thus older participants perceived value in learning on ML interfaces, even without having used one, whereas younger participants did not. One older participant who preferred learning on the ML interface explained, "learning [the Full- Functionality layer] . . . is more complex, so it's more difficult to get . . . the hang of how the thing works . . . going from simpler to complex [is] a natural progression of learning" (P19, ML). By contrast, some older participants (5/15) reported they would prefer learning on a non- layered interface over the ML interface. The primary reason for this preference was that the non- layered interface did not require the user to learn how to use one layer and then later learn another. Having different menu content for each layer also meant that menu items would likely be in different positions in the menu depending on the layer, which a number of participants felt would be confusing. One older participant commented, "if you mastered these [functions in Reduced- Functionality layer] and then go to these [in Full-Functionality layer], then you've got two things to worry about" (P26, control condition). The majority of younger participants in the ML interface condition (5/8) reported that they would prefer learning with the ML interface, while none of the younger participants in the control condition reported this preference. Younger participants in the ML interface condition preferred learning on the ML interface because it allowed them to quickly learn to perform basic tasks. Younger participants in the control condition reported preferring the non-layered control interface because the address book interface was overall easy to learn to use and they preferred learning the functions all at once without having to relearn when transitioning to a new layer. 4.3.10 Interface preference for long-term use was mixed Older participants reported a mix of preferred interfaces for long-term use. Over half of the older adults (8/15) would prefer using a non-layered interface over the long run. The main reason for this preference was that having all functions in one layer made it easier to find a function. In fact, some participants commented that it would be better if all functions were in one menu even if more scrolling and menu pages were required, as opposed to one menu for the contact list screen and another for the contact details screen. Six of the older participants would prefer to use a ML interface over the long run, as long as the Reduced-Functionality layer had all the functions that they would commonly use. P27 (ML condition), an older participant, commented that she "would rather have [the ML interface, where the Reduced-Functionality layer] shows commonly used functions . . . if I set [the Reduced-Functionality layer] up with my personal preferences, then 67 that's probably what I would use most of the time." Over half of the older participants in each of the interface conditions (ML or Reduced Functionality layer only: 5/8; control: 5/7) expressed a preference for the interface they had used, so the interface used in the experiment may have had some influence on their long-term preference. However, while neither interface was preferred by a large majority of the older participants for long-term use, their comments offer important design- related insights. The majority of younger participants (10/16) indicated a preference for the ML interface for long-term use. The main reason for this preference was that they could imagine the ML interface helping them be more efficient in performing common tasks. In contrast, five younger participants preferred the non-layered control interface for long-term use because this interface allowed access to all functions without requiring any switching between layers. 4.3.11 Summary of results This study revealed the following critical findings concerning each of our hypotheses. H1. Basic Task Acquisition: partially supported. The ML interface's Reduced-Functionality layer (compared to the control's Full-Functionality layer) helped participants to master basic task sets in significantly fewer extra steps (Attempts 1-3). However, our ML interface did not help participants to master basic task sets in fewer attempts nor helped younger participants to master these task sets in less time. H2. Retention: supported. The ML interface's Reduced-Functionality layer (compared to the control's Full-Functionality layer) helped participants to better retain their task mastery in terms of significantly fewer extra steps. H3. Transition: supported. Transitioning from the Reduced-Functionality to Full- Functionality layer in the ML interface (compared to using the control interface and remaining on the Full-Functionality layer) negatively affected participants' performance in terms of significantly fewer extra steps and lower task completion times. H4. Advanced Task Acquisition: not supported. We found no effect of interface on number of extra steps, task completion time or number of attempts in acquiring advanced tasks. H5. Benefit to Older Adults: partially supported. The ML interface's Reduced-Functionality layer (compared to the control's Full-Functionality layer) helped older participants significantly more than younger ones to master basic task sets in less time (Attempts 1-3). Trend results in the data suggest that the ML interface's Reduced-Functionality layer (compared to the control) helped older participants more than younger ones to master basic task sets in fewer extra steps (Attempts 1-2); and better retain their mastery of basic tasks in terms of task completion times. Compared to younger participants, older participants also perceived the ML interface to be less complex and preferred it for learning. 68 4.4 Discussion In this section, we discuss the learning-related benefits that a ML mobile user interface might have for older adults. We then discuss the limitations of our research and the generalizability of our findings. 4.4.1 ML interfaces help users learn basic mobile application tasks As hypothesized, we found that ML interfaces can help both younger and older adults to learn to perform tasks on a mobile application. Specifically, our ML interface's Reduced-Functionality layer (compared to the control) helped participants during initial attempts to learn to perform basic task sets in fewer extra steps. This finding is consistent with findings of past studies on desktop ML interfaces (Findlater & McGrenere, 2007; Carroll & Carrithers, 1984) including a study that focused on older adults (Dickinson, Smith, Arnott, Newell, & Hill, 2007). We also found that ML interfaces helped users to remember and perform tasks (in terms of fewer extra steps) that they had mastered 30 minutes earlier. We hypothesized that learning on the ML interface's Reduced-Functionality layer would improve participants' performance in learning the advanced task set on the Full-Functionality layer, because we expected that it would be easier for participants to form a mental model of the UI when learning on the simpler Reduced-Functionality layer. However, our results did not support this hypothesis. Based on responses in our semi-structured interview, we found that most of our older participants from both interface condition groups appeared to have had difficulty forming a mental model of how the menus worked. Thus, they could not use such a model to help them learn the advanced task set. In contrast, we found through our semi-structured interview that most younger participants were able to form an accurate mental model of the options menus. However, younger participants' performances on advanced tasks were similarly good on both interfaces, perhaps because the tasks required little effort for them to master. Thus it remains unclear whether learning on a Reduced-Functionality layer helps the user better form a mental model and more research is needed to answer this question. Our ML interface's lack of benefit for learning advanced tasks stands in contrast to Catrambone and Carroll's (1986) finding that their participants were able to more quickly perform an advanced task after acquiring a basic task on the reduced-functionality Training Wheels interface. This discrepancy may be due to differences in the two ML interface designs because the Training Wheels interface blocked but still showed advanced functions, which helps users to transition from their initial interface layer to a full-functionality interface. The discrepancy in findings could also be due to other differences between the studies, such as the type of application, tasks, or learning process prescribed in the two studies. For example, our experiment had participants repeatedly perform the same type of short mobile address book tasks until mastery, while Catrambone and Carroll had participants perform longer word processing tasks (one basic 69 and one advanced) on a desktop computer until completion. More work is needed to better understand the types of basic and advanced tasks, application and learning process that would most benefit from ML interfaces. 4.4.2 ML interfaces offer performance and preference benefits to older users Prior to this study, we expected that the ML interface would benefit older adults more than younger ones. Although we did not find a benefit on all measures, we did find evidence in participants' performance data that ML interfaces help older adults more than younger ones during the initial learning process. For example, older participants using the ML interface were able to learn to perform the basic task set for the first time in 90 fewer extra steps and 5 fewer minutes, on average, than those in the control condition. Even after mastering the basic task set and then taking a break, older participants using the ML interface were able to perform the basic task set in 32 fewer extra steps and 47 fewer seconds, on average, than those in the control condition. We predict that this initial performance benefit of ML interfaces would reduce one of the barriers older adults experience in adopting mobile technology. Making errors, such as the extra steps taken in our study, on a new interface often requires a high cost for recovery, which can be particularly frustrating for older adults (Birdi & Zapf, 1997). It is not clear from the literature how many attempts and how much time an older adult will spend on learning new technology before abandoning it. However, it is likely that the first few attempts are critical for novices to gain or lose confidence in their ability to learn to use the device. As noted earlier, older adults have been found to be more negatively affected by errors (Birdi & Zapf), so we expect that minimizing the number of extra steps and time required to perform tasks on new technology will reduce frustration and increase the chance of technology adoption by older adults. Participants' subjective data were in line with the quantitative performance results. The majority of older participants preferred learning basic tasks on the ML interface because it was perceived as simpler and easier to use, which is consistent with findings by Dickinson et al. (2007). By contrast, younger participants were more mixed in their interface preference for learning. Many older participants also found that learning on a simpler layer first before progressing to a more complex layer was a more natural and comfortable way to learn. Participants in the ML condition were also less frustrated during the learning process than those in the control condition. The reduced complexity of the initial ML interface layer seems to have improved the initial learnability of our mobile address book application for older adults. Many older participants commented that when using the Full-Functionality layer, they often forgot which menu a function was located in. We frequently observed older participants closing a menu when using the Full- Functionality layer and immediately reopening the same menu to read through its items; this was not observed in the Reduced-Functionality layer where menus only had two or three items. Older 70 participants, as well as some of the younger ones, commented that entering text required considerable mental effort and that it was challenging to remember which button to press to switch to the appropriate text entry mode. The reduced complexity of the initial ML layer appeared to make it substantially easier for older participants to learn to use the menus and other parts of the interface, including entering text. Younger participants did not appear to experience this benefit; they performed similarly regardless of interface, perhaps because the mobile application and tasks were so simple that it did not matter which interface they used. This age-related difference is consistent with past studies on younger and older adults' performance on mobile device tasks (Ziefle & Bay, 2004; 2005) and is likely due in part to differences in visuo-spatial working memory between our two age groups. Because ML interfaces provide an initial benefit to learning basic tasks on a reduced- functionality layer and no hindrance to learning advanced tasks on a full-functionality layer, ML interfaces can be used to increase the overall learnability of the application, particularly for users who do not want or need to learn to perform advanced tasks. For example, many of our older participants commented that they normally would not need voice dialing or other advanced features in their daily lives; thus, a ML interface with basic task functions in its reduced- functionality layer would be more learnable for these participants compared to users who also need to perform advanced tasks. There was nothing in the participant feedback to suggest that providing a third layer would add any value. 4.4.3 Simplicity valued When choosing an interface to use, we found that older participants generally chose the one they perceived as being simpler. However, simplicity appeared to carry different meanings for learning a new application compared to using it long term. The ML interface was preferred for its simpler Reduced-Functionality layer. In this case, simpler referred to a reduced amount of information (e.g., available functions) in the interface to learn, while trying to perform a new task. For long- term use, older participants still chose the interface that they perceived as being simpler, but their interface preferences were mixed. Those who preferred the ML interface for long-term use wanted all commonly used functions in the Reduced-Functionality layer and felt that not having the "fancy stuff" in this layer would make the ML interface simpler. In contrast, older participants who preferred the non-layered control interface felt that having all functions in one layer made it simpler to search for a desired function; one participant commented that he preferred scrolling through a long options menu over remembering where, amongst several menus, a function was located. A design approach that arose from our participant interviews that may meet the needs of both groups of older adults would be a personalized ML interface that allows users to choose which functions to place into the Reduced-Functionality layer. Users could place commonly used functions or all functions that they would ever want to use into the Reduced-Functionality layer, as 71 they desired. McGrenere et al. (McGrenere, Baecker, & Booth, 2002) evaluated this approach implemented on a desktop word processing application, which was found to help participants better navigate menus and learn the application. McGrenere et al. also found that participants who favoured a simpler interface were willing to take the time to personalize it. Based on these findings, we expect that older users can benefit from a customizable ML mobile interface. 4.4.4 ML mobile application design We have found that designing a ML mobile application requires addressing similar challenges as designing a ML interface for a traditional desktop platform. For example, designers for both mobile and desktop platforms need to carefully determine how many layers to implement and which functions should be included in the different layers. We chose to start with a two-layer application because no previous study had formally evaluated the learning benefits of a ML mobile application. For this application, we surveyed existing users to determine which functions they thought they should learn first. We placed those functions in the initial layer. Alternative criteria for selecting initial layer functions include choosing the most commonly-used functions, the simplest functions, or functions that are the easiest to learn. While designing ML mobile and desktop applications has similar challenges, working within the constraints unique to mobile user interfaces requires special design considerations. For example, once the function sets for each layer have been determined, the mobile application designer needs to decide how the functions will be shown to the user on the mobile device's small screen. In our application, users access functions through menus. We needed to determine how to order the functions in those menus. This ordering is particularly important on mobile applications because the device's small screen size limits how many functions can be shown at once; if there is not enough space to show all functions, then functions may need to be placed on different screens/pages. We chose to group related functions together, which is common in many mobile and desktop applications. As discussed in Section 4.2.2, an alternative approach would have been to group functions by layer so that basic functions are at the top of the menu, followed by advanced functions, which may better help users to transition from the first layer to the second. There are also several options for setting the visibility of functions that are not used in a layer. For example, advanced functions in an initial layer can be hidden or marked as being disabled (Findlater & McGrenere, 2007). We felt that hiding functions that are not used in a layer was the best approach for our application because it minimized switching between menu pages and it reduced the visual complexity in the initial layer's menu. The designer needs to implement some mechanism for the user to switch between the multiple layers. Although we did not implement this in our application, this mechanism could be implemented by adding a function at the end of each menu for switching to another layer. Another approach would be to dedicate a button on the device for cycling through the layers. The designer 72 needs to ensure that controls for switching between layers do not significantly interfere (e.g., visually) with performing regular tasks. 4.4.5 Limitations The research presented in this chapter has a number of limitations. The study involved 16 older and 16 younger participants, which is a relatively small sample; a larger sample size is needed to see whether or not the trend results that we identified are trustworthy. Our older participants were generally well educated, had computer experience, and thus may not be fully representative of the older adult population. Future work is needed to see whether our findings hold for larger, more diverse samples of older adults. The duration of a study session required a trade-off between investigating user performance over many phases of using a ML interface, and participants' energy levels. The study might have been too long for some participants, which may have led to more errors. Conversely, the study was too short to explore long term learning, retention and preferences. A longitudinal study is required in the future to research the benefits of the ML interface over a longer term. 4.4.6 Generalizability Our findings may be more applicable to older adults who are more experienced with using such mobile devices and computer technology than the whole older adult population. As stated in Section 4.2.1, four older participants who could not complete the study were replaced. One of these participants (in the ML group) almost completed the Basic Task Acquisition phase but became noticeably tired and uncomfortable using the device; this discomfort may have been due to arthritis in her hands, which she only informed us of in the middle of the study session. The other three participants (one in the ML group, two in the control group) were able to complete the first two learning phases but took so long to master the basic tasks that they did not have time in the four-hour study session to complete the last two learning phases. These participants appeared to perform similarly to other participants in the first few attempts but continuously made typing errors in later attempts, so it took them much longer to reach our mastery criterion and move on to the next phase of the study. From our observations, these typing errors seemed to be caused by fatigue, as well as carelessness (e.g., saving a contact's information without verifying that no typing errors were made). We speculate that the effect of age on our results may have been stronger had these participants been able to complete the study and their data were included in the analysis. We expect our results may generalize to many other mobile devices. Many mobile devices, even those with different form factors (e.g., smaller screen with fewer buttons than the E61i, large touch screen), use multiple function menus and have relatively similar ways of navigating from one application screen to another. We expect that one of the differences among mobile devices that may most influence learning effort for performing new tasks is the device's input method, 73 particularly for text entry, which can vary widely (e.g., single button press vs. multi-tap, on physical vs. on-screen keyboard). We also expect our results to generalize to many other mobile applications. The address book's interface that was developed for this experiment required the use of many standard mobile interactions, such as soft keys and a direction pad for navigation, and menus (one per screen) for browsing and executing functions. The application itself is relatively simple compared to many existing mobile applications (e.g., calendars, Internet browsers); we expect that ML interfaces would provide greater initial learning benefits to both older and younger novices on more complex mobile applications. 4.5 Summary This chapter presented the findings of an experiment that explored the effects of a ML interface for helping older adults to learn mobile applications. We found that our ML mobile address book application provided participants an initial learnability benefit in terms of fewer extra steps taken. The ML application also helped participants to better retain the ability to perform the tasks they had mastered 30 minutes previously. When participants transitioned to a full-functionality interface layer, they experienced a temporary decrease in basic task performance presumably because they needed to relearn the function menus. However, no negative effect was found on learning advanced tasks in the full-functionality layer. We found that our ML application helped our older participants more than our younger ones to perform initial basic tasks in less time. Further, the majority of older participants preferred learning on the ML interface and found it less complex than the non-layered control interface. Given the initial performance benefit, the overall preference for the ML interface for learning, and lack of major drawbacks, the ML interface appears to be a suitable design approach for improving mobile applications for older adults and lowering barriers for adoption. 74 Chapter 5 5. Augmenting the Mobile Device's Display: A Survey In this and the next chapter, we present an investigation of our third design approach – augmenting the mobile device's small display with a larger display – to support older adults in the learning process. Little was known prior to our research about how an additional display could help older adults to learn. Published findings on older adults' needs and preferences for learning to use mobile devices were mixed (see Section 2.4). This chapter describes a comprehensive questionnaire we created to survey older adults on these topics. We begin the chapter by specifying our investigation objectives and research methodology. We then explain the design of the questionnaire, and present key survey results related to designing interactive technology resources to help older adults learn to use mobile devices. We then discuss the implications of our findings. Findings from this survey were used to inform the design of an augmented display learning system, which is presented in Chapter 6. 5.1 Objectives and methodology Our third design approach focused on providing supportive scaffolding to help older adults during the learning process. As mentioned earlier in this dissertation, current mobile devices can easily connect to other displays, offering an opportunity to augment a device's small screen and build new types of learning support resources. Although the literature offers some principles for the design of learning and help resources (Section 2.4.1), it is unclear how well these principles apply to today's mobile phones and whether they are appropriate for the unique needs of older adults. We chose to conduct a survey instead of using other research methods in order to rapidly gather responses from a large sample drawn from the older adult population, which is significantly more heterogeneous than the younger adult population in terms of physical, sensory and cognitive abilities, and working and living environments (Gregor, Newell, & Zajicek, 2002). Surveys are 75 commonly used to inform design requirements and have been used specifically to investigate older adults' self-reported learning preferences, which in turn have uncovered important implications for designing and deploying help/learning resources (e.g., Rogers, Cabrera, Walker, Gilbert, & Fisk, 1996; Selwyn, Gorard, Furlong, & Madden, 2003). We chose to create a more comprehensive questionnaire rather than one that focused solely on designing an augmented display system, in order to both inform our augmented display system design and to obtain more generalizable results. We focused on a range of mobile devices (e.g., digital cameras, cell phones, electronic organizers) instead of focusing on only one type of device to obtain results that are more generalizable across mobile devices and that might apply to future mobile devices. We also surveyed younger adults, as well as older adults, to ground our findings. Our survey had two primary objectives. Our first objective was to better understand older adults' existing needs and preferences in learning to use mobile devices. Our second objective was to identify ways to design more suitable and effective learning support resources for the population of older adults. 5.2 Methods We begin by describing our participants. We then briefly present the questionnaire used in this research, followed by the procedure and data analysis. 5.2.1 Participants We recruited adults of age 20+ and formed three age groups. Similar to our previous user studies, we recruited a group of older adults (age 65+) and a group of younger adults (ages 20-49). We were also interested in middle-aged adults (ages 50-64), a common age group in aging and technology research studies (Smith, 2010; Ziefle & Bay, 2004). While middle-aged adults may have begun to experience the same natural declines in cognitive abilities experienced by older adults, this middle-aged adult group differs from the older adult group in its proportion of those still employed, and thus group members' work environments and social networks. Work environment and social networks are factors that are likely to affect access to learning resources and learning preferences, which are relevant to our study. We recruited participants from senior homes, community centers, and libraries in the Greater Vancouver Regional District, as well as by means of online classifieds (http://vancouver.en.craigslist.ca/) and BCNAR's (BC Network for Aging Research) SMART program (Senior Mentors Assisting Researchers and Trainees). We sought individuals who had used mobile devices in the past, as specified in our call for participation posters/postings (see Appendix C.1). Participants were offered a chance to enrol in a draw for one of ten gift cards. One hundred and thirty eight completed surveys were returned, but seven had to be discarded because the respondent's age was less than 20 (two surveys) or because they included mostly 76 incoherent responses (five surveys). Participants were asked to indicate their perceived mobile device expertise (see Table 5.1 for definitions that appeared on the questionnaire). Of the 131 respondents, just over half of those who self-reported being "advanced" mobile device users were younger users (12 out of all 22 advanced users). The vast majority of self-reported "beginners" were either older or middle-aged adults (14 out of all 15 beginners). In order to maximize the discovery of age-related differences, we analyzed only the data from the respondents who reported being "novice" and "intermediate" mobile device users (N=94). Findings reported in this chapter are based on the questionnaires completed by those individuals. Expertise level Definition Beginner Starting to use and have no or very little experience. Novice user Can use one to three programs or features on device/computer with help. Intermediate user Can use several programs or features on device/computer without help. Advanced user Can use "advanced" features on device/computer and/or install new programs. Table 5.1: Definitions used in questionnaire for different levels of expertise in using mobile devices and desktop computers. After the exclusion of the beginners and advanced mobile device users, the three age groups were similar on many levels (see Table 5.2). We ran Kruskal-Wallis tests on the demographic data and did not find significant group differences with respect to gender, education, housing status, reported computer expertise, or years of experience with mobile devices. There was a significant difference with respect to employment status (χ2=16, df=2, p<.001); as expected a larger number of younger respondents were students, while a larger number of older respondents were retirees. Even after our attempt to minimize differences in reported mobile expertise, there was still a significant difference between the three groups (χ2=6.4, df=2, p=.041). More younger respondents were classified as "intermediate" mobile device users than as "novice" users while the older and middle-aged respondents were evenly divided across the two expertise levels. Thus our younger age group appeared to have had more mobile device expertise than the other two groups. This needs to be considered when interpreting the results. 77 Younger respondents Middle-aged respondents Older respondents N 28 34 32 Age* M (SD) 27.7 (7.7) 57.1 (3.9) 73.1 (5.5) Gender # male # female 8 20 11 23 15 17 Employment status* # student # working # retired 11 17 0 0 23 11 0 2 30 Computer expertise # "novice" # "intermediate" # "advanced" 2 18 8 4 23 7 3 26 3 Mobile expertise* # "novice" # "intermediate" 7 21 19 15 16 16 Mobile experience # 0-5 years # 6-10 years # 10+ years 7 18 3 11 13 10 25 11 6 *: significant difference between age groups Table 5.2: Participant characteristics of the three age groups (N=94). 5.2.2 Materials The materials for this study consisted primarily of a questionnaire we created to assess older adults' needs and preferences in learning to use mobile devices. Questionnaire The Learning Methods for Mobile Devices Questionnaire we created for this study (referred to hereafter as the Learning Methods Questionnaire; see Appendix C.3 for the paper version and exact wording of questions) includes both closed response questions to solicit quantitative data, and also many open ended questions (18 total) to obtain rich data that we could analyze qualitatively. The Learning Methods Questionnaire has five main sections. The first section has demographic-related questions (Q1-Q5) on: age, education, gender, housing, and work status. The second section has questions related to participants' current mobile device experiences and needs when learning to use mobile devices (Q6-Q10, Q16). Learning needs are assessed with questions about how often participants have wanted to learn to do something new on their mobile device or forgot something they had learned. Learning needs are also assessed with a question about how important it is for respondents to be able to figure out the exact steps required to perform a task on their mobile device, and how important it is for them to gain a general understanding of how the software works. The third section consists of one question (Q11) that focuses on the qualities/features in a learning resource that are important to participants. Participants are asked to rate how important various qualities and features are to them (see Table 5.3). A participant can optionally add one other quality or feature of their choice. 78 Resource qualities and features Is very affordable (e.g., free) Is easy to access (e.g., convenient, readily available) Is easy to understand (e.g., clear, simple language) Is friendly and patient (e.g., not condescending or intimidating) Is it interactive (e.g., gives feedback, answers your questions) Does it allows me to learn by myself Does it allows me to learn in a group (e.g., with friends, classmates) Does not require much time to use Demonstrates how to perform task Explains how the device and programs work Provides detailed information Provides opportunities to practice task Provides step-by-step instructions Table 5.3: Learning resource qualities and features listed in Learning Methods questionnaire. The fourth section includes four questions to assess participants' motivations for using particular learning methods from four different angles. This section assesses participants' motivations for using each of 11 learning methods (see Table 5.4). The list of learning methods includes all the learning methods identified by Mitzner et al. (2008) and all learning resources listed by Selwyn et al. (Selwyn, Gorard, Furlong, & Madden, 2003), except talking to "neighbours" or "other member of household" (Selwyn et al, 2003, p. 574) because Selwyn et al. found that these methods were almost never used by their older adult respondents. A participant can optionally add one other learning method of their choice. For each of the 11 learning methods, the first question (Q12) asks respondents to rate how likely they would use this method and explain in a few words why they would or would not use the method. Two follow-up questions ask respondents to list the three learning methods they would most prefer using if they had easy access to all methods (Q13), and list the three learning methods that best help them retain what they learned (Q15). The fourth question (Q14) asks participants to rate how helpful each of the 11 learning methods are, on a scale of 1-6, in learning to use a mobile device (1=not at all helpful, 6=very helpful). While some people may prefer learning methods that are helpful, others may prefer ones that are most convenient or more familiar but are not necessarily ones that most effectively help them to learn. Learning methods I try working it out for myself by trial and error I use the device's help feature I use the device's instruction manual I phone customer or IT support I search the Internet for help I take a class (e.g., at library, community centre) I talk to my partner/spouse I talk to my children I talk to family/friends from my generation I talk to family/friends from younger generation I talk to my work colleagues Table 5.4: Learning methods listed in Learning Methods questionnaire. 79 The fourth section also includes a question focused on the use of hand-written notes for using mobile devices and desktop computers (Q17). Although notes can help users learn to perform new tasks, we looked at notes separately from the other 11 learning methods as notes can be self- authored and used as reminders when performing learned tasks. The fifth section of the questionnaire consists of one question about a hypothetical augmented display learning system. This section describes in text an imaginary system that enables users to connect their mobile device to their home computers, and that can guide users step by step through the types of mobile device tasks that they would want to carry out. The sole question in this section (Q18), which had four subparts, asks participants about perceived benefits and drawbacks of the system, how easy it would be to operate both the device and a home computer at the same time, and whether they would use such a system. To increase the accessibility of our questionnaire, we created both an online version and a paper version. Other than the manner in which they were presented and filled out, the online and paper versions were exactly the same. Both versions of the questionnaire began with consent information, specifying that a person's consent to participate in the survey was assumed if that person completed and submitted/mailed the questionnaire to us. Both versions of the questionnaire also ended with a page/screen on which participants could enter their contact information if they wanted to take part in the gift card draw and/or wanted to take part in a possible follow-up interview. Of the 94 questionnaires we analyzed, 74 were completed online (16 by older adults, 30 by middle-aged adults, and 28 by younger adults) and 20 were completed on paper (16 by older adults, 4 by middle-aged adults). Questions were presented in a readable text size (13-point Arial font). The paper questionnaire consisted of 13 pages, including the two-page consent information pages, each page printed single-sided on paper. This questionnaire was given to participants with a stamped envelop that was addressed to the dissertation author. The online questionnaire was delivered through UBC's officially supported survey system, Enterprise Feedback Management (EFM). This survey system stores and backs up all data in Canada and thus complies with the BC Freedom of Information and Protection of Privacy Act (University of British Columbia, 2010). Respondents' contact information was collected in a separate survey following the main research questionnaire using the same system. The main online questionnaire consisted of a total of nine web forms (see Figure 5.1 for a screen capture). 5.2.3 Procedure As mentioned earlier, participants either completed an online version or a paper version of the questionnaire. Participants could complete the online questionnaire (available during the study at https://www.surveyfeedback.ca/surveys/wsb.dll/s/1g58b) on any desktop computer with Internet access. Alternatively, participants could fill out the paper questionnaire in any location, at their own pace, and then mail back the completed questionnaire. Paper questionnaires were distributed 80 to the front desk of community centres, senior homes and libraries. One community centre and one senior home allowed us to hand out questionnaires directly to their members who expressed interest in the study. After filling out the main learning needs and preferences questionnaire, participants were invited to submit their contact information in a completely separate and optional questionnaire to allow for participation in the gift card draw. We estimated, based on pilot studies, that participants would take around 20-40 minutes to complete either questionnaire version. Figure 5.1: Screen capture of one section of the online questionnaire. 5.2.4 Data analysis We analyzed our quantitative data using non-parametric tests (i.e., Kruskal-Wallis, Mann-Whitney U and Wilcoxon signed-rank). The alpha was set to .05. Survey questions were analyzed using age-group (20-49, 50-64, 65+) as the independent variable to find age-related differences unless specified otherwise. We also examined respondents' qualitative responses. A coding scheme was created based primarily on salient concepts identified in the literature, such as learning styles, which have been found to differ across age groups (Truluck & Courtenay, 1999). The coding scheme was also based on reoccurring concepts found in the data (e.g., control over learning). Each text response was given a single code to represent the dominant idea expressed in the response. Responses that 81 were ambiguous, blank or incoherent were coded as not answering the question. The coding instructions and scheme (see Appendix C.4 and Appendix C.5) were found to be reliable: two researchers coded the responses from a random 20% sample of the surveys, and a substantial degree of interrater agreement was found (K=.80, p<.001). After this reliability check, one of these two researchers coded all of the remaining text responses. 5.3 Results We present key survey findings here, focusing primarily on older adults' learning needs and preferences and how these needs and preferences differ from those of younger adults. We highlight differences and similarities between older and younger respondents. We show data from middle-aged respondents along side with the data from the other two groups (e.g., in charts), but we only discuss our middle-aged respondents' data in cases when something interesting was found. We first present how often respondents have a learning need, followed by whether respondents want to learn to perform task steps or just gain a general understanding of how to use a device. We then present results on what learning resource features and qualities older respondents perceive as being important, followed by which learning methods they use and why. Finally, we present respondents' feedback on using a hypothetical augmented display system to learn to use a mobile device. 5.3.1 Older respondents need to learn and relearn more frequently To assess our participants' needs for learning to use mobile devices, we asked participants to indicate how frequently they needed or wanted to learn to perform something new, encountered a problem or error that they were not sure how to recover from, and forgot how to perform a task that they had previously learned (Q8). A trend result in the data suggests that older respondents wanted/needed to learn more frequently than younger respondents (Kruskal-Wallis test, χ2=5.9, df=2, p=.051). Inspection of the data showed that more than half of older respondents reported needing or wanting to learn to perform something new more than once a month (shown by combining the "regularly" and "frequently" bars in the top chart of Figure 5.2), while only a quarter of younger respondents reported wanting to do so. This finding may be due to the fact that older respondents have difficulty learning mobile device tasks and thus their learning need remains unmet. With respect to recovering from problems, most respondents (both younger and older) reported encountering problems/errors less than once a month that they were not sure how to recover from. No significant effect of age was found on this measure. Older adults were found to forget more frequently than younger adults how to perform a previously learned mobile device task (see Figure 5.2, bottom chart). A significant main effect of 82 age was found on this measure (χ2=6.1, df=2, p=.047), and two post-hoc Mann-Whitney U tests showed that older respondents forgot learned tasks much more often than did younger respondents (20-49 vs. 50-64: U=329, p=.014; 50-64 vs. 65+: U=463, p=.478). Almost half of the older respondents, compared to only one fifth of younger respondents, regularly (i.e., more than once a month) forgot how to perform a previously learned mobile task. The difficulties older adults have in remembering how to perform a mobile device task may be attributed to older adults' general difficulties in learning new procedures (Fisk, Rogers, Charness, Czaja, & Sharit, 2009), or their device usage frequency, which is likely relatively lower than that of younger adults. Regardless of the cause, these findings support our initial speculation that older adults, compared to younger adults, more frequently need to relearn mobile device tasks they had previously learned, regardless of usage frequency, making it more important to improve the learnability of mobile devices and associated learning resources. 0% 20% 40% 60% 80% 100% 20-49 50-64 65+ % o f r es po nd en ts rarely regularly frequently 0% 20% 40% 60% 80% 100% 20-49 50-64 65+ % o f r es po nd en ts rarely regularly frequently Figure 5.2: Respondents' self-reported frequency of needing/wanting to learn to perform new tasks (N=93, top) and forgetting how to perform learned tasks on their mobile device (N=94, bottom). Frequency: "rarely"=<1/month, "regularly"=1-3/month, "frequently"=1+/week. 5.3.2 Older respondents most want to learn to perform task steps Respondents were asked how important it was for them during the learning process to figure out the exact steps required for a task, and to gain a general understanding of how the software works (Q16). Respondents from older and middle-aged adult groups reported that figuring out the steps was very important and significantly more important than gaining a general understanding (Wilcoxon signed ranks test; 65+: Z=-3.8, p<.001; 50-64: Z=-3.6, p<.001). In contrast, younger adults reported that both options were similarly important (Z=-0.18, p=.86). The top chart in Figure 5.3 highlights this pattern of preferences across the age groups. 83 Respondents were also asked to indicate how important it was for learning resources to provide step-by-step instructions and explanations on how the device and software work (Q11). The results are shown in bottom chart in Figure 5.3. Consistent with the above results, the data showed that older and middle-aged respondents felt it was very important for learning resources to provide step-by-step instructions, and that it was significantly less (but still) important for learning resources to explain how the device and programs work (Wilcoxon signed ranks test; 65+: Z=-2.9, p=.004; 50-64: Z=-2.0, p=.0495). Younger respondents, by contrast, rated these two learning resource qualities as being equally important (Z=-0.12, p=.22). 1 2 3 4 5 6 not important at all very important 65+ 50-64 20-49 1 2 3 4 5 6 not important at all very important Figure 5.3: Median rating scores on the importance of learning exact task steps vs. gaining general understanding (N=93, top), and importance of having learning resources provide step-by-step instructions vs. explain how device and programs work (N=94, bottom). We found that older respondents' ratings on the importance for learning resources to provide both step-by-step instructions and explanations on how the device works were generally higher than those of the younger respondents. As we show later, younger adults strongly prefer learning independently through trial & error compared to using any learning resources, which may explain younger respondents' weaker desired support from learning resources. The findings presented in this section are consistent with findings from Mitzner et al. (2008) and show an age-related difference in learning needs and suggest that learning resources for older adults should be designed primarily to help them to learn the exact steps required for performing a task. "Figuring out the exact steps required for performing a task" "Gaining general understanding of how the software works" "Provides step-by- step instructions" "Explains how device and programs work" 84 5.3.3 Older respondents want demonstrations, opportunities to practice, and to learn individually Respondents were asked to indicate how important it was for learning resources to have a number of other qualities and features (beyond providing step-by-step instructions and explanations on how the device works, Q11). The results generally showed that older adults, relative to younger adults, placed greater importance on a variety of learning resource qualities and features. As shown in Figure 5.4, older respondents, compared to younger ones, placed significantly more importance on having learning resources that demonstrate how to perform tasks (Kruskal-Wallis test, χ2=17, df=2, p<.001), and provide opportunities for practicing tasks (χ2=29, df=2, p<.001), which is consistent with past research findings (Fisk et al., 2009). Older respondents also placed significantly more importance than did younger adults on the interactive nature of a learning resource (χ2=19, df=2, p<.001). We found that our older respondents, as well as younger ones, placed much importance on support for individual learning (median scores: 5 out of 6; no significant effect of age) but little importance on support for learning in a group (2 out of 6; no significant effect of age). This finding supports findings by Selwyn et al. (2003) and Mitzner et al. (2008) that older adults prefer learning by themselves rather than with family or friends. Our survey also revealed that respondents from all age groups felt that it was important for a learning resource to be accessible (median score: 6 out of 6), understandable (6 out of 6), friendly and patient (5 out of 6), affordable (5 out of 6), and provide detailed information (5 out of 6). (See Appendix C.3 for wording used in survey.) These findings are consistent with the literature (Fisk et al., 2009). No significant effects of age on these measures were found. 1 2 3 4 5 6 not important at all very important 65+ 50-64 20-49 Figure 5.4: Median rating scores on the importance of having learning resources be interactive, demonstrate task, provide practice opportunities, and support group learning (N=94). Is Interactive Demonstrates how to perform task Provides opportunities to practice Allows Group learning 85 5.3.4 Older respondents prefer manuals, younger adults prefer trial & error To assess respondents' learning method preferences, we asked questions that elicited both quantitative data scores and qualitative responses to provide additional insights into respondents' quantitative scores. Respondents were asked four related questions to assess their learning method preferences. One of these questions asked participants to choose three most preferred learning methods from our list of 11 methods (Q13). Participant responses to this question revealed age- related differences in learning method preferences. Respondents, regardless of age group, generally listed methods that allowed them to learn alone as one of their top three learning method choices (see Figure 5.5, top chart) over methods that involved learning with others (see Figure 5.5, bottom chart). This preference is consistent with an earlier finding that respondents place much importance on learning independently (see Section 5.3.3). An exception to this finding is middle-aged and older respondents' preference for IT support, which appears to be stronger than their preference for searching the Internet for help. 0% 20% 40% 60% 80% 100% Tr ial & E rro r Int ern et De vic e h elp Ma nu al 20-49 50-64 65+ 0% 20% 40% 60% 80% 100% IT su pp ort Cl as s Sp ou se /P art ne r My ge ne rat ion Co -w ork ers Ch ild ren Yo un ge r g en era tio n Figure 5.5: Percentage of respondents who listed the method as one of their top three choices (N=94). The top chart shows methods for learning alone. The bottom chart shows methods for learning with others: domain experts (two methods on the left) and friends and family (five methods on the right). % o f r es po nd en ts in a ge g ro up % o f r es po nd en ts in a ge g ro up 86 Looking specifically at the four methods that support learning alone, older adults clearly reported different preferences than those reported by younger adults. Older respondents most frequently chose the device's instruction manual as one of their three preferred learning methods (see Figure 5.5, top chart), followed by the device help feature and then trial & error. In contrast, the vast majority of younger respondents chose trial & error as one of their three preferred learning methods, followed by searching the Internet. In fact, a significant effect of age on trial & error preference was found (χ2=16, df=2, p<.001); significantly fewer older respondents chose trial & error as one of their 3 preferred learning methods compared to younger respondents (Mann- Whitney U; 20-49 vs. 50-64: Z=-2.7, p=.006; 50-64 vs. 65+: Z=-1.6, p=.117). A trend result was also found in the data suggesting a possible age-effect on respondents' preference for the Internet (χ2=6.0, df=2, p=.0503) suggesting that older adults have a lower preference than do younger adults for searching the Internet for help. No significant age-related differences were found on manual and device help preferences. Analyzing the 7 learning methods that involved meeting with someone to get help (see Figure 5.5, bottom chart), a number of older respondents reported a preference for getting help from the younger generation (i.e., their children, family/friends), while no younger respondents expressed this preference. This difference was likely due to the fact that older adults' peers are likely no more informed about technology than they are, whereas younger respondents' children and family/friends from a younger generation were too young to be knowledgeable about mobile devices to help the respondents. In contrast, fewer older respondents, compared to younger ones, reported a preference for getting help from their partner/spouse and friends/family from their generation, although this difference was not significant. Our qualitative analysis presented later sheds some light on these preference differences between older and younger respondents. The above findings are based on participants' preferred learning methods (if they had access to all of them) but these findings are also supported by the responses given to the three other questions related to learning method preference. The three other questions ask participants to: rate on a six-point Likert scale how likely they would be to use a particular learning method (Q12), rate how helpful they perceived the method to be (Q14), and list their three preferred methods to help retain what they had learned (Q15). To see how well scores from the three questions support participants' three most preferred methods scores (Q13), we calculated the correlation between the participants' three most preferred methods scores and the scores from each of the three questions. We found significant correlations for almost all learning methods (see Table 5.5) suggesting that answers from these three questions support the above findings. 87 Q13. Preferred method if accessible Q14. Perceived Helpfulness Q15. Preferred method for retention Q12. Likely to use Trial & error (.534**) Internet (.254*) Device help (.450**) Manual (.515**) IT (.447**) Class (.234) Partner (.319*) My Gen. (.421**) Co-worker (.379**) Children (.317*) Young Gen. (.356**) Trial & error (.573**) Internet (.575**) Device help (.519**) Manual (.656**) IT (.463**) Class (.610**) Partner (.718**) My Gen. (.650**) Co-worker (.619**) Children (.517**) Young Gen. (.542**) Trial & error (.601**) Internet (.231*) Device help (.474**) Manual (.465**) IT (.398**) Class (.391**) Partner (.403**) My Gen. (.491**) Co-worker (.352**) Children (.276) Young Gen. (.237*) Q13. Preferred method if accessible Trial & error (.373**) Internet (.487**) Device help (.348**) Manual (.324**) IT (.303**) Class (.394**) Partner (.560**) My Gen. (.374**) Co-worker (.195) Children (.225) Young Gen. (.244*) Trial & error (.534**) Internet (.501**) Device help (.597**) Manual (.739**) IT (.514**) Class (.497**) Partner (.764**) My Gen. (.561**) Co-worker (.457**) Children (.355**) Young Gen. (.563**) Q14. Perceived Helpfulness Trial & error (.561**) Internet (.437**) Device help (.355**) Manual (.370**) IT (.344**) Class (.571**) Partner (.562**) My Gen. (.431**) Co-worker (.308**) Children (.341*) Young Gen. (.269*) ** Correlation is significant at the .01 level (2-tailed) * Correlation is significant at the .05 level (2-tailed) Table 5.5: Pair-wise correlations between responses to questions Q12, Q13, Q14 and Q15 given by novice and intermediate mobile device users from all three age groups (N=94). Learning methods with correlations that are not statistically significant are shown in light grey. In designing our questionnaire, we expected that a learning method's perceived helpfulness would affect a person's preference for using that method, but acknowledge that there might be other influential factors. To assess the effect of a method's perceived helpfulness on respondents' preference, we calculated the correlation between scores on how likely they would use a method, and scores on how helpful a method was perceived to be (see Table 5.6). When we analyzed data of younger and middle-aged respondents, we found significant correlations (p<.01) for almost all 11 learning methods, confirming our expectation that perceived helpfulness is a key reason why respondents of these two age groups chose to use a learning method. However, when we analyzed data of older respondents, we only found significant correlations (p<.05) on data for learning methods that support independent learning and that involve getting help from younger people. This finding suggests that the likelihood of older adults getting help from IT support, classes, partner/spouse, family/friends in their generation and co-workers does not depend on how helpful 88 (or unhelpful) these resources are perceived to be. As we will present next, other qualities such as cost to access a resource may affect how likely older adults are to use a learning method. Q14. Perceived Helpfulness Younger respondents Middle-aged respondents Older respondents Q12. Likely to use Trial & error (.516**) Internet (.714**) Device help (.655**) Manual (.655**) IT (.682**) Class (.531) Partner (.747**) My Gen. (.860**) Co-worker (.933**) Children (.907) Young Gen. (.818**) Trial & error (.604**) Internet (.588**) Device help (.522**) Manual (.881**) IT (.662**) Class (.868**) Partner (.896**) My Gen. (.734**) Co-worker (.623**) Children (.644**) Young Gen. (.363) Trial & error (.541**) Internet (.478*) Device help (.442**) Manual (.541**) IT (.057) Class (.437) Partner (.464) My Gen. (.364) Co-worker (-.149) Children (.460*) Young Gen. (.591**) ** Correlation is significant at the .01 level (2-tailed) * Correlation is significant at the .05 level (2-tailed) Table 5.6: Pair-wise correlations between responses on likeliness to use a learning method and perceived learning method helpfulness, by age group. Learning methods with correlations that are not statistically significant are shown in light grey. 5.3.5 Respondents' reasons for their learning method preferences Participants' qualitative responses shed additional light into the age-related differences in learning method preferences. Through our analysis of the qualitative data, we uncovered the reasons respondents most frequently gave for using and not using a particular learning method assuming they had access to it. To filter out infrequently given reasons, we only list reasons that were captured by at least 1/3 of the responses and given by at least three respondents. In our summary of the qualitative data, we include the number of respondents who gave a particular reason for using (or not using) a particular method. We also state the total number of respondents who gave reasons for using (or not using) a particular method. We show these numbers as a ratio: "(<# of respondents who gave a particular reason>/<total # of respondents who gave reasons>)." The greater the ratio, the more important a particular reason was for explaining why a method was or was not used. Approximately one half of qualitative responses did not offer a clear enough reason for using or not using a learning method, and thus were not included in our analysis; we discuss this in the Limitation section (Section 5.4.4). We summarize the qualitative result findings here. Older respondents preferred using the device's manual primarily because the manual supported their learning styles. Of all the reasons why older and middle-aged respondents use the device's manual, almost half of the responses (65+:3/6, 50-64:5/12) were related to learning style. One older respondent (age 72) wrote "I'm print oriented – [I] like to read to understand" and one middle-aged respondent (age 59) explained, "If I have directions I can usually figure it out." 89 While using the manual was the overall preferred learning method by older respondents, many respondents from all three age groups (18/31) indicated that the manual's key shortcoming was unhelpful content (e.g., not enough detail to address specific issues, not written clearly). One older respondent, age 67, wrote, "lack of detail is [the] biggest problem … too much left out … [very] frustrating." Older adults' preference for manuals over trial & error is consistent with findings by Mitzner et al. (2008) and helps to clarify the mixed findings from Selwyn et al. (2003) and Kurniawan (2006). Younger respondents preferred using trial & error because this method supported their learning style. Of all the reasons why younger respondents use trial & error, a strong majority (12/16) of the responses were related to learning style. One respondent age 25 expressed, "I like to test things out for myself", and another age 26 wrote, "You learn better from failure than success." A strong majority (14/16) of middle-aged respondents' reasons for using trial & error were also related to learning style. As stated earlier, significantly fewer older respondents preferred using trial and error. Of all the reasons older respondents gave for why they do not use trial & error, half of the responses (4/8) were related to negative past experiences, including frustration ("I get frustrated when it 'doesn't work' at once!", respondent age 72) and unwanted changes ("I … think I could 'mess [the device] up'", respondent age 74). We discuss this finding in Section 5.4.1. Respondents of all ages preferred searching the Internet for help but some older respondents do not use it because they think this method takes too much time and often does not help. The main reason why respondents from all age groups (15/32) searched the Internet for help on learning to use mobile devices was because they thought the Internet had useful information on using their device. However, a trend result noted earlier suggests that older respondents may have a lower preference for searching the Internet for help than do the younger respondents. Almost half (3/7) of older respondents' reasons for not using this learning method were related to not being able to find the information they needed ("I lose patience there are so may wrong turns to take", respondent age 71). Similarly, almost half (4/9) of middle-aged respondents' reasons for not using this learning method were related to taking too much time to find desired information ("usually too time consuming", respondent age 58). Respondents reported not using IT support or attending classes because of high access and time cost. Cost was the reason most frequently given by the three age groups for not contacting IT support (30/41) and taking a class (26/35). Regarding IT support, one younger respondent, age 28, wrote, "This [method] is a last resort for me. I hate waiting on hold" and an older respondent, age 77, wrote similarly, "[this method is] a last resort. Many IT telephones put the customer on hold for several minutes when I'm looking for an immediate answer." In a similar way, respondents from 90 all age groups expressed that taking a class required too much time and money, and that classes were often not available. Older respondents felt younger people were often not able to explain this knowledge in ways that the respondents could understand. Almost all of the reasons (9/10) older respondents gave for preferring to seek help from younger adults was because they were knowledgeable. However, the main reason many older respondents (3/6) gave for not seeking this help was because they felt younger people were not able to explain their knowledge in a helpful way. One respondent (age 71) wrote, "[family/friends from the younger generation] are most knowlegable [sic] but sometimes a bit too into it to relate to my needs." Conversely, the main reason (4/8) middle-aged respondents gave for not seeking help from younger people is because these respondents reported not knowing many younger people and the ones they did know were often not easily accessible. One middle-aged respondent, age 63, indicated that "finding [people from a younger generation is] not so easy." Younger respondents did not have comprehension or access problem, which helped to explain their relatively strong preferences for asking their partners/spouses and people from their generation for help. 5.3.6 Many older respondents make notes to use mobile devices Over a third of older respondents (12/32) and middle-aged respondents (13/34) reported using hand written notes to perform tasks on their mobile devices. A smaller proportion of younger respondents (6/28) reported using hand written notes, but this difference between age groups was not significant (Kruskal-Wallis test, χ2=2.4, df=2, p=.30). Looking at the use of notes to support computer tasks, a trend result in the data suggests that a larger proportion of older respondents reported using hand written notes to perform tasks on their computers than younger respondents did (Kruskal-Wallis test, χ2=5.9, df=2, p=.052). Inspection of the data showed nearly two thirds of older respondents (20/32) used hand written notes, whereas around a third of middle-aged respondents (14/34) and younger respondents (9/28) used notes. Almost all notes were written by the respondent (43/48) and no age-related difference was found. The vast majority of participants' notes were step-by-step text instructions (42/48), and no age-related difference was found. In contrast, an age-related difference was found in the number of hand-drawn images used in respondents' notes (Kruskal-Wallis test, χ2=10.7 df=2, p=.005). Although very few older and younger respondents who used notes (age 65+: 1/22, ages 20-49: 2/10) had hand-drawn images of the user interface on their notes, a larger proportion of middle- aged respondents who used notes used such images in their notes (8/16), which we did not expect. The difference between older and middle-aged respondents who used hand-drawn images in their notes was significant (Mann-Whitney U; 20-49 vs. 50-64: Z=-1.5, p=.134; 50-64 vs. 65+: Z=-3.2, p=.017). We expected more age-related differences in respondents' use of notes. Further studies are needed to explore older adults' use of notes in learning to use mobile devices 91 5.3.7 Older respondents would use an augmented display for learning Respondents were asked to give feedback on a hypothetical augmented display help system. The vast majority of older and middle-aged respondents (86% and 91% of responses, respectively) responded that they would try to use such a system. In sharp contrast, the majority of younger participants (65% of responses) responded that they would not, based on the conviction that they would not need such help to learn to use mobile device applications. We also asked respondents whether they thought it would be easy to operate both the mobile device and the desktop computer software. Half of the respondents felt that it would be easy, while one third of the respondents thought that this would depend on a number of factors, such as the complexity of the tasks being learned and the overall usability of the system. Responses were similar across age groups. Respondents were asked to comment on the perceived benefits and drawbacks of this hypothetical system. The ability for more control over the learning process was a key benefit; one respondent (age 71) wrote that the system would allow one to "go @ your own speed & repeat if necessary." However, some caution was expressed by older respondents about difficulties being able to communicate their questions to the described system in order to access desired learning content. 5.3.8 Findings generalize to older "beginner" mobile device users As mentioned earlier, much of our analysis was carried out on data from the respondents who reported being "novice" or "intermediate" mobile device users so that we could maximize the discovery of age-related differences. However, we were also interested in the responses from older respondents who reported being "beginner" mobile device users as they are more representative of the users we want to support (we refer to these respondents hereafter as older beginners). It was not practical to run statistical tests on responses from these beginners as the sample was relatively small (six respondents). However, we inspected the older beginners' quantitative and qualitative responses to see how consistent these responses were with our findings from older novice and intermediate mobile device users. Inspection of the responses by older beginner mobile device users revealed that these beginners generally had similar learning needs and preferences that we found with older novice and intermediate mobile device users. For example, older beginners appeared to learn as often (rarely: 40%, regularly: 40%, frequently: 20%; N=5) as older novice and intermediate mobile device users. Older beginners also felt it was very important for a learning resource to be interactive (median score: 6 out of 6), provide demonstrations (6 out of 6) and provide opportunities to practice (5 out of 6). Furthermore, older beginners' preferences for the manual over trial and error were very similar to older novice and intermediate mobile device users. Older beginners also reported a similar lack in preference for IT support due to high access and time 92 cost. One older beginner (and three middle-aged beginners) also reported that they did not phone customer or IT support because of "vocabulary challenges" (respondent, age 69), an issue not raised by older novice and intermediate mobile device users. We did find a few differences between responses from older beginners, novices and intermediate users that suggest that older beginners may need more support as they learn to perform new mobile device tasks. Older beginners reported that understanding how the software works was much less important than figuring out the exact steps required to perform a task, as shown in Figure 5.6. In addition, these older beginners felt it was just as important for learning resources to support individual learning (4 out of 6) as group learning (4 out of 6). 1 2 3 4 5 6 not important at all very important Intermediate Novice Beginner Figure 5.6: Median rating scores on the importance of learning exact task steps vs. gaining a general understanding, by older "novice" (N=16), "intermediate" (N=16), and "beginner" mobile device users (N=6). Our survey findings appear to generalize well to older beginner mobile device users. In some cases, our findings apply better to these older beginners. Older beginners appear to have less desire to learn how the technology works compared to learning how to perform tasks. The challenges older beginners have with technical terminology also underscore the need for learning resources to use familiar terms or provide better help for understanding unfamiliar terms. 5.3.9 Summary of results In summary, we found that our older respondents primarily want to learn to perform the steps of mobile device tasks and felt it was very important for learning resources to support this type of learning. Older respondents also appear to prefer using resources that allow them to learn to use mobile devices by themselves, rather than with other learners. We similarly found that our older respondents thought it was more important that learning resources provide extra guidance, such as demonstrations, opportunities to practice and more feedback. In addition, they primarily preferred to use manuals, as well as trial & error and the help provided on the device, to learn to use mobile devices. As we had hoped, we found that our older respondents were open to using an augmented display system for learning although they noted some reservations. "Figuring out the exact steps required for performing a task" "Gaining general understanding of how the software works" 93 5.4 Discussion In this section, we discuss our key survey findings related to older adults' learning needs and preferences. We then identify based on our findings two key opportunities for designing resources that will better help older adults learn to use mobile devices and other computer technology. 5.4.1 Effect of learning styles on learning method preference Respondents' learning method preference appeared to be influenced by their preference for learning alone and learning styles. Preference for learning alone We found that older adults had a stronger preference for learning alone than we expected. This finding is contrary to studies that suggest that older adults prefer learning in traditional classroom settings (Van Wynen, 2001) or with peers (Kurniawan, 2006), but is consistent with the finding by Selwyn et al. (2003) and Mitzner et al. (2008). This preference for learning independently may be due to older adults' preference to learn at their own pace (Fisk et al., 2009), and because getting help from someone, particularly IT support or a teacher, takes more time than they want to spend on getting help. Further, based on respondents' comments, it seems that many older adults do not seek help from people in their own social circle because many in their circle are less knowledgeable than they are. Influence of learning style on preference The age-related difference that we found in learning method preferences are consistent with past studies that found that learning styles differ by age group (Truluck & Courtenay, 1999; Van Wynen, 2001). Older adults have been found to become less active and hands-on, and more observant and reflective, while learning (Truluck & Courtenay). We found that younger respondents strongly preferred trial & error and searching the Internet, which are more hands-on and active than other methods. Older respondents felt it was important to have demonstrations, which promote learning through observation and reflection. Reading an instruction manual to perform a task may also promote reflection before executing a task step. Our survey data suggest that older adults do see value in using trial & error and searching the Internet, but we found that the key reason why older adults do not use these methods is because of past negative experiences using them. Many older adults have expressed past frustration when using trial & error. Although the exact cause of frustration was not identified in our study, past research has found that older adults can more easily get lost when navigating mobile device UIs (Ziefle & Bay, 2004) and they are more negatively affected by errors (Birdi & Zapf, 1997). Many of our older respondents expressed that they cannot find answers they are looking for on the Internet. We suspect that older adults could benefit more from trial & error and searching the 94 Internet if they had better support (e.g., through the web browser, online help) to find the information they need and to reduce errors. 5.4.2 Help resources preferred over training Respondents from all three age groups did not want to use formal training programs (e.g., in- person classes, structured online courses) or training materials. Alternatives to traditional training and teaching resources are help resources, which primarily provide on-demand help to enable learners to perform new tasks (Duffy, Palmer, & Mehlenbacher, 1992). Although classes and courses on learning to use desktop computers and programs are common, help resources (e.g., online help, Minimal Manual [Carroll, Smith-Kerker, Ford, & Mazur-Rimetz, 1987]) may be more suitable for helping older as well as younger adults learn to use mobile devices. Mobile user interfaces and tasks are generally less complex than those on desktop computers, so formal training may require too high of a time cost for the expected benefit. Alternatively, help resources such as instruction manuals and Internet content offering troubleshooting advice are more conveniently available and may offer the user enough support to perform tasks. Further, mobile device UIs differ much more across devices and mobile applications than do desktop computers, making it harder to create a course that will help people learn to use different devices. 5.4.3 Designing better help resources Based on our survey findings, we think there are two key opportunities for designing resources that will better help older adults learn to use mobile devices and other computer technology. The first opportunity is to provide more support for trial & error. Trial & error is different from the other learning methods because it is self-directed, exploratory and users learn to use the user interface primarily from interacting with the interface and learning from both successful and failed task attempts. While respondents from all age groups saw value in this method, many older respondents reported that they are unlikely to use trial & error because it often did not help them to successfully complete a task and could lead them to make unwanted changes to the device, thus causing frustration. Providing additional support while older adults use trial & error may make the learning experience more positive. To support trial & error, we propose an exploratory mode, to encourage users to try out new tasks or practice tasks without worrying about inadvertently changing the device settings or data. This exploratory mode might allow learners to try out mobile tasks and have the option to save their changes or to return to the state that the device was in before entering the exploratory mode. We include this feature in the design of the Help Kiosk system that we present in the next chapter. Related, a second opportunity is to provide more support for searching the Internet for help. This method requires finding helpful information in the sea of web content. Success depends on whether such content exists and the Internet search skills of the user. Older adults, particularly novice Internet users, have been found to take much longer and take more steps to find desired 95 web content than do younger adults (Hanson, 2010; Meyer, Sit, Spaulding, Mead, & Walker, 1997). Respondents from all age groups saw value in this method, but many older respondents reported that they would not search the Internet for help because this method was time consuming for them and they often got lost, which led to impatience and frustration. Providing extra support to older adults while they search the Internet may improve their learning experience. For example, a portal for help information related to their device might help older beginners more quickly find online information they are looking for and minimize opportunities for getting lost. In addition, user-provided or domain-expert ratings on solutions that have worked for others could help learners identify which solutions to try first. Another opportunity for designing better help and learning resources for older adults is to improve the content of existing resources. In particular, the content in some instruction manuals was criticized by respondents, both younger and older ones. Respondents reported that they often found manuals "poorly written", used unfamiliar terminology, targeted a level of technology experience higher than theirs, and lacked details they were interested in. While much work has looked at designing paper manuals (e.g., Carroll et al., 1987) that addresses these issues, the increasing pervasiveness of online manuals and eBooks provides an opportunity to provide dynamic content information that offers personalized and additional help on content not clear to the user. For example, a dynamic manual could consist of different versions of content targeting different technology experience levels. Such a manual could also provide definitions of terms unfamiliar to the user, and more details if the user wanted more help. The Help Kiosk system that we present in the next chapter offers such support in an effort to help users interpret unclear manual instructions. 5.4.4 Limitations One limitation of our survey is that our younger age group appeared to have more mobile device expertise than did the two older age groups. Thus, the age-related differences we found in our survey results might be in part attributed to differences in mobile device expertise. Another limitation of our study is that participants were current mobile device users and were self-selected (i.e., not randomly chosen). The findings may not capture the needs and preferences of people who have had so much difficulty learning to use mobile devices that they were unable to become users. Participants, especially those that filled out the online questionnaire, may have had more computer experience and interest in technology than does the general population. We expect that our findings will generalize somewhat to the general population, but more work is needed to confirm this. We also acknowledge that self-reported data can be different from actual behaviour. Our survey produced many rich findings that we could use, along with guidelines from the literature, to design and prototype our augmented display help system. Because of the novelty of the augmented display approach, we decided to quickly move forward with designing, prototyping and evaluating 96 the augmented display help system without spending more effort gathering additional data. However, more observational data is still needed to confirm and build on our self-reported-based findings. One of the known limitations inherent in survey studies where participants complete the survey without a researcher being present is that researchers have less control on participants completing the surveys fully. This is a cost associated with administering a survey to a large population. This limitation applied to our questionnaire survey. Respondents who filled out paper surveys often skipped questions that asked for free-form answers, perhaps because of the effort involved. To minimize this inherent drawback, we configured our online questionnaires to force participants to enter data into all required fields, including free-form text fields. Although we found that a number of respondents typed one or two characters into the text field (i.e., did not answer the question) in order to move onto the next question, we expect that forcing participants to answer the question helped increase the number of complete responses. 5.5 Summary Our study findings supported many existing guidelines for the design of training, and helped us to prioritize them for the next step in our research. Our findings also helped us determine which features and guidelines are more beneficial for helping older adults learn to use mobile devices. In summary, we found that our older respondents primarily want to learn to perform the steps of mobile device tasks and to learn to use mobile devices independently. We also found that our older respondents thought it was more important that learning resources provide extra guidance, such as demonstrations, opportunities to practice and more feedback. In addition, they primarily preferred to use manuals, as well as trial & error and the help provided on the device, to learn to use mobile devices, and this preference appeared to be partly influenced by individual learning styles. Help systems appeared to be preferred over formal learning programs. Based on our survey results, there appear to be opportunities to help older adults learn to use mobile devices by supporting them in trial & error and searching the Internet for help content, as well as improving device manuals, which we considered in the design of our augmented display system. Further, our older respondents appeared open to using the proposed augmented display system for learning, and offered comments and suggestions about it. We used these findings, along with related design guidelines from the literature, to inform our augmented display help system design, which we present in the next chapter. 97 Chapter 6 6. Augmenting the Mobile Device's Display: Design and an Informal User Study The previous chapter described a survey study that we conducted to better understand the learning needs and preferences of older adults and inform the design of an augmented display help system that we prototyped. In this chapter, we present the second part of our investigation of using augmented displays to help older adults to learn to use mobile devices. After identifying the objectives and research methodology, we present our design for an augmented display help system called Help Kiosk. We describe through a use scenario how we imagine Help Kiosk would help an older person perform a common mobile device task. We then describe five principles we incorporated into the design of Help Kiosk, as well as details of a prototype implementation. We then present the methods and results of an informal user study that we conducted to obtain early feedback on the design direction of Help Kiosk. In this user study, six participants were asked to learn to perform common tasks on a smart phone using either the Help Kiosk or the phone's instruction manual. We discuss preliminary evidence that Help Kiosk can support older adults while they learn to perform new mobile device tasks. 6.1 Objectives and methodology As discussed in the Introduction chapter, our investigation into using an additional display to help older adults learn to use mobile devices was motivated by the limitations of the mobile devices' small screens and their increasingly powerful technical capabilities. A device's small screen limits the amount of interactive visual help that can be shown to its user. For example, it is difficult to simultaneously show on the same device screen the application that is being learned and the related help information. However, mobile devices can now easily connect to other devices and 98 displays, offering an opportunity to overcome the limitations of the small screen, and build novel resources to support learning. This second part of our investigation of augmented displays had two goals. The first goal was to design an augmented display system to help older adults learn to use mobile devices, based on our survey findings and guidelines from the literature. To meet this goal, we aimed to identify design principles to guide the design. We chose to design and prototype Help Kiosk to support Google's Nexus One smart phone, one of the more advanced commercial touchscreen mobile devices that was available in 2010 when the research described in this chapter took place. The Nexus One was chosen because it offered multiple ways to connect to other devices such as another computer and the computer's monitor (e.g., USB, Wi-Fi, Bluetooth). This smart phone was also chosen because it runs the Android OS, which has many software development tools that allow a computer to retrieve and process the smart phone's state and log entries in real-time. We chose to support smart phones in the Help Kiosk prototype because these phones are one of the most pervasive types of mobile devices and many older adults have reported wanting to use smart phones (Ofcom, 2008). A second goal was to gain early feedback on the design concept of our first prototype. It was unclear at the outset of this study whether older adults could benefit from the extra guidance and feedback from the additional display given the increased complexity of having to operate two devices and needing to cognitively process more information. In addition, it was unclear whether our design decisions, which were based on our survey findings and existing guidelines, would lead to features that would work well together in a system. Thus, after our first prototype was developed, we conducted a small informal user study to gather preliminary data on the design approach and determine next steps, before committing to further prototype development and a larger, more formal user study. 6.2 Help Kiosk design Help Kiosk is an augmented display system consisting of a desktop computer and a touchscreen monitor to help older adults learn to use smart phones. The Help Kiosk design is based on guidelines from the literature and informed by findings from our survey. We first describe a scenario for using Help Kiosk, and then we present the five design principles we followed. 6.2.1 Use scenario Mary, age 70, is retired and lives alone. She wants to use her new smart phone to wake her up in the morning but is not sure how to do this. She positions her phone close to her desktop computer and is automatically prompted by the phone with a "Launch Help Kiosk?" dialog. She accepts, and the Help Kiosk software quickly launches on her desktop display. Mary then accesses the desired help instructions on the desktop display through Help Kiosk's context-aware browse and search 99 feature that suggests a number of topics related to the applications available on her specific model of phone (see Figure 6.1, callout a). Mary selects "Alarm" and Help Kiosk lists a number of related tasks. Alternatively, Mary could have searched on any words she had in mind, such as "alert" (words would be entered at Figure 6.1, callout b). Mary then selects "Setting an Alarm" from this list (the list would be shown at Figure 6.1, callout c), which directs her to a page that lists the instructions (see Figure 6.1, center screenshot). She performs Step 1 easily (opening the clock application by touching its icon on her phone), because she has done this before, but then she gets stuck on Step 2. Help Kiosk offers Mary different types of support, in addition to text instructions, to help her learn how to perform the step. Mary reads the instruction for Step 2, "Touch the Alarm icon", but is unsure where to find the icon because, for some reason, she is looking for the label "Alarm." She touches Step 2 on Help Kiosk, highlighting its text instructions (Figure 6.1, callout d), to get more help on performing this step. This does two things. First, it annotates Live View, a real-time screen capture of Mary's phone, highlighting (with an orange rectangle) the specific phone UI control for performing the step (Figure 6.1, callout e). In this case, Mary looks at Live View and learns which specific icon to touch on her phone. Second, it queues up a demonstration video that shows how to perform the step and the expected outcome (Figure 6.1, callout f). Mary now thinks she knows how to perform the step based on Live View but touches the Demo Video area to play a 10 second video just to make sure. Help Kiosk can also offer Mary additional instructions on performing sub-steps of more complex steps (see Figure 6.1, bottom screenshot). Mary performs Step 2 and 3 and then reaches Step 4, which simply instructs her to set the alarm attributes. She can watch a demonstration video to see an example of someone setting various attributes, but she is most interested in learning how to set the days of the week on which she wants the alarm to sound. Help Kiosk offers more support for this step, indicated by the presence of a "Show More Help" button (Figure 6.1, callout g). She presses this button and the More Help panel appears, listing various sub-steps phrased as questions (Figure 6.1, callout h). Mary touches the question she is interested in and reads the help she needs. Mary wants to perform one more sub-step, labelling this new alarm "weekday mornings", but is unfamiliar with the on-screen keyboard and is concerned that she might make accidental and unwanted changes to her previously entered alarm information. To address this concern, she switches to Help Kiosk's exploratory mode (toggle button not shown in Figure 6.1) to attempt to finish the step. She makes a few mistakes while trying to type the label but eventually learns how to complete the task properly. At this point, she exits exploratory mode (re-hitting the toggle), which gives her the option to revert the device back to its state prior to entering exploratory mode or to apply all changes made while in exploratory mode. Given that in the end she did enter the label correctly, she accepts the changes and touches the Done button to complete the task. 100 Figure 6.1: Help Kiosk Find Info pane (top) and Learn/Do pane for adding an alarm task (Step 2, middle; Step 4, bottom). Labelled callouts a-g are described in the text (Use scenario section). f a b c d e g h 101 Mary has successfully created her desired alarm. She moves her phone away from her desktop computer, which minimizes the Help Kiosk program on the desktop computer, and she continues on with the rest of her day. 6.2.2 Design principles Help Kiosk is a type of supportive scaffolding that helps users during the learning process with the goal that users will eventually be able to use their mobile phone without this support. It is intended to run on any desktop computer (e.g., at home, at a library) or laptop, making it convenient to access at most times, which we found to be important to both older and younger adults. The Help Kiosk design is based on the concept of an instruction manual, found to be the learning method most preferred by older adults, but it provides additional real-time interactive guidance and feedback to help the user understand the text instructions if they need help. We followed five principles, many building on existing guidelines, when designing the Help Kiosk. 1. Support realistic learning contexts and self-directed learning Help Kiosk supports realistic learning contexts by having learners perform tasks directly on the smart phone itself, instead of on an emulated version. As Fisk, Rogers, Charness, Czaja, and Sharit (2009) recommend, we wanted users to gain experience and confidence in interacting with the actual device, so that this knowledge, confidence and perhaps muscle memory would be better retained when the phone is used without the scaffolding. Help Kiosk puts learners in control over the learning process. Users are able to repeat instructions as often as they want. Unless an instruction step is dependent on an earlier one, Help Kiosk allows users to skip ahead to subsequent steps and go back one or more steps. 2. Utilize real-time device state to personalize experience Help Kiosk keeps track of the smart phone's state in real-time and uses this contextual information to personalize the user's learning experience. One way Help Kiosk uses the phone's state is to keep track of what the learner is doing and whether or not the learner has correctly followed the instructions, which previous studies found to be difficult for new users (Carroll, Smith-Kerker, Ford, & Mazur-Rimetz, 1987). Another way Help Kiosk uses the phone's state is to help the user more quickly find desired help through context-aware browse and search. (This browse and search feature has not been implemented in the prototype, but we envision that initial versions will look as shown in the top pane in Figure 6.1.) 3. Provide both generic and specific instructions Although our survey showed that older adults prefer using a device's manual when learning to use the device, the survey results also revealed that many older adults find manual instructions difficult to understand (e.g., content assumes an inappropriate level of expertise with the 102 technology). Help Kiosk presents each instruction step in three distinct ways, differing in media format to accommodate a wider variety of learning styles (e.g., visual, verbal) and levels of details specific to the learner's device to accommodate learners of varying expertise: 1. Text instructions prescribe the steps needed to perform a task. Content not directly related to performing the main task steps, such as secondary information to help users with potentially problematic sub-steps, is put aside in the More Help panel, to be accessed only if desired. 2. Live View shows the screen contents of the learner's device in real-time to highlight the specific UI elements the learner should use to perform a particular step. A similar technique, highlighting actual physical controls, has been used in past Augmented Reality systems (e.g., Bonanni, Lee, & Selker, 2005). 3. Demonstration videos show how a step is performed, and the expected outcome. 4. Offer opportunities to explore and practice safely Older users often express concerns that they might break the device or cause some unwanted change to settings or stored data. Help Kiosk allows users to enter an exploratory mode in which they can try out mobile tasks and have the option to save their changes or return to the state that the device was in before entering the exploratory mode. (Exploratory mode has not yet been implemented. We envision having users perform tasks on the larger display in this mode to remind and assure them that their actions do not affect their device.) 5. Minimize demands on working memory We sought to minimize demands on working memory, as recommended by Fisk et al. (2009). For example, Help Kiosk UI elements remain as much as possible in the same screen location during a session to reduce the demands on users' visuo-spatial working memory. Using a smart phone and Help Kiosk together requires going back and forth between two displays, which may be difficult for older adults because it requires remembering instructions presented on the larger display to perform task steps on the device, and visually swapping between the two displays, which might be situated at different viewing distances from the user (Hawthorn, 2005). Help Kiosk's design addresses this potential difficulty by showing the mobile device's screen contents on the larger display through Live View to reduce the need for visual swapping between the device and the large screen, and using larger font sizes on the larger display to make it easier to swap when needed. 6.2.3 Prototype Our current Help Kiosk prototype consists of a 19" touchscreen monitor (ELO 1938L IntelliTouch, which is designed for public kiosks) and a desktop computer running a version of the Windows operating system. We chose a computer monitor for the prototype because Help Kiosk is 103 intended to run on a desktop computer. We chose a touch-based monitor because interacting through touch appeared to be suitable for our design and is an increasingly popular feature in computer monitors and other devices. Help Kiosk can also be used with a mouse instead of touching the monitor directly. The prototype can connect to an Android smart phone via USB. It currently supports Google Nexus One (Android OS 2.2) touchscreen smart phones. Other than enabling the USB debugging setting on the smart phone, no additional software needed to be installed on the device to make it work with Help Kiosk. The Help Kiosk software was written in Java and uses adb (Android Development Bridge), a software tool provided by Google, to collect the smart phone's system information (e.g., what application screens are showing) by monitoring its device logs. Help Kiosk uses the open source Droid@Screen to continuously capture the device screen (at a rate of 1 frame/sec, which we found to be adequate). The prototype currently supports most of the Help Kiosk features mentioned in the scenario except for the context-aware browse and search UI, and the exploratory mode. The prototype offers help for three tasks: take photo, add contact entry, and set an alarm. 6.3 User study methods As stated earlier, we designed and conducted an informal user study to gather preliminary data on the design approach and determine next steps, before committing to further prototype development and a larger formal user study. Our study focused primarily on assessing how well Help Kiosk assists older adults in performing unfamiliar smart phone tasks. To help us get the most of this study, some effort was made to structure the study; for example, it includes a comparison between Help Kiosk and an instruction manual, and participants were prescribed a series of tasks to perform. However, with only six participants and minimal control over balancing the design features between interfaces, the results should be interpreted as very preliminary. 6.3.1 Participants We focused on recruiting adults from our survey respondents who reported being beginner or novice mobile device users, as we expected these users would most benefit from using Help Kiosk. For expediency we recruited a total of six adults, who were either older or middle-aged (see Table 6.1). No participants had previously used a touchscreen smart phone. ID P1 P2 P3 P4 P5 P6 Age 75 72 69 67 56 56 Gender male male female female female female Mobile expertise beginner novice beginner beginner beginner novice Computer expertise intermediate intermediate novice intermediate novice intermediate Table 6.1: Participant characteristics (N=6). 104 6.3.2 Materials Conditions To evaluate the Help Kiosk design, we chose to informally compare it to the device's printed manual (Google, 2010), which is representative of a comprehensive manual written for a wide variety of users. An online PDF version of the manual was offered but no participants used it. The same text instruction wording, icon images and step order from the manual were used in Help Kiosk to promote a fair comparison. Core procedural information in Help Kiosk is separated from the other information, which was put into the More Help panel (Figure 6.2 shows text from the manual and Help Kiosk). Thus, both Help Kiosk and the manual offered the same text context, but presented it differently. Half of the participants learned Task A using Help Kiosk and Task B using a printed version of the manual, while the other half of the participants had the opposite task-to-scaffold mapping. We also balanced the order of assigned tasks: half of the participants were asked to perform Task A first while the other half were asked to start with Task B. Tasks Participants were asked to perform two types of tasks. Task A involved adding a new contact to the Contacts application. Task B involved changing the setting of an existing alarm in the Clock application. Participants were asked to carry out three sequential attempts for each task type, two with a scaffold and one without. Specifically, in Attempts 1 and 2, participants had access to a scaffold to help them learn to perform the assigned task, to simulate the learning process. In Attempt 3, participants were asked to perform the assigned task without a scaffold, to simulate a condition where participants had to perform tasks that they spent time learning with a scaffold. The task given in Attempt 1 was relatively simple, while tasks given in Attempts 2 and 3 were a bit more complex (see Appendix D.7). Attempts 2 and 3 were very similar expect for variations in task- related information (e.g., contact name and phone number, alarm time). Measures An experimenter manually recorded whether attempts were successfully completed. Some participants had difficulty entering text using the on-screen keyboard due to physical limitations (e.g., difficulties targeting a key, slight trembling) rather than not knowing how to complete the task, so we considered a task to be complete when participants performed all required steps, even if they did not enter text values exactly as prescribed. 105 Figure 6.2: Text instructions used in Help Kiosk (bottom) were adapted from the phone's manual (top). The wording and step order were largely preserved. Core procedural information in Help Kiosk is separated from the other information, which was put into the More Help panel (as shown in the bottom pane in Figure 6.1). Labelled callout a is referred to in the text (Discussion section). a 106 Think-aloud comments during the attempts were recorded by a video camera and captured through experimenter's notes. We also collected participants' opinions about the usefulness of various Help Kiosk features and perceived workload (six NASA-TLX subscales on a six-point scale) when using each of the scaffolds. We also asked participants specific interview questions about using Help Kiosk to learn, such as how much the demo videos and highlighted suggestions in LiveView helped (Appendix D.4 lists the interview questions). 6.3.3 Procedure All study sessions took place in a usability lab on the university campus and were recorded using a video camera. Each session was conducted by two experimenters. Each participant took part individually in a study session. In a study session, participants were first given a demonstration on how to perform a task (taking a photo) on the smart phone and were given an opportunity to repeat the task to familiarize themselves with the touch interactions on the device. Participants were then asked to perform Task A and Task B in an assigned order, each using a different scaffold (Attempt 1). They were then asked to perform another attempt of Task A and B in the same order, using the same assigned scaffolds as before (Attempt 2). Participants were not given any help by the experimenter unless they were clearly stuck or lost for a few minutes. Participants then went through a short semi- structured interview on their experiences using Help Kiosk and the manual, and also filled out a background questionnaire. Participants were then asked to perform Task A and B again in the same order but without any scaffold (Attempt 3). Participants were then given another semi- structure interview. Appendix D.5 contains the script for introducing the Help Kiosk and manual, instructions for participants, and the background questionnaire used in the study. The study took one and a half to two hours. Participants were each given an honorarium. 6.4 User study results We first present the number of task attempts the Help Kiosk and manual helped our participants to complete, followed by participants' perceived workload in using each of the two scaffolds. 6.4.1 Help Kiosk appeared to help participants perform new tasks The results of our informal evaluation suggest that Help Kiosk aided participants in learning how to perform new tasks, perhaps better than the manual. Only one participant missed completing the first attempt with Help Kiosk, and by the second attempt, all participants were able to complete all tasks using this scaffold (see Table 6.2). When participants used the manual, they were not able to complete as many tasks; by the second attempt, half the participants could still not complete the tasks (although P2 curiously completed the task on the first attempt but not on the second). By the 107 third attempt, when participants performed tasks without any scaffolds, only one participant failed to complete a task. The manual had been used to learn that task. Attempt Condition ID Order 1 2 3 Help Kiosk Task A P4 1st P1 2nd • P5 1st Task B P2 1st P3 2nd P6 1st Manual Task A P2 2nd P3 1st • • • P6 2nd Task B P4 2nd P1 1st • P5 2nd • • Table 6.2: Participants' success in completing tasks with Help Kiosk and manual scaffolds (Attempts 1,2) and without either scaffold (Attempt 3). =completed, •=not completed. 6.4.2 Help Kiosk appeared to be easy to use Our six participants reported a lower perceived workload when using the Help Kiosk compared to when using the manual (see Figure 6.3, left). Participants also reported finding it easy to use Help Kiosk at the same time as operating a smart phone (median score: 4.5 out of 6; 6=very easy to use), dispelling our initial concerns that they might find this task challenging. Participants generally felt that both the Live View highlighting and demonstration videos helped them to follow the instructions (median scores: 4.5 and 4.5 out of 6, respectively). Overall, participants felt that it was much easier to follow instructions on Help Kiosk than in the manual (see Figure 6.3, right). As one participant explained, "it takes more effort to interpret [the manual than Help Kiosk]" (P2). 1 2 3 4 5 6 mental physical temporal performance effort frustration Pe rc ei ve d w or kl oa d Help Kiosk Manual 1 2 3 4 5 6 H ow e as y to fo llo w in st ru ct io ns Figure 6.3: Participants' median scores on perceived workload (left; 1=very low, 6=very high, N=6) and ease in following instructions (right; 1=very difficult, 6=very easy, N=6). 108 6.5 Discussion In this section, we discuss the findings from our informal user study that provide preliminary evidence that Help Kiosk helped our participants perform new smart phone tasks. We discuss what aspect of the design seemed to provide a learning benefit, and the generalizability of the Help Kiosk design to other contexts. 6.5.1 Help Kiosk may help some older adults perform new tasks Our informal evaluation of Help Kiosk found preliminary evidence that an augmented display system incorporating a number of our design principles can help older adults learn how to perform new tasks on a smart phone. Our six participants completed more tasks using Help Kiosk than with the phone's manual during the two initial learning attempts. Participants found both the Live View highlighting and the demonstration videos useful for learning. Overall, participants indicated that it was much easier to follow instructions on Help Kiosk than in the manual. At the end of the study, our participants, who had very little prior experience with smart phones and some had even expressed apprehension about using these devices, generally reported being encouraged that they were able to learn to use a smart phone. 6.5.2 Successful initial attempts are crucial Our informal user study showed a noticeable difference in task completion between the two scaffold conditions in the first two attempts: one incomplete attempt out of twelve attempts for the Help Kiosk relative to five such attempts out of twelve attempts for the manual. The fact that by the third attempt almost all participants were able to perform tasks learned with either Help Kiosk or the manual is of less importance, because this similarity is likely due to ceiling effects. We argue that success in the earlier attempts has a larger effect on device adoption because errors have been found to affect older adults more negatively (Birdi & Zapf, 1997) than other users. Some of our participants expressed that they are willing to try using new mobile phones and other technologies, but will quit if they are not making progress within a short amount of time. 6.5.3 Allowing older adults to access instructions in a variety of ways helps them to learn We found some evidence that Help Kiosk's learning support is related to the design principles that we incorporated. Help Kiosk's presentation of text instructions were designed to help users focus on instructions to help them perform task steps and hide secondary information, thus reducing demands on their working memory, one of our design principles. Although the text instructions used in Help Kiosk were sourced from the phone's manual and the wording and step order were largely preserved, Help Kiosk separates core procedural information from the other information, which was put into the More Help panel. A number of participants were frustrated when using the phone's manual because they felt they were wading through content that did not directly help them 109 perform the task at hand. One participant noted, "[the manual is] talking about unrelated stuff I don't want" (P2). Help Kiosk users could focus on the steps for completing the task at hand and could access optional help if they needed, from the More Help panel, as well as the Live View and Demo Video panes. In addition, our user study provided evidence that giving older adults access to instructions presented in different media formats and levels of details is helpful because older adults (and other users) have different learning styles and technology experiences. A small number of text instructions, present in both Help Kiosk and the manual, were unclear and were difficult for participants to interpret without assistance. An example of these instructions was "Press Menu <options menu icon> and touch New contact" (see Figure 6.2, callout a). Some participants had difficulty because they looked for the text label "Menu" on the device UI, which did not exist, and they did not look for the icon embedded in the text instructions. Participants using the manual spent several minutes stuck trying to interpret this text instruction. In contrast, participants using Help Kiosk made use of the suggestions offered through the Live View highlighting and demonstration videos to determine which UI controls to interact with. As a result they did not get stuck on this step. One participant stated: "I found [the manual] a bit incomplete. I was looking for [more] guidance" (P1). 6.5.4 Generalizability of Help Kiosk design We think that Help Kiosk can be easily extended beyond self-directed learning to support more directed materials, such as interactive tutorials or courses. Moving further beyond support for learning a smart phone, one can imagine other contexts where augmented displays might support learning. For example, at an attraction such as a museum or art gallery, visitors could use a specialized application (perhaps wirelessly installed on their mobile device upon entry) to explore the space and artefacts in the attraction. Having users temporarily stand beside a wall display or kiosk that provides the screen space for an interactive tutorial on how the new application works on their device could be quite effective. Augmented display systems that use very large wall size displays may also be suitable for training a group of learners to use new mobile devices and applications, such as salespeople and customer support workers. Determining which learning components from Help Kiosk (text, video, and Live View) would apply in a group setting and how to compose them for this context would be interesting for future work. 6.6 Summary We used guidelines from the literature, informed by our survey results (Chapter 5), to design an augmented display help system, Help Kiosk, following five principles. Our informal evaluation with six older adults provided encouraging preliminary evidence that Help Kiosk can support older adults in learning to perform new smart phone tasks. Help Kiosk shows promise as a system 110 to help older adults learn to use smart phones and other devices, which may lead to increased adoption by older adults and benefits in their daily lives. Further development and evaluation of the system appear to be warranted. 111 Chapter 7 7. Conclusions Mobile devices offer many benefits to users but older adults have difficulties learning to use current devices. The goal of the research reported in this dissertation was to investigate a number of design approaches to assess how they could improve the learnability of mobile devices for older adults. We investigated three promising design approaches that have not been well-explored in the literature: increasing the interpretability of graphical icons, incorporating a multi-layered interface, and augmenting the mobile device's display. To investigate the first approach, we conducted a qualitative exploratory study and a follow-up experiment with older and younger adults to determine which icon characteristics affect initial icon interpretability for older adults. To investigate the second approach, we conducted an experiment with older and younger adults to study the effects of learning on a multi-layered mobile application over four distinct learning phases. To investigate the third approach, we first conducted a survey study with older and younger adults to research the needs and preferences of older adults when learning to use mobile devices. We then designed and prototyped an augmented display help system, Help Kiosk, and conducted an informal user study with older adults to assess whether Help Kiosk can help older users learn to use a smart phone. In this chapter, we summarize the contributions of the research presented in this dissertation. We then reflect briefly on our overall research strategy and the generalizability of our results. Finally, we suggest some directions for future work. 7.1 Primary research contributions We claim four primary contributions that arose from our design approach investigations. 112 7.1.1 Identified the effect of concreteness, semantic distance, and labels on initial icon interpretability for older adults Prior to our research, we suspected that older adults, compared to younger adults, had more difficulties interpreting existing mobile device icons, but little was reported in the research literature about the icon usability issues that older adults experience, beyond size (Siek, Rogers, & Connelly, 2005), and colour and contrast (Hawthorn, 2000). Building on past work that examined the effect of concreteness (McDougall, de Bruijin, & Curry, 2000) and labels (Wiedenbeck, 1999) on initial icon interpretability, we conducted an experiment to examine the effect of these two icon characteristics, as well as semantic distance, on icon interpretability. We also explored age-related effects of these three icon characteristics on icon interpretability. To increase the ecological validity of our results, our study used existing icons from commercially-available mobile devices (e.g., Apple iPhone, Blackberry 7730) and presented other contextual information such as an image of the device and a description of the application the icon was used in. Results from our experiment with 36 participants confirmed our hypothesis that older adults had more difficulties than did younger adults when using existing mobile device icons. This finding further motivated our work to identify types of icons that were easier for older adults to use. We found that, out of the three characteristics we studied, semantic distance had the largest effect on icon interpretability for older adults. We developed a number of guidelines, including ones for reducing semantic distance, to help practitioners choose or design icons that are easier for older adults to correctly interpret. 7.1.2 Demonstrated that multi-layered interfaces offer learnability benefits to older adults Past studies (Dickinson, Smith, Arnott, Newell, & Hill, 2007; Hawthorn, 2005) showed some evidence that multi-layered interfaces can help older adults learn to perform tasks on a new UI, but those studies focused on desktop applications and did not involve controlled experiments. We conducted an experiment with 32 participants to compare a two-layer ML interface with a Full- Functionality control interface to explore age-related differences in performance and preferences. We studied ML interfaces in a number of new ways by focusing on a mobile device application instead of traditional desktop computer programs, and by comparing performance across four stages of learning that are directly relevant to ML interfaces. To increase the ecological validity of our results, we prototyped the ML interface on a mobile device instead of a mobile UI simulated on a desktop computer. The latter is common in previous mobile HCI research, which is a limitation we wanted to avoid. We found that a ML interface's Reduced-Functionality layer, compared to the control interface's Full-Functionality layer, helped older participants more than younger ones to perform the first basic task attempts in less time. When both older and younger participants transitioned from the Reduced-Functionality to the Full-Functionality interface layer, their performance on the 113 previously learned tasks was negatively affected, but no negative effect was found on learning new, advanced tasks. Older participants, compared to our younger participants, preferred the ML interface over the Full-Functionality control interface for learning to use the application, and perceived the ML interface to be less complex than the control interface. We thus found that ML interfaces offer a number of learnability benefits to older adults. We discussed some of the design issues related to incorporating ML interfaces into mobile applications. 7.1.3 Gained insights into older adults' needs and preferences in learning to use mobile devices Recent studies have investigated which resources older adults prefer when learning how to use technology (Kurniawan, 2006; Mitzner et al., 2008; Selwyn, Gorard, Furlong, & Madden, 2003), but the findings have been mixed in terms of which learning methods older adults generally prefer to use (e.g., alone vs. with others, trial & error, manual). Further, these studies did not include data from younger adults to ground their findings. We created a comprehensive questionnaire to survey mobile device users about their needs and preferences in learning to use their devices. This questionnaire included many open-ended questions to capture underlying reasons for respondents' reported needs and preferences. We analyzed data from 94 survey respondents across three age groups to uncover age-related differences in learning needs and preferences. The survey results gave insights into the learning needs and preferences of older adults. In terms of learning needs, we found that older adults reportedly needed to relearn how to perform previously learned tasks more frequently than did younger adults. We also found that it was more important for older adults to learn the exact steps of a mobile device task than to gain an overall understanding of how the software worked. In terms of learning preferences, we found that older adults want learning resources to provide demonstrations and opportunities to practice, and to enable them to learn individually rather than in a group. The results also showed that older adults generally prefer using manuals while younger adults prefer learning by trial & error; both groups attributed their preferences to the learning method fitting their learning style. Such results may be useful for instructional designers (Mayer, 2003), as well as improving technical support for mobile phone or medical devices. 7.1.4 Designed an augmented display system that can support older adults in learning to use a mobile device We used the findings from our survey study, as well as guidelines from the literature, to design and prototype Help Kiosk, an augmented display system to assist older adults in learning to use a smart phone. To our knowledge, we are the first to propose augmenting a device's small display with a larger display for the purposes of learning to use the mobile device. We designed Help Kiosk according to five design principles. An informal evaluation of our system with six older adults offers preliminary evidence that Help Kiosk is easy to follow and can assist older adults in 114 learning how to perform new smart phone tasks. We suggested other ways that an augmented display system can support learning as topics for future work. 7.2 Secondary contributions We claim two secondary contributions related to our overall research strategy and methodology. 7.2.1 Took a multi-pronged strategy to addressing main research problem To our knowledge, little work has examined how to improve the learnability of mobile devices or other computer technology for older adults, beyond improving training resources and programs and improving the initial usability of some UI elements. We investigated three design approaches that have not been well-explored and found evidence that they can each be used to improve the learnability of mobile device UIs for older adults. By choosing three different design approaches, we were better able to explore the design space and understand which approach might be more appropriate to take in a particular situation. 7.2.2 Systematically included both older and younger adults to uncover age-related differences Previous aging and technology research often only involved older participants, and not younger ones. This practice may sometimes make it difficult to determine whether findings from these studies can be attributed to participants' age or generation, and not other factors such as culture, social-economic status, or computer experience. We recruited younger adults for our studies to ground the older participants' data and help tease apart the age-related differences from other differences. Within the limits of practicality, we recruited groups of older and younger participants from the same community who were similar in computer and mobile device experience. Analyzing both older and younger participants allowed us to determine age-related effects. For example, our icon interpretability experiment helped us determine that the presence of labels increases the initial interpretability of icons for older adults, as well as for younger adults (i.e., a main effect of label but not an effect of age), but that icons with depicted objects semantically closer to the icon's function are significantly easier for older adults to use, than for younger adults to use (i.e., age-related effect). Without including younger adults in the experiment, there might have been a risk of incorrectly concluding that labels only help older adults in using icons and incorrectly attributing the effect to labels meeting some age-related characteristic. 7.3 Reflections on overall research strategy and findings In this section, we take a step back and reflect on the overall strategy we took for carrying out the research presented in this dissertation. We also reflect on whether our findings are related to age- or generational-related differences. 115 7.3.1 Breadth- vs. depth-based research strategy We decided at the outset of our research to take a breadth-based strategy to meet our main goals. Rather than choosing one design approach and conducting a series of studies to iteratively investigate just one approach, we chose three different design approaches that appeared to be promising. We reflect here on the benefits and drawbacks of following this research strategy. As expected, one of the benefits of a breadth-based strategy is that it enabled us to study and assess different design approaches. Investigating three approaches increased the chances of finding at least one approach that would help older adults learn to use mobile devices. Further, our breadth-based strategy allowed us to gain experience in many different design spaces (e.g., icons, ML interfaces, multi-display systems). The main drawback to our breadth-first strategy was that investigating three different design approaches increased the number of specific research areas that we needed to familiarize ourselves with and reduced the opportunities to conduct follow-up studies. We spent much time and effort researching background information and related past work for each design approach. Thus, we tended to design a large study for each of our three investigations, which we felt was more time efficient than conducting multiple smaller studies. Although conducting a single larger study seemed more time efficient, conducting multiple smaller studies may have allowed more reuse of study materials and methods, reducing risk for technological and study design problems in subsequent studies. 7.3.2 Age-related vs. generation-related findings In any cross-sectional research that compares older adults with younger adults, it is important to understand whether differences found between the two age groups are generational- or age-related. Generational- (or cohort-) related differences are attributed to differences in growing up and living in different times. Age-related differences are attributed to internal changes to one's abilities that are caused by aging. It is possible, but not a certainty, that today's younger adults who are more experienced with today's computers will be able to leverage this experience as older adults and learn new computer technology with greater ease than do today's older adults. However, research has shown that as people age, they become less open to new experiences (less likely to try new things) (Fisk, Rogers, Charness, Czaja, & Sharit, 2009), and become more averse to risk (more afraid of making errors) (Birdi & Zapf, 1997). We believe that tomorrow's older adults will show similar personality changes and experience similar declines in cognitive functioning as experienced by today's older adults. Thus, the need for better learning resources is likely to endure. 116 7.4 Future work There are a number of potentially fruitful avenues to continue investigating our three design approaches. 7.4.1 Longitudinal user studies on graphical icon interpretability Having explored initial icon interpretability, future work should look at the effects of icon characteristics on the interpretability of these icons over multiple exposures or over long periods of time (e.g., weeks or months). Older users probably interact less frequently with their devices than do younger users, decreasing the frequency of contact with icons, and making it more important to design icons whose meanings are easy to remember as well as learn. McDougall et al. (2000) found that, for their university student participants, abstract icons tended to become as usable as concrete icons over multiple exposures. Wiedenbeck (1999) found that undergraduate students had more problems retaining the meaning of unlabelled icons over a one-week period, compared with labelled icons and labels without icons (i.e. label-only), but that these difficulties diminished over further exposures. We are interested in seeing whether such learning effects can also be found in older populations. On the basis of the literature and our findings, we speculate that there are a number of age- related differences in icon interpretability over time. Older adults have been found to have more difficulty learning new associations (Chalfonte & Johnson; 1996, Naveh-Benjamin, Guez, & Shulman, 2004), which may make it more difficult to learn unfamiliar abstract icons or those with semantically further meanings. In addition, the recall of associations learned in old age has been found to be harder, especially in multitasking situations (Hawthorn, 2000), which are common in mobile contexts. Older adults may also have more difficulty than do younger adults in retaining learned meanings of unlabelled icons and may continue to rely on icon labels longer than younger adults. We speculate that concrete icons, which were found to help older adults to identify more icon objects, may be more effective as cues for recalling associations and be easier for older adults to use over time. More research is needed to validate all of these speculations. 7.4.2 Longitudinal studies on multi-layered interface Future work is also needed to study the use of a ML mobile application over time. First we would need to add to our ML address book the ability to switch between layers because our study did not examine the added complexity in ML interfaces of switching between layers. A longitudinal study could help us to understand why and how often users switch between layers, and measure the effect of the switching overhead on an application's learnability and long-term use. Comments made by our older participants suggest that they would primarily use the mobile address book in one layer and not switch much between layers. Research is needed to confirm whether this 117 generally holds true for this and other ML mobile applications. A longitudinal study could also be used to explore the effects of personalizing the Reduced-Functionality layer on longer term use. 7.4.3 Incorporating multi-layered interfaces in more complex mobile applications As stated in Chapter 4, the ML mobile application evaluated in our study was relatively simple. One area of future work is to evaluate the ML interface approach in the context of learning to use different types of mobile applications, particularly those that are unfamiliar to older adults. As stated earlier, existing mobile applications are generally more complex than the one we studied; they have more features (e.g., home screen with many different icons and widgets), offer a wide variety of interaction methods (e.g., multitouch gestures for navigation and interacting with content), and they present more detailed, composite content (e.g., in a navigation application enabled by Global Positioning System). Future work is needed to study how ML interfaces can be used to layer functions or content for different types of existing mobile applications and how to evaluate the effectiveness of ML interfaces on an application's learnability. Alternatively, one could also evaluate the effect of using ML interfaces to learn to use mobile applications that involve concepts and tasks that users are not familiar with (e.g., social networking tools). We expect that ML mobile interfaces with an initial reduced-functionality layer will greatly help older adults learn to use unfamiliar and more complex mobile applications. 7.4.4 Continued development and evaluation of augmented display help system Our next steps in investigating the use of augmented displays to support learning include prototyping more Help Kiosk features (e.g., exploratory mode, browse and search) and refining the UI design and text instructions. To our knowledge, the exploratory mode (i.e., safe practice) concept has not been applied, beyond the confines of an interactive tutorial, to mobile device interfaces nor used to help older adults learn to use technology. We plan to further develop Help Kiosk to support more tasks and operate across multiple devices, offering suggestions personalized for each device and its content (e.g., Live View would highlight appropriate UI element(s) wherever the UI places them on the screen). After adding more features and more task support to Help Kiosk, the next step would be to more formally evaluate the system. Future work in this direction includes conducting a controlled experiment, based on our existing informal user study, to compare Help Kiosk to the device's instructional manual. Future work could also include a longitudinal field study to see how Help Kiosk is actually used in everyday situations (e.g., home, community centre, library). A field study could also be conducted to observe how older adults actually learn to use mobile devices, to confirm and extend our survey findings. 118 7.4.5 Investigate other promising design approaches There are other design approaches, beyond the three we chose, that appear promising for improving the learnability of mobile devices for older adults. For example, one design approach we considered but did not choose to investigate was incorporating more supportive error messages into the early stages of the learning process (Birdi & Zapf, 1997). This is an example of motivational scaffolding. Error messages could be redesigned to tell users what actions are needed to recover from the error and perform the desired task, and also to direct users to more information (e.g., in a specific section in a manual). There are likely other design approaches that are also worth investigating. 7.5 Concluding comments The global population is growing older, and many older adults want to use existing and future mobile computing devices. Unfortunately, most devices are not designed for older adults and are increasing in complexity, making it even more difficult for older adults to learn how to use them. By investigating three promising design approaches, we gained insights into how to improve the learnability of mobile devices for older adults. We found evidence that each design approach can be used to improve the device's learnability for older adults. Our hope is that researchers will continue to investigate these and other approaches, and that technology developers will use them to build mobile technologies that are easier for older adults to learn how to use and adopt into their daily lives. 119 References Adams, D. (1999). How to stop worrying and learn to love the Internet. Retrieved April 10, 2011, from Douglas Adams site: http://www.douglasadams.com/dna/19990901-00-a.html Associated Press. (2011, February 8). Smart phones outsell PCs for the first time. CBS News. Retrieved July 11, 2011, from: http://www.cbsnews.com/stories/2011/02/08/scitech/main20031095.shtml Baecker, R., Small, I., & Mander, R. (1991). Bringing icons to life. Proceedings of the SIGCHI conference on human factors in computing systems (CHI), 1-6. doi:10.1145/108844.108845 Benson, C., Clark, B., & Nickell, S. (2010). Designing effective icons. GNOME human interface guidelines 2.2.1. Retrieved March 4, 2011, from: http://library.gnome.org/devel/hig-book/stable/icons-design.html.en Benton, A. L., & Hamsher, K. (1978). Multilingual aphasia examination manual. University of Iowa. Birdi, K. S., & Zapf, D. (1997). Age differences in reactions to errors in computer-based work. Behaviour & Information Technology, 16(6), 309-319. Blair, J. R., & Spreen. O. (1989). Predicting premorbid IQ: A revision of the National Adult Reading Test. The Clinical Neuropsychologist, 3(2), 129-136. Bonanni, L., Lee, C.H., & Selker, T. (2005). CounterIntelligence: Augmented reality kitchen. Extended Abstracts of CHI, 2239-2245. Carroll, J. M., & Carrithers, C. (1984). Training wheels in a user interface. Communications of the ACM, 27(8), 800-806. Carroll, J. M., Smith-Kerker, P., Ford, J., & Mazur-Rimetz, S. (1987). The Minimal Manual. Human-Computer Interaction, 3(2), 123-153. Catrambone, R., & Carroll, J. M. (1986). Learning a word processing system with training wheels and guided exploration. SIGCHI Bulletin, 17(SI), 169-174. Chalfonte, B. L., & Johnson, M. K. (1996). Feature memory and binding in young and older adults. Memory & Cognition, 24(4), 403-416. Cohen, J. (1988). Statistical power analysis for the behavioural sciences. 2nd ed. Hillsdale, NJ: Erlbaum. 120 Czaja, S. J. (1997). Computer technology and the older adult. In M. Helander, T. K. Landauer, & P. Prabhu (Eds.), Handbook of human-computer interaction (pp. 797-812). New York, NY: Elsevier Science Inc.. Czaja, S. J., & Lee, C. C. (2007). The impact of aging on access to technology. Universal Access in the Information Society, 5(4), 341-349. doi: 10.1007/s10209-006-0060-x DeYoung, C. G., & Spence, I. (2004). Profiling information technology users: En route to dynamic personalization. Computers in Human Behavior, 20(1), 55-65. doi:10.1016/S0747-5632(03)00045-1 Dickinson, A., Arnott, J. L., & Prior, S. (2007). Methods for human - computer interaction research with older people. Behaviour & Information Technology, 26(4), 343-352. doi: 10.1080/01449290601176948 Dickinson, A., Smith, M. J., Arnott, J. L., Newell, A. F., & Hill, R. L. (2007). Approaches to web search and navigation for older computer novices. Proceedings of CHI, 281-290. doi:10.1145/1240624.1240670 Docampo Rama, M., de Ridder, H., & Bouma, H. (2001). Technology generation and age in using layered user interfaces. Gerontechnology, 1(1), 25-40. Doro. (2010). Doro HandleEasy 330gsm US. Retrieved February 8, 2011, from http://www.dorousa.us/Our-products/Cell-phones-and-accessories/330GSM-US/ Duffy, T. M., Palmer, J. E., & Mehlenbacher, B. (1992). Online help: Design and evaluation. Norwood, NJ: Ablex Publishing. Echt, K. V., Morrell, R. W., & Park, D. C. (1998). Effects of age and training formats on basic computer skill acquisition in older adults. Educational Gerontology, 24(1), 3-25. Findlater, L., & McGrenere, J. (2007). Evaluating reduced-functionality interfaces according to feature findability and awareness. Proceedings of IFIP TC13 International Conference on Human-Computer Interaction (INTERACT), 592-605. Fisk, A. D., Rogers, W. A., Charness, N., Czaja, S. J., & Sharit, J. (2009). Designing for older adults: Principles and creative human factors approaches (2nd ed.). London: CRC Press. Fozard, J. L., Bouma, H., Franco, A., & van Bronswijk, J. E. M. H. (2009). Homo Ludens: Adult creativity and quality of life. Gerontechnology, 8(4), 187-196. Retrieved March 25, 2011, from http://www.gerontechnology.info/Journal/pdf.php?a=1171 Freudenthal, T. D. (1998). Learning to use interactive devices: Age differences in the reasoning process (Doctoral dissertation). Eindhoven University of Technology, Eindhoven, Netherlands. Goodman, J., Brewster, S., & Gray, P. (2004). Older people, mobile devices and navigation. Proceedings of HCI and the Older Population Workshop at British HCI, 13-14. Retrieved 121 March 25, 2011, from http://www.dcs.gla.ac.uk/~stephen/research/utopia/workshop/goodman.pdf Google. (2010). Nexus One user guide. Retrieved March 25, 2011, from http://www.google.com/googlephone/NexusOneOwnersGuide.pdf Greenberg, S., Boyle, M., & Laberge, J. (1999). PDAs and shared public displays: Making personal information public, and public information personal. Personal and Ubiquitous Computing, 3(1-2), 54-64. doi:10.1007/BF01305320 Gregor, P., Newell, A. F., & Zajicek, M. (2002). Designing for dynamic diversity: Interfaces for older people. Proceedings of the 5th International ACM Conference on Assistive Technologies (ASSETS), 151-156. doi:10.1145/638249.638277 Grossman, T., Fitzmaurice, G., & Attar, R. (2009). A survey of software learnability: Metrics, methodologies and guidelines. Proceedings of CHI, 649-658. doi: 10.1145/1518701.1518803 Gus Communications. (2009). Gus! Pocket Communicator. Retrieved June 1, 2009, from http://www.gusinc.com/pocketcomm.html Hanson, V. L. (2010). Influencing technology adoption by older adults. Interacting with Computers, 22(6), 502-509. doi:10.1016/j.intcom.2010.09.001 Hanson, V. L., & Crayne, S. (2005). Personalization of Web browsing: adaptations to meet the needs of older adults. Universal Access in the Information Society, 4(1), 46-58. doi: 10.1007/s10209-005-0110-9 Hawthorn, D. (2000). Possible implications of aging for interface designers. Interacting with Computers, 12(5), 507-528. Hawthorn, D. (2005). Training wheels for older users. Proceedings of the 19th conference of the computer-human interaction special interest group (CHISIG) of Australia on Computer- human interaction, 1-10. Heim, S. (2007). Icons. The resonant interface: HCI foundations for interaction design (pp. 407- 451). Boston, MA: Addison Wesley. Hickman, J. M, Rogers, W. A., & Fisk, A. D. (2007). Training older adults to use new technology. The Journals of Gerontology, 62B(Special II), 77-84. Horton, W. (1994). The icon book: Visual symbols for computer systems and documentation. New York, NY: John Wiley & Sons. Inglis, E. A., Szymkowiak, A., Gregor, P., Newell, A. F., Hine, N., Shah, P., Wilson, B. A., & Evans, J. (2003). Issues surrounding the user-centred development of a new interactive memory aid. Universal Access in the Information Society, 2(3), 226-234. doi: 10.1007/s10209-003-0057-7 Jackson, S. L., Krajcik, J., & Soloway, E. (1998). The design of guided learner-adaptable scaffolding in interactive learning environments. Proceedings of CHI, 187-194. 122 Ju, W., Bonanni, L., Fletcher, R., Hurwitz, R., Judd, T., Post, R., Reynolds, M., & Yoon, J. (2002). Origami desk: Integrating technological innovation and human-centric design. Proceedings of the 4th Conference on Designing Interactive Systems (DIS), 399-405. Kelley C. L., & Charness N. (1995) Issues in training older adults to use computers. Behaviour and Information Technology, 14(2), 107-120. Kurniawan, S. (2006). An exploratory study of how older women use mobile phones. Proceedings of Ubicomp, 105-122. Kurniawan, S. (2007). Mobile phone design for older persons. Interactions, 14(4), 24-25. Kurniawan, S., Mahmud, M., & Nugroho, Y. (2006). A study of the use of mobile phones by older persons. Extended Abstracts of CHI, 989-994. Landauer, T. K. (1997). Behavioral research methods in human-computer interaction. In: M. G. Helander, T. K. Landauer, & P. V. Prabhu (Eds.), Handbook of Human-Computer Interaction, 2nd ed. (pp. 203-227). Amsterdam: Elsevier Science B.V.. Lepper, M. R., Woolverton, M., Mumme, D. L., & Gurtner, J. (1993). Motivational techniques of expert human tutors: Lessons for the design of computer-based tutors. In S. P. Lajoie & S. J. Derry (Eds.), Computers as cognitive tools (pp. 75-105). Hillsdale: Lawrence Erlbaum Associates. Li, H., & Graf, P. (2007). Cognitive, perceptual, sensory and verbal abilities as predictors of PDA text entry error and instructions across the lifespan. Proceedings of the 7th International Conference on Engineering Psychology and Cognitive Ergonomics, 349-358. Lindberg, T., & Nasanen, R. (2003). The effect of icon spacing and size on the speed of icon processing in the human visual system. Displays, 24(3), 111-120. Lingraphicare America. (2009). Lingraphica product overview. Retrieved June 1, 2009, from Lingraphicare America Web site: http://www.aphasia.com/patients/product.aspx Luchini, K., Bobrowsky, W., Curtis, M., Quintana, C., & Soloway, E. (2002). Supporting learning in context: Extending learner-centered design to the development of handheld educational software. Proceedings of the IEEE International Workshop on Wireless and Mobile Technologies in Education, 107-111. Luchini, K., Quintana, C., & Soloway, E. (2004). Design guidelines for learner-centered handheld tools. Proceedings of CHI, 135-142. Marmor, M. F. (1998). Normal age-related vision changes and their effects on vision. Aging & Vision News [online]. Retrieved March 27, 2008, from: http://www.lighthouse.org/education-services/professional-education/patient- management/managing-the-patient-with-low-vision/normal/ Massimi, M., Baecker, M., & Wu, M. (2007). Using participatory activities with seniors to critique, build, and evaluate mobile phones. Proceedings of ASSETS, 155-162. 123 Mayer, R. (2003). Elements of a science of e-learning. Journal of Educational Computing Research, 29(3), 297-313. McDougall, S. J., Curry, M. B., & de Bruijn, O. (1999). Measuring symbol and icon characteristics: Norms for concreteness, complexity, meaningfulness, familiarity, and semantic distance for 239 symbols. Behavior Research Methods, Instruments, & Computers, 31(3), 487-519. McDougall, S. J., de Bruijn, O., & Curry, M. B. (2000). Exploring the effects of icon characteristics on user performance: The role of icon concreteness, complexity, and distinctiveness. Journal of Experimental Psychology Applied, 6(4), 291-306. McGrenere, J., Baecker, R. M., & Booth, K. S. (2002). An evaluation of a multiple interface design solution for bloated software. Proceedings of CHI, 164-170. McKnight, P. E., McKnight, K. M., Sidani, S., & Figueredo, A. J. (2007). Missing data: A gentle introduction. New York, NY: Guilford Press. Meyer, B., Sit, R. A., Spaulding, V. A., Mead, S. E., & Walker, N. (1997). Age group differences in World Wide Web navigation. Extended Abstracts of CHI, 295-296. Milner, B. (1971). Interhemispheric differences in the localization of psychological processes in man. British Medical Bulletin, 27(3), 272–277. Mitzner, T. L., Fausset, C. B., Boron, J. B., Adams, A. E., Dijkstra, K., Lee, C. C., Rogers, W. A., & Fisk, A. D. (2008). Older adults' training preferences for learning to use technology. Proceedings of Annual Meeting of Human Factors and Ergonomics Society, 2047–2051. Moffatt, K. A. (2010). Addressing age-related pen-based target acquisition difficulties (Doctoral dissertation). University of British Columbia, Vancouver, Canada. Morrell, R. W., Park D. C., Mayhorn, C. B., & Kelly, C. L. (2000). Effects of age and instructions on teaching older adults to use Eldercomm, an electronic bulletin board system. Educational Gerontology, 26(3), 221-235. Naveh-Benjamin, M., Guez, J., & Shulman, S. (2004). Older adults' associative deficit in episodic memory: Assessing the role of decline in attentional resources. Psychonomic Bulletin & Review, 11(6), 1067-1073. Nielsen, J. (1993). Usability engineering. San Francisco, CA: Morgan Kaufmann Publishers Inc.. Norman, D. (1988). The design of everyday things, New York, NY: Currency Doubleday. Novick, D. G., and Ward, K. (2006). Why don't people read the manual? Proceedings of ACM International Conference on Design of Communication (SIGDOC), 11-18. NTT DoCoMo. (n.d.). RakuRaku phone basic. Retrieved February 8, 2011, from http://www.nttdocomo.com/features/diverse_needs/raku.html NTT DoCoMo. (2001). Mobile phones increasingly popular among the elderly (Report No.11), NTT DoCoMo, Toyko, Japan. 124 Ofcom. (2006). Media literacy audit: Report on media literacy amongst older people. Ofcom, London, UK. Ofcom. (2008). Media literacy audit: Report on media literacy amongst older people. Ofcom, London, UK. Ofcom. (2009). Media literacy audit - digital lifestyles: Adults aged 60 and over. Ofcom, London, UK. Peirce, C. S. (1932). The collected papers of Charles Sanders Peirce. Cambridge, MA: University Press. Quintana, C., Carra, A., Krajcik, J., & Soloway, E. (2002). Learner-centered design: Reflections and new directions. In Carroll, J. (ed.) Human-computer interaction in the new millennium (pp. 605-626). New York, NY: Addison-Wesley. Rieman, J. (1996). A field study of exploratory learning strategies. ACM Transactions on Computer-Human Interaction, 3(3), 189-218. Rogers, W. A., Cabrera, E. F., Walker, N., Gilbert, D. K., & Fisk, A. D. (1996). A survey of automatic teller machine usage across the adult lifespan. Human Factors, 38(1), 156-166. Samsung. (n.d.) Samsung Jitterbug J cell phone. Retrieved February 8, 2011 from http://www.samsung.com/us/mobile/cell-phones/SPH-A310AWADYN Santos, P. J., & Badre, A. N. (1995). Discount learnability evaluation (GVU Technical Report GIT-GVU-95-30). Georgia Institute of Technology. Scheffer, J. (2002). Dealing with missing data. Research Letters in the Information and Mathematical Sciences, 3, 153-160. Selwyn, N., Gorard, S., Furlong, J., & Madden, L. (2003). Older adults' use of information and communications technology in everyday life. Ageing and Society, 23(5), 561-582. Shneiderman, B. (2003). Promoting universal usability with multi-layer interface design. Proceedings of the 2003 Conference on Universal Usability, 1-8. Siek, K. A., Rogers, Y., and Connelly, K. H. (2005). Fat finger worries: How older and younger users physically interact with PDAs. Proceedings of INTERACT, 267-280. Sixsmith, A. J., Orpwood, R. D., & Torrington., J. M. (2010). Developing a music player for people with dementia. Gerontechnology, 9(3), 421-427. doi:10.4017/gt.2010.09.03.004.00 Smith, A. (2010). Mobile Access 2010, Pew Internet. Soloway, E., Jackson, S. L., Klein, J., Quintana, C., Reed, J., Spitulnik, J., Stratford, S. J., Studer, S., Eng, J., & Scala, N. (1996). Learning theory in practice: Case studies of learner- centered design. Proceedings of CHI, 189-196. Sun Microsystems. (1999). Design guidelines: Application graphics. Retrieved January 8, 2011, from: http://java.sun.com/products/jlf/ed1/dg/high.htm 125 Truluck, J. E., & Courtenay, B. C. (1999) Learning style preferences among older adults. Educational Gerontology, 25(3), 221-236. University of British Columbia. (2010). Survey tool description - UBC Information Technology. Retrieved April 12, 2011 from University of British Columbia web site: http://www.it.ubc.ca/service_catalogue/social_media_collaboration/survey_tool/survey_d escription.html Van Wynen, E. A. (2001). A key to successful aging: Learning-style patterns of older adults. Journal of Gerontological Nursing, 27(9), 6-15. Wechsler, D. (1981). WAIS-R manual: Wechsler Adult Intelligence Scale-Revised. The Psychological Corporation, New York. Wiedenbeck, S. (1999). The use of icons and labels in an end user application program: an empirical study of learning and retention. Behaviour and Information Technology, 18(2), 68-82. Ziefle, M., & Bay, S. (2004). Mental models of a cellular phone menu: Comparing older and younger novice users. Proceedings of 6th International Conference on Human Computer Interaction with Mobile Devices (MobileHCI), 25-37. Ziefle, M., & Bay, S. (2005). How older adults meet complexity: Aging effects on the usability of different mobile phones. Behaviour and Information Technology, 24(5), 375-389. Ziefle, M., Schroeder, U., Strenk, J., & Michel, T. (2007). How younger and older adults master the usage of hyperlinks in small screen devices. Proceedings of CHI, 307-316. 126 Appendices Appendix A Graphical icon experiment materials and a portion of the raw results 127 A.1 Call for participation Advertisements published in Vancouver Courier 128 Poster 129 A.2 Consent form 130 131 A.3 Phone pre-screening questions 1. How old are you? (desired answer: 20-39 or 65+) 2. You are asked to take part in two studies. The studies take a total of 3-4 hours, with some breaks. Are you able to participate? (desired answer: yes) 3. The two studies take place at UBC, in the computer science building. Can you make it here? (desired answer: yes) 4. Do you wear glasses? Yes, for seeing far away Yes, for seeing up close No 5. Are you able to read the text in a book or newspaper? in dim light? (desired answer: yes) Yes No 6. Are you able to make out different colours, especially blue, green, and red? (desired answer: yes) Yes No 7. How much experience do you have with computers? How often do you a computer these days? What do you use the computer for? (e.g., email, word processing, surfing Internet) (desired response: much computer experience) 8. How much experience do you have with cell phones? If you have a cell phone, what do you use your cell phone for? Taking photos? Text Messaging? Surfing the Internet? (desired response: some cell phone usage but little smart phone experience) 9. How much experience do you have with PDAs/handheld computers? (e.g., Palm Pilot, Black Berry, iPAQ) (desired response: little PDA/handheld computer experience) 10. What is your mother tongue? (desired response: fluent in English) 11. How fluent are you in Spoken English? (ask only if person has not said much or if English is not good) 132 A.4 Interview questions Background 1. What is your highest level of education achieved? No formal education Less than high school graduate High school graduate/GED Vocational training Some college/Associate's degree Bachelor's degree (BA, BS) Master's degree (or other post-graduate training) Doctoral degree (PhD, MD, EdD, DDS, JD, etc.) 2. (If over 65 years old) Do you have a caregiver who takes care of you at home (e.g., family, employed caregiver)? Yes No 3. What is your occupational status? Student Work full-time Work part-time Homemaker Retired If working, what is your primary occupation? If retired: what was your primary occupation? what year did you retire? 133 A.5 Three sets of 20 icons used in experiment Icons were taken from screen captures, which were posted on the Internet, of the following mobile devices: Sony Ericsson W610i and W850i, Blackberry 7730, Nokia N95, HP iPaq rx3715, Palm Treo 650, and Apple iPhone. Icons and their associated labels and devices are shown below. Set A 134 Set A (continued) 135 Set B 136 Set B (continued) Set C 137 Set C (continued) 138 A.6 Steps used to control for familiarity data in icon interpretation data analysis In this section, we describe in detail the steps we took to control for icon familiarity ratings in analyzing participants' icon interpretation scores (icon object identification accuracy, interpretation accuracy, and interpretation confidence). The purpose of this section is to provide guidance to future students who are interested in performing the same analysis that we did. What we did and describe here is equivalent to conducting an ANCOVA. However, the ANCOVA methods available in SPSS require using the same covariate for all variables that are entered into the analysis. By contrast, we had repeated measures on 3 independent variables, and a different covariate score (icon familiarity rating) for each of them. To accommodate the different familiarity ratings, we removed the influence of familiarity from each variable by means of a regression analysis, saved the residuals and examined those by means of an ANOVA. We describe how we did this using SPSS 17.0. We also present example data to further illustrate the steps. Example: Assume we have the following sample of icon interpretation accuracy data for two participants shown in Table A.1. Our experimental design has three within-subjects factors, which combine to form 12 experimental conditions (2 levels x 2 levels x 3 levels). Each participant produces a dependent measure score (interpretation accuracy) and a familiarity score for each of the 12 experimental conditions. Icon-only Icon+Label Label-Onl CC CF AF AC CC CF AF AC CC PID Fam Int Fam Int Fam Int Fam Int Fam Int Fam Int Fam Int Fam Int Fam Int 2 7.0 0.7 4.0 0.0 2.0 0.0 7.0 1.0 7.0 0.8 4.0 0.5 1.5 0.7 4.0 1.0 6.0 0.8 3 8.5 0.7 4.5 0.3 1.0 0.0 1.5 0.5 7.5 0.8 6.5 0.8 6.0 1.0 7.0 1.0 8.0 1.0 Table A.1: Sample icon interpretation accuracy data for two participants in wide format (i.e., one participant per row). Step 1. Create a new spreadsheet file and added the following four headings to the top of the spreadsheet: • Participant ID • Experimental condition label • Covariate score • Dependent measure score Example: We create a new spreadsheet and add four headings as shown in Table A.2. PID Condition Covar Score Table A.2: New spreadsheet with four headings. 139 Step 2. Copy repeated measures data into the new spreadsheet, arranging the data in a long format (i.e., one row per dependent variable score) under the four headings. Come up with a unique label for each experimental condition to populate the Condition column. Example: We copy the above example data into our new spreadsheet file, as shown in Table A.3. PID Condition Covar Score 2 IOCC 7.0 0.7 3 IOCC 8.5 0.7 2 IOCF 4.0 0.0 3 IOCF 4.5 0.3 2 IOAF 2.0 0.0 3 IOAF 1.0 0.0 2 IOAC 7.0 1.0 3 IOAC 1.5 0.5 2 ILCC 7.0 0.8 3 ILCC 7.5 0.8 Table A.3: Sample icon interpretation accuracy data, in long format (i.e., one row per dependent variable score). Step 3. Import the data in SPSS and calculate unstandardized residuals for the dependent measure scores, controlling for familiarity. In SPSS 17.0: • Select Analyze->General Linear Model->Univariate • Enter dependent variable name in Dependent Variable text field • Enter covariate name in Covariate(s) text field • Press Save… button and check the Residuals: Unstandardized check box. Then press the Continue button, and then the OK button. Example: We perform the above steps and then SPSS generates a new column of residual data as shown in Table A.4. PID Condition Covar Score RES_1 2 IOCC 7.0 0.7 -0.14 3 IOCC 8.5 0.7 -0.19 2 IOCF 4.0 0.0 -0.69 3 IOCF 4.5 0.3 -0.45 2 IOAF 2.0 0.0 -0.61 3 IOAF 1.0 0.0 -0.57 2 IOAC 7.0 1.0 0.20 3 IOAC 1.5 0.5 -0.09 2 ILCC 7.0 0.8 0.03 3 ILCC 7.5 0.8 0.01 Table A.4: Sample icon interpretation accuracy data in long format, with generated residuals data. 140 Step 4. Copy residuals data into SPSS and arrange residual scores in a wide format. Example: We copied over the residuals data into SPSS as shown in Table A.5. PID AgeGrp IOCC IOCF IOAF IOAC ILCC ILCF ILAF ILAC ILCC ILCF ILAF ILAC 2 1 -0.14 -0.69 -0.61 0.20 0.03 -0.19 0.08 0.31 0.07 -0.30 -0.17 0.18 3 1 -0.19 -0.45 -0.57 -0.09 0.01 0.03 0.24 0.20 0.16 -0.05 0.16 0.16 Table A.5: Residuals for sample icon interpretation accuracy data, arranged in wide format. Step 5. Run repeated-measures ANOVA as usual on residual scores. Example: We ran an ANOVA on our residuals data and identified significant effects of our independent variables on icon interpretation accuracy, controlling for familiarity. 141 A.7 ANOVA tables with and without icon familiarity controlled Icon identification accuracy Tests of within-subjects effects ANOVA ANOVA with familiarity- controlled scores Source SS df MS F p ηp2 SS df MS F p ηp2 label .251 1 .251 15.388 .000 .312 .171 1 .171 10.641 .003 .238 label * AgeGrp .039 1 .039 2.366 .133 .065 .032 1 .032 2.020 .164 .056 Error(label) .554 34 .016 .546 34 .016 CC .118 1 .118 6.989 .012 .171 .087 1 .087 4.559 .040 .118 CC * AgeGrp .099 1 .099 5.843 .021 .147 .144 1 .144 7.528 .010 .181 Error(CC) .575 34 .017 .651 34 .019 SD .486 1 .486 31.769 .000 .483 .234 1 .234 15.761 .000 .317 SD * AgeGrp .139 1 .139 9.100 .005 .211 .215 1 .215 14.508 .001 .299 Error(SD) .520 34 .015 .504 34 .015 label * CC .031 1 .031 .855 .362 .025 .047 1 .047 1.287 .264 .036 label * CC * AgeGrp .008 1 .008 .214 .647 .006 .009 1 .009 .242 .626 .007 Error(label*CC) 1.242 34 .037 1.247 34 .037 label * SD .014 1 .014 .825 .370 .024 .022 1 .022 1.375 .249 .039 label * SD * AgeGrp .008 1 .008 .464 .500 .013 .002 1 .002 .101 .753 .003 Error(label*SD) .572 34 .017 .547 34 .016 CC * SD .204 1 .204 13.045 .001 .277 .131 1 .131 8.099 .007 .192 CC * SD * AgeGrp .132 1 .132 8.440 .006 .199 .116 1 .116 7.189 .011 .175 Error(CC*SD) .532 34 .016 .551 34 .016 label * CC * SD .001 1 .001 .109 .743 .003 .003 1 .003 .288 .595 .008 label * CC * SD * AgeGrp .010 1 .010 1.211 .279 .034 .018 1 .018 1.934 .173 .054 Error(label*CC*SD) .271 34 .008 .315 34 .009 Tests of between-subjects effects ANOVA ANOVA with familiarity- controlled scores Source SS df MS F p ηp2 SS df MS F p ηp2 Intercept 237.14 1 237.136 10000 .000 .997 .000 1 .000 .000 1.000 .000 AgeGrp .697 1 .697 29.387 .000 .464 .358 1 .358 14.862 .000 .304 Error .806 34 .024 .820 34 .024 142 Icon interpretation accuracy Tests of within-subjects effects ANOVA ANOVA with familiarity-controlled scores Source SS df MS F p ηp2 SS df MS F p ηp2 label 11.37 2 5.685 192.91 .000 .850 7.962 2 3.981 91.427 .000 .729 label * AgeGrp .009 2 .005 .159 .854 .005 .131 2 .065 1.500 .230 .042 Error(label) 2.004 68 .029 2.961 68 .044 CC .130 1 .130 5.507 .025 .139 .270 1 .270 8.475 .006 .200 CC * AgeGrp .013 1 .013 .533 .470 .015 .056 1 .056 1.768 .193 .049 Error(CC) .804 34 .024 1.084 34 .032 SD 11.18 1 11.18 279.63 .000 .892 7.418 1 7.418 163.204 .000 .828 SD * AgeGrp .476 1 .476 11.893 .002 .259 .730 1 .730 16.057 .000 .321 Error(SD) 1.360 34 .040 1.545 34 .045 label * CC .160 2 .080 3.261 .044 .088 .224 2 .112 4.468 .015 .116 label * CC * AgeGrp .051 2 .025 1.028 .363 .029 .066 2 .033 1.307 .277 .037 Error(label*CC) 1.672 68 .025 1.704 68 .025 label * SD 1.512 2 .756 29.760 .000 .467 1.646 2 .823 22.523 .000 .398 label * SD * AgeGrp .199 2 .099 3.911 .025 .103 .077 2 .038 1.051 .355 .030 Error(label*SD) 1.728 68 .025 2.484 68 .037 CC * SD .371 1 .371 23.282 .000 .406 .096 1 .096 4.021 .053 .106 CC * SD * AgeGrp .005 1 .005 .326 .571 .010 .002 1 .002 .087 .770 .003 Error(CC*SD) .542 34 .016 .808 34 .024 label * CC * SD .082 2 .041 1.409 .251 .040 .046 2 .023 .900 .411 .026 label * CC * SD * AgeGrp .016 2 .008 .280 .757 .008 .051 2 .025 .998 .374 .029 Error(label*CC*SD) 1.979 68 .029 1.735 68 .026 Tests of between-subjects effects ANOVA ANOVA with familiarity-controlled scores Source SS df MS F p ηp2 SS df MS F p ηp2 Intercept 196.47 1 196.47 2475.684 .000 .986 .000 1 .000 .000 1.000 .000 AgeGrp 2.153 1 2.153 27.134 .000 .444 .336 1 .336 4.180 .049 .109 Error 2.698 34 .079 2.73 34 .080 143 Interpretation confidence Tests of within-subjects effects ANOVA ANOVA with familiarity-controlled scores Source SS df MS F p ηp2 SS df MS F p ηp2 label 15.608 2 7.804 26.078 .000 .703 10.488 2 5.244 15.359 .000 .583 label * AgeGrp 1.150 2 .575 1.922 .170 .149 .319 2 .159 .467 .633 .041 Error(label) 6.583 22 .299 7.511 22 .341 CC .014 1 .014 .103 .755 .009 .003 1 .003 .023 .881 .002 CC * AgeGrp .003 1 .003 .022 .885 .002 .005 1 .005 .040 .846 .004 Error(CC) 1.535 11 .140 1.364 11 .124 SD 12.740 1 12.740 70.383 .000 .865 8.338 1 8.338 45.849 .000 .807 SD * AgeGrp .218 1 .218 1.204 .296 .099 .158 1 .158 .868 .371 .073 Error(SD) 1.991 11 .181 2.000 11 .182 label * CC .284 2 .142 .746 .486 .064 .119 2 .059 .444 .647 .039 label * CC * AgeGrp .326 2 .163 .857 .438 .072 .214 2 .107 .799 .462 .068 Error(label*CC) 4.183 22 .190 2.938 22 .134 label * SD 3.300 2 1.650 10.080 .001 .478 3.972 2 1.986 10.087 .001 .478 label * SD * AgeGrp .388 2 .194 1.184 .325 .097 .202 2 .101 .513 .606 .045 Error(label*SD) 3.601 22 .164 4.332 22 .197 CC * SD .077 1 .077 .493 .497 .043 .001 1 .001 .008 .931 .001 CC * SD * AgeGrp .001 1 .001 .004 .952 .000 .000 1 .000 .003 .959 .000 Error(CC*SD) 1.716 11 .156 1.636 11 .149 label * CC * SD .216 2 .108 .453 .642 .040 .116 2 .058 .226 .799 .020 label * CC * SD * AgeGrp 1.673 2 .836 3.515 .047 .242 2.063 2 1.031 4.021 .032 .268 Error(label*CC*SD) 5.236 22 .238 5.643 22 .256 Tests of between-subjects effects ANOVA ANOVA with familiarity-controlled scores Source SS df MS F p ηp2 SS df MS F p ηp2 Intercept 1366.02 1 1366.020 915.012 .000 .988 2.016 1 2.016 2.181 .168 .165 AgeGrp 1.344 1 1.344 .900 .363 .076 .182 1 .182 .197 .666 .018 Error 16.422 11 1.493 10.165 11 .924 144 Appendix B Multi-layered interface experiment materials 145 B.1 Call for Participation 146 B.2 Consent form 147 148 B.3 Interview questions 1. Introduction 1.1. Imagine having to learn to use a program on a smart phone. What aspects of the physical device (buttons, screen size) might make it hard for you to learn the program? 1.2. What aspects of the graphical interface (font size, getting around the program) might make it hard for you to learn the program? 2. Before starting basic task set 2.1. Before we go on, could you tell me what will think you'll first do to try learning to perform these tasks? What approach will you take? 3. At the end of the study session 3.1. Imagine trying to teach your friend how to use this address book. What pointers would you give them? 3.2. Did you notice that there were two separate options menus? 3.3. Show list of functions. How do you think they were the menus were different? What do you think the logic was behind how the functions were grouped? We're looking for contact list functions were for using contact list info, and individual contact functions were for updating contact list. 3.4. Think back to your experience learning to use the address book, what part of the address book took the most effort to learn? 3.5. What did you do that helped you learn to use the address book? Text for questions 3.6 and 3.7: The goal of this study is to make programs on mobile devices easier to learn and use. We are exploring two different ways of organizing task functions. Show sheet of 2 address books. You used an address book, where you learned to perform one set of tasks on Interface 1, which only had functions that you first needed to learn. Then you learned to perform another set of tasks on Interface 2, which allowed you to perform all address book functions. This address book also allows the user to switch between the two interfaces as desired, although we did not use this switching feature today. Some other participants in this study used a similar but different address book, where they learned to perform both tasks sets only on Interface 2. 3.6. Imagine having to learn to use this mobile Address Book again, would you prefer to learn on: an address book with the two interfaces and the ability to switch between them? Why? o Under what circumstances would you use interface 1 while learning? Why? o Under what circumstances would you use interface 2 while learning? Why? or an address book that shows all address book functions? Why? 3.7. Imagine having to use this mobile Address Book over the long term in your daily life, would you choose to use: only Interface 1, which only shows commonly used functions? Why? only Interface 2, which shows all address book functions? Why? both, with the ability to switch between the two when needed? Why? o Under what circumstances would you use interface 1 in your daily life? Why? o Under what circumstances would you use interface 2 in your daily life? Why? 149 3.8. Let's think back to your experiences before this study. What has been an easy to learn mobile device feature or program? What made it easy? What has been a hard to learn feature or program? What made it hard? What helped you to learn? (people? manuals? a class?) 3.9. Think about a few computer programs that you learned to use. What has been an easy program to learn to use? What made it easy? What has been a hard program to learn? What made it hard? What helped you to learn? 150 B.4 Instructions for learning and performing tasks Instructions for going through keyboard tutorial In this session, you'll be asked to learn a number of tasks on this mobile phone [point to it]. To get you familiar with using the phone's buttons, you'll go through a tutorial on the mobile phone. You can take as long as you need to go through the tutorial to become comfortable with entering text. Please note that you won't need to use these buttons that are covered in paper [point to them] so please avoid pressing them. But if you do, there's no harm done and we can easily restart the program. You cannot break or crash the program. If the program stops working, it is most likely caused by a problem with the programming and is not because you did anything wrong. Do you have any questions? Hand mobile device to participant to start Instructions for learning to perform basic task set You'll now learn to perform a number of tasks on this mobile address book. This address book may work differently than any one that you've used before. Learn the application as best as you can on your own. Again, you cannot break the device. If the program stops working, this would most likely be caused by a problem with the programming and not something you did wrong. You will be learning to perform three tasks until you master them: 1. Adding 2. a person's contact information to the address book Editing 3. the information of an existing contact in the address book Deleting an existing contact in the address book Start the address book program on the device and show to participant. Ask the following question and record responses on Tasks form: You will perform this set of three tasks a number of times until you have reached a certain level of mastery. Specifically your goal is to learn to perform the tasks with as few extra steps as possible. You will know that you've successfully completed a task when you see "Task Completed" on the device. When you see this, please show it to me so that I can mark down the number shown. If you do not see "Task Completed", then the task has not quite been completed successfully and you will need to redo the task or revise the information that you've entered. Note that that the add and edit tasks require text to be entered precisely using the shown punctuation and letter case. Instructions for performing basic task set after break We now turn our attention back to the mobile address book. You will now perform 2 sets of the tasks that you've leaved earlier. Please complete the tasks as accurately and quickly as you can. 151 Instructions for learning to perform advanced task set Participants in ML group now get the full-functionality interface You will now learn to perform three new tasks. These tasks are: 1. Adding voice dialing 2. for a person in the address book; that is, you can record a contact's name, spoken by you, in the address book so that you can later dial that person's phone number by speaking their name; for example, this feature would allow you to record you saying "mom" with your mom's phone number so that you can later just say "call mom" to the phone and it will do so, Sending a text message 3. to a person in the address book, Adding a custom ringtone Again, your goal is to learn to perform the tasks with as few extra steps as possible. When you successfully complete a task, you'll see a Task Complete screen as before. While you are learning these new tasks, you will also be asked to perform the add, edit and delete tasks that you have already have learned. to a person in the address book, so that the phone will produce a distinctive ring tone when that person calls [Only for ML participants: Note that you will be using an interface that now has more functions, which are used to perform the first set of tasks, the new tasks that you'll be learning, and others tasks as well.] 152 B.5 Instructions for administering cognitive tests Note: Non-italicized text is read out loud. Instructions for the experimenter are shown in italicized text. Reversed Digit-Span Test I will now administer the Reverse Digit Span Test, which measures verbal working memory – the ability to hold on to information while doing something with it. The task involves a series of short trials. On each trial, I will read aloud a list of numbers. Your job is to listen carefully, and then at the end of the list, to recall the numbers in the reverse order. So, if I say the numbers [read one number a second] 2…9…5, your job would be to say them backwards as: 5, 9, 2. Do you have any questions? Read each digit series listed below (see Table B.1) out loud, clearly, at a rate of about 1 digit per second. Begin with the shortest series and proceed through the lists. Immediately after you speak the last digit, don't say any more and wait for the participant to recollect the series in reverse order. And don't give any feedback on whether the recollection was correct or not. Continue through the test until the subject fails to recollect the spoken series on two successive lists. Do not proceed to any longer lists. The subject's RDST score is the total number of successfully recollected lists. Series Digit Lists 1 2 2 – 8 – 3 4 – 1 – 5 3 4 3 – 2 – 7 – 9 4 – 9 – 6 - 8 5 6 1 – 5 – 2 – 8 – 6 6 – 1 – 8 – 4 – 3 7 8 5 – 3 – 9 – 4 – 1 – 8 7 – 2 – 4 – 8 – 5 – 6 9 10 8 – 1 – 2 – 9 – 3 – 6 – 5 4 – 7 – 3 – 9 – 1 – 2 – 8 11 12 5 – 8 – 1 – 9 – 2 – 6 – 4 – 7 3 – 8 – 2 – 9 – 5 – 1 – 7 – 4 Table B.1: Reverse Digit Span Test series of digits Corsi Forward Span Test I will now administer the Corsi Forward Span Test, which measures visual-spatial working memory. Like the previous working memory test, this one involves a series of short trials. On each trial, I will point to a series of squares in a particular order. Your job is to watch carefully, and when I am done, to touch the squares in the same order. So, if I touch these three squares in this order, your job would be to touch the same squares in the same order. Do you have any questions? Begin with the shortest series from the Reversed Digit Span Test (see Table B.1) and proceed through the lists Touch the squares in the specified order at a rate of ~1 square/second (see Figure B.1 for position of squares located on a piece of paper). Make sure that the participant can see which square has been touched, and lift your finger up above the sheet (~6 inches) after touching each square. Begin with the shortest series and proceed through the lists. After you touch the last square don't make any further movements and wait for the participant to start recalling the order. Continue through the test until the 153 subject fails to recollect the spoken series on two successive lists. Do not proceed to any longer lists. The participant's CFST score is the total number of successfully recollected lists. Figure B.1: Position of Corsi Forward Span Test squares, each corresponding to a digit from 1 to 9. Digit Symbol Substitution Test (i.e., Coding Test) I will now administer the standardized Digit Symbol Substitution Test, which measures motor and mental speed, coordination and attention. Please flip to the next page on the clipboard. At the top of this page, there is a row of digits each of which is paired with a unique symbol. The rest of the page shows similar boxes each with a digit, but the symbol is missing. Your job is to write the missing symbol clearly in each empty box -- the symbol that goes with each digit. Let's give it a try on this first row. Let the participant practice the task. After they are done, look over the practice section to ensure that they clearly understood the instructions. The actual test is on the next page, but don't flip to it yet; it has a lot more boxes with missing symbols but you do exactly the same thing as you just did. Your task is to fill in as many symbols as you can in 90 seconds. Please draw/write clearly so that I can read the symbols. You are free to refer to the key as often as you like throughout the test, but you must fill in the boxes in order – starting at the top row, completing squares from left to right. Do you have any questions? When you are ready, flip to the next page. Start the 90 sec timer when they start writing the first symbol. North American Adult Reading Test (NAART) We will administer the North American Adult Reading Test (NAART). This test measures your familiarity with irregularly-pronounced English words. I want you to read slowly down this list of words starting here [point to DEBT] and continuing down this column and on to the next. When you have said a word, wait until I acknowledge it before going on to the next one. There are words here that many people don't know how to pronounce, so just guess at any you don't know. Go ahead. 154 B.6 Sample task instructions Basic task set Advanced task set 155 B.7 Background questionnaire 156 B.8 Post-Basic Task Acquisition phase questionnaire 157 158 B.9 Post-Advanced Task Acquisition phase questionnaire 159 160 161 Appendix C Survey study materials 162 C.1 Call for participation Call for participation posted at community/senior centres where paper surveys are available at front desk 163 Call for participation posted at libraries and at community/senior centres where paper surveys are not available at front desk 164 C.2 Consent form 165 166 C.3 Learning methods questionnaire 167 168 169 170 171 172 173 174 175 176 177 C.4 Qualitative analysis coding instructions General instructions Aim to code each comment with one code; if a couple codes could apply, choose the code that best classifies the entire comment, even if there are many ideas. Suggestion: Code the most important concept that specifically affects user's learning and the design of a learning resource. [supports learning style] and [control over learning] is often more specific of an idea than [learning outcome], and is more important to us. [learning outcome] is often more important than [minimal cost]. [usability] is probably the least important concept of all because we can easily improve it. Q12: "Explain in a few words why you are (or are not) likely to use this particular method or resource (or explain in a few words why you would (or would not) use a particular method or resource if you actually had access to it.)" For each text response: 1. Read the text response and the Likert scale value to the left of it, which provides some context (the Likert scale value is related to how likely the responder is to use the method/resource; 1=not at all likely, 6=very likely). 2. Code text response using one or more codes in the Coding Scheme. 3. Place code(s) that refer to positive qualities or experiences under the "+ codes" column and codes for negative qualities/experiences under the "- codes" column. Note: • Even if respondents don't have access to a resource, we asked them to imagine why they would or wouldn't use it if they actually had access to it. Q15: "Which methods and resources better help you to retain what you've learned (i.e., remember longer)? Explain in 1-2 sentences why the [three best methods/resources for retention] better help you to retain what you've learned:" For each text response: 1. Read the text and the three letters to the left of it; these letters refer to methods/resources that the participant thought best helped him/her to retain what they've learned (see below). 2. Code text response using one or more codes in the Coding Scheme. 3. Place code(s) that refer to positive qualities or experiences under the "+ codes" column and codes for negative qualities/experiences under the "- codes" column. Notes: • Retention here refers to how well the user has learned to use the device (e.g., does not have to learn the same "material" again). However, many users seem to think that it also refers to helping them find learning content to refresh their memory, so these comments are also applicable. • Unless comments indicate otherwise, assume that respondent is directly answering the question, even if they don't connect their responses with retention or gaining an understanding. 178 Q18: "What benefits do you think this type of computer program would have [for you] over getting help from someone?", "What drawbacks do you think this type of computer program would have [for you] over getting help from?" For each text response: 1. Code text response using one code in the Coding Scheme. 2. For codes that refer to qualities inherent in the system (vs. design quality), post-fix the code text with an "–i" (e.g., "[minimal cost-i]") Notes: • Design qualities are generally more important to code than inherent qualities. • Definitions: • Inherent quality refers to qualities of the computer system that cannot be changed (i.e., computer supports presentation of both visual and text content, using proposed computer system does not involve another person) • Design quality refers to qualities of the computer that a user interface designer or learning content author can change (e.g., graphics, amount of information shown, the amount of control a user has on the user interface) 179 C.5 Qualitative analysis coding scheme Coding scheme for responses to: • Q12: "Why use (or not use) this method?" • Q15: "Explain in 1-2 sentences why the methods/resources you listed…better help you to retain what you've learned." • Q18a: "What benefits do you think this type of computer program would have over getting help from someone (including the most helpful person you know)?" • Q18b: "What drawbacks do you think this type of computer program would have over getting help from someone (including the most helpful person you know)?" Note that comments in the examples column that have the opposite meaning to a particular concept are prefixed with a "[-]" Concept ([Code]) concept Definition for + score/categorization examples Key Idea/Comments qualities of learning method/resource [supports learning style] group learning method/resource supports learning in a group setting [-]: "not a group learner" key idea: learn with and from other learners learning-by-doing method/resource supports experiential learning where person learns by performing tasks in a realistic context (e.g., on user's mobile device) "I learn by doing mostly." doing authentic (realistic) tasks during learning process problem solving method/resource allows the learner to make mistakes and try to solve the problem "trial and error teaches you what can go wrong and how to correct it", "work it out by myself", "figuring it out by myself" real mental effort to solving learning problem demonstration resource provides demonstrations to the learner "I learn better if I am shown" seeing someone else perform task and then copy practice method/resource supports the learning in practicing tasks repeatedly "I like to learn and practice one task a time until I am competent then go onto the next task" performing task repeatedly until mastery step-by-step resource provides step-by-step instructions to the learner "a step-by-step guide so it gets the user to do the work themselves" visual content resource presents content using visual illustrations (e.g., figures, pictures) "usually gives detailed instructions and illustrations." does not refer to reading text, but rather interpreting images text content resource presents content using text that learners can read [-]: "I hate reading instructions" oral content resource presents content orally "I usually ask them to tell me how to fix/solve an issue" [learning content] content quality resource provides accurate, consistent information system would "never vary, not being human", reliable information content clarity resources provides content in an easy- to-understand language and that is organized well (e.g., logical sections and flow, systematic, content is divided up into easily digestible chunks) communicates in user's language, doesn't use jargon; I don't have to listen to more than I can retain/understand. "prevents learning from going in wrong direction" information communicated in easy to understand language and flow presentation timeliness content is given by system at the optimal learning time It gives you instant feedback - which is important because I get really mad when I go just a few steps in the wrong direction. more details available resource provides more details to the learner if desired [-]: "It usually doesn't have the answer or provides insufficient details" detailed info doesn't need to always be shown but is available 180 Concept ([Code]) concept Definition for + score/categorization examples Key Idea/Comments qualities of learning method/resource (continued) [control over learning] navigation through content learner can easily control over what content is shown (e.g., can easily skip to or find desired content, can learn at own pace, pause, and repeat); interaction not too complex "At least with someone I can tell them to skip parts I already know or am familiar with." control over what they learn easy-ask questions learner finds it easy to ask questions during the learning process and get helpful answers; user knows question(s) to ask and the "right" words/terms for asking the question [-]: "because I am not sure what is the problem and what questions to ask" [usability] easy to use learner finds it easy to use the resource/method despite lack of certain knowledge/experience, or declines in abilities (memory, processing speed) "Easy to use" includes responsive UI feedback cost to the user to use learning method/resource [minimal cost] time efficient method/resource provides satisfactory learning value (or more) given the time and effort invested "fast", available when I want it [-]: "I would if necessary, but it is often very time wasting and not always successful." conveniently available resource is easy for learner to access and available when learner needs it "Readily available" affordable method/resource is financially affordable "free" learn without owing others later using method/resource does not require involving another person, and thus no need to compensate that person now or in the future I wouldn't owe them a favour for them having helped me; a computer "would not get bored or frustrated by having to repeat" main point here is that the respondent thinks learning without others will prevent him from needing to compensate the other person for the "negative" experience of having to teach them; this code doesn't apply to the negative feelings that the learner experiences during the learning process, which should be coded [learning outcome] [more time] more time method/resources requires learner to spend more time in the learning process "They require me to spend more time on the subject at hand and thus, I am able to retain it better." learner's experience during the learning process and learning outcome [learning outcome] learning experience learner experiences positive feeling(s) during the learning process positive (+) feelings: confidence, satisfaction, enjoyment, empowerment; negative (-) feelings: discouragement, frustration, annoyance; judgement, feel stupid refers to how learner feels during learning process (and not what he/she thinks other people are thinking about him/her); includes comments about learner feeling that a resource is patient learning success learner feels that he is generally successful in learning (e.g., steps, general understanding) with this method/resource "often i can figure it out", "best way to learn", "this usually helps me to learn" refers to successes using a particular method/resource to *learn* (specific learning style not named) better retention learner feels that he will retain what he has learned with this method/resource "When I can figure things out for myself I can retain it better" refers to successes using a particular method/resource to *retain learning* (specific learning style not named) 181 Concept ([Code]) concept Definition for + score/categorization examples Key Idea/Comments Did not answer [dna] blank respondent did not enter comment "0" gibberish respondent did not enter intelligible comment "x" irrelevant comment respondent answered but response is completely unrelated to question general endorsement learner states that he likes or prefers method/resource but does not say why "best way to learn", "yes", "I could just do it myself" refers to generally positive comments (e.g., "best", "good", "great", "helps") but does not specify why (e.g., why it is good or how it helps); such comments are generally too brief and do not help to answer research question other methods/resources ineffective learner states that he uses this method/resource because it is better than other ones, but does not say why "only as last resort" too vague comment could be coded validly in several ways "easiest", "maybe not being able to communicate in real time", "it's less time", "not friendly" some comments may hint at some insights but they are too brief or too general to code it with certainty; also, comment could be unclear about what the person is referring to Coding Scheme for responses to 18c: "How easy/difficult do you think it would be to operate both a mobile device and desktop computer program at the same time?" Concept ([code]) examples Comments [easy] "no difficulty", "not too difficult", "OK" if comments sounds really possible [depends] "I think if you have a problem using a mobile device in the first place - very difficult." uncertainty about whether respondent thinks they would be able to operate both mobile device and computer program (e.g., multitasking, visually going between two displays); look for "if", "depends" statements (i.e., qualifiers); look at respondent's Q18d (and maybe Q18a&b) comments to help you decide how uncertain the respondent is; [hard] if comment sounds like a complaint, then comment probably difficult [dna] "n/a" did not answer, irrelevant or ambiguous; also applies when comment talks about other people but it's unclear whether it applies specifically to the respondent Coding scheme for responses to 18d: "Would you use such a computer program to learn to use a mobile device? Why or why not?" Concept ([code]) examples Comments [yes] "Yes it would be faster" if comments sounds really positive [depends] "Maybe", "if the program was…" look for "if", "depends", "maybe" statements [no] if comment sounds like a complaint, then comment probably difficult [dna] "n/a" did not answer, irrelevant or ambiguous; also applies when comment talks about other people but it's unclear whether it applies specifically to the respondent 182 Appendix D Help Kiosk informal user study materials 183 D.1 Call for participation 184 D.2 Consent form 185 186 D.3 Background questionnaire 187 D.4 Interview questions 1. How mentally demanding was using the kiosk/manual to perform the mobile phone task? (1=very low, and 6=very high) 2. How physically demanding was using the kiosk/manual to perform the task? (1=very low, and 6=very high) 3. How hurried or rushed was the pace of learning with the kiosk/manual to perform the task? (1=not rushed, and 6=very rushed) 4. How successful were you in using the kiosk/manual to perform the task? (1=very successful, and 6=not at all successful) 5. How hard did you have to work to use the kiosk/manual to perform the task? (1=not at all hard, and 6=very hard) 6. How insecure, discouraged, irritated, stressed and annoyed were you? (1=not at all, and 6=very) 7. How difficult or easy were the instructions to follow? (1=very difficult, 6=very easy) 8. How much did the demo videos help you follow the instructions? (1=not at all helpful, 6=very helpful)? 9. Were the demo videos useful in helping you figure out what should happen after you perform the step? 10. How much did the highlighted suggestions help you follow the instructions? (1=not at all helpful, 6=very helpful) 11. How difficult or easy was it to use both the mobile phone and the touchscreen monitor at the same time? (1=very difficult, 6=very easy) 12. Would you use this Help Kiosk at home? In a library? In a store? Why or why not? 13. When do you see yourself using this? 14. How could the design be changed to make it more appealing for you to walk up and use in a store? 15. I will be describing a number of features we are thinking of implementing and I would like to get your thoughts on whether these would be useful to you. What would you like to see in this feature? What are some potential issues in using it? • Getting to the desired help. Go to find info screen. Users can go to information through two ways. You can click on a word/phrase in this cloud of words, or can search on any words that you can think of. How useful and usable is the tag cloud? • Exploratory/practice mode. The user is learning to perform steps on their mobile phone, updating information on it and changing settings. Let's say the user isn't comfortable with doing this while he is still learning to use the device, in case he makes unwanted changes. The user can switch to an exploratory mode where the user can change anything they want and change everything back to the way it was before entering the exploratory mode o Remote Administration of the phone o Personalization o What other tools or programs do you think you would use on a mobile phone? 188 D.5 Script for introducing Help Kiosk and manual Today we will have you perform tasks on this smart phone using a traditional manual, and also with a computer system named Help Kiosk. The Help Kiosk is a very new system and we need your opinions to see where we're on the right track and what areas need improvement. The main purpose of this session is to get your true opinions on using the Help Kiosk versus using the manual. You aren't being evaluated. You are helping us to evaluate the manual and Help Kiosk. If the program stops working, this would most likely be caused by a problem with the programming and not something you did wrong. It is usually not because of something you did, so feel free to do things as you normally would. Here's a touchscreen mobile phone. Show Nexus One. Describe Back, Menu, Home, Search buttons. Introducing Help Kiosk Guide participant through taking a photo with the phone. Let them do it themselves again. Then demo the Help Kiosk for supporting the taking picture instructions. There are two panes: Find Info and Learn/Do. I'll guide you through the finding task instructions. Let's start with an example: taking a photo. On the bottom right pane Help Kiosk provides demonstration videos that show someone perform a step in the instructions. On the top right pane, Help Kiosk provides a Live View of your mobile phone screen, sometimes highlighting areas of the interface you will use in performing a particular step. You cannot control the mobile phone through Live View; touching in this area doesn't do anything. You click on each step to go through them. Introducing instruction manual Here is the manual (or user guide) that comes with this phone. As you flip through it, you can see that it covers many topics. Start with the bookmarked pages to perform the task. Feel free to go to other pages. We also have an electronic version of this manual. Let me know if you want to use that one instead. 189 D.6 Instructions to participants on how to perform tasks Attempt 1 Use the (Help Kiosk/Manual) to help you perform the task. Use the resource as naturally as you can. There's no time pressure, nor pressure to perform the task correctly the first time, so relax as much as you can. As you're performing the task, please try to say your thoughts out loud. For example, "I think I should do this", "this doesn't make sense", and "I would like to do this." Attempt 2 Perform this task again with the (Help Kiosk/Manual). Again, try to say your thoughts out loud as you're performing the task. Attempt 3 We now turn our attention back to the mobile phone. You will now perform the two tasks that you've leaved earlier. Please complete the tasks as accurately and quickly as you can. 190 D.7 Task instructions Add contact tasks for Attempts 1 (top), 2 (centre), and 3 (bottom) 191 Set alarm tasks for Attempts 1 (top), 2 (centre), and 3 (bottom)
UBC Theses and Dissertations
Featured Collection
UBC Theses and Dissertations
Improving the learnability of mobile devices for older adults Leung, Rock Anthony 2011
Item Metadata
- Media
- 24-ubc_2011_fall_leung_rock.pdf [ 3.49MB ]
- Metadata
- JSON: 24-1.0052115.json
- JSON-LD: 24-1.0052115-ld.json
- RDF/XML (Pretty): 24-1.0052115-rdf.xml
- RDF/JSON: 24-1.0052115-rdf.json
- Turtle: 24-1.0052115-turtle.txt
- N-Triples: 24-1.0052115-rdf-ntriples.txt
- Original Record: 24-1.0052115-source.json
- Full Text
- 24-1.0052115-fulltext.txt
- Citation
- 24-1.0052115.ris
Full Text
Citation Scheme:
Usage Statistics
Customize your widget with the following options, then copy and paste the code below into the HTML of your page to embed this item in your website.
<div id="ubcOpenCollectionsWidgetDisplay">
<script id="ubcOpenCollectionsWidget"
async >
Our image viewer uses the IIIF 2.0 standard. To load this item in other compatible viewers, use this url: