{'fraction_non_alphanumeric': 0.02255308762509153, 'fraction_numerical': 0.007029533805223334, 'mean_word_length': 5.20411871592974, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 1, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Usability: An introduction to and literature review of usability testing for educational resources in radiation oncology

Usability, or the ease with which something can be used, is a key aspect in ensuring end-users can achieve the best possible outcomes from a given educational resource. Ideally usability testing should take place iteratively throughout the design of the resource, and there are several approaches for undertaking usability testing described in the wider literature. Within radiation oncology education, the extent to which usability testing occurs remains unclear. This literature review aimed to assess current practice and provide a practical introduction to usability testing for educational resource design within radiation oncology. Two web databases were searched for articles describing planned or completed usability testing during the design of a radiation oncology educational resource. Fifteen studies were identified. Data was gathered describing the type of usability testing performed, the number of cycles of testing and the number of test subjects. Articles described design of educational resources for both patients and trainees, with the number of test subjects ranging from 8 to 18. Various testing methods were used, including questionnaires, think aloud studies and heuristic evaluation. Usability testing comprised a range of single cycle through to several rounds of testing. Through illustrative examples identified in the literature review, we demonstrate that usability testing is feasible and beneficial for educational resources varying in size and context. In doing so we hope to encourage radiation oncologists to incorporate usability testing into future educational resource design.


Introduction
The past two decades have seen a meteoric rise in the use of digital methods and resources within medical education [1], an increase which has been further fuelled by the Covid-19 pandemic and the accompanying restrictions on face-to-face teaching [2][3][4]. In parallel, digital educational resources have become an important source of accurate and up-to-date information for patients [5,6].
Usability testing focusses on ensuring the user of any digital tool or information system can navigate and engage with the resource easily and effectively. It is widely performed in system design within the software industry and its importance in design of educational interventions is increasingly recognised [7]. It is not clear to what extent usability is applied within radiation oncology.
The aims of this article are to outline the main ways of testing usability and assess how this has been done already within radiation oncology by means of a literature review. In doing so we aim to provide a practical guide for readers of this special issue in education to incorporate usability testing into design of their own radiation oncology educational resources in future.

Background to Usability
What is Usability? In its simplest terms, usability is "the ease with which a person can use a product in a particular set of circumstances" [8]. Usability has been more formally defined by the International Organization for Standardization as "The extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency, and satisfaction in a specified context of use" [9].
In her introductory book on the subject, Barnum highlights the importance of these specified users, specified goals and specified context when considering usability [10]. To consider these within the realm of medical education: • Users -An online oncology educational resource may be highly usable for a young adult, but relatively unusable to an elderly person (statistically more likely to be diagnosed with cancer and therefore receive radiotherapy) who does not regularly access the internet [11]. • Goals -An educational resource designed to facilitate oncology treatment decisions for patients, which contains detailed information but provides no final summary page, may be highly informative but ultimately unusable for its defined goal as users struggle to remember and assimilate what they have read. • Context -A resource designed to be used on a Safari or Chrome webbrowser but released in a hospital where all computers operate a legacy version of Internet Explorer, may be useless in its specified context.
Within this review, we will be considering the assessment of usability of educational resources within radiation oncology. Users are therefore usually either patients or healthcare practitioners and goals range from facilitating treatment decisions to improving communication or radiotherapy contouring skills.

Why is Usability Important?
Usability is relevant in the design of any educational resource, but its importance is never clearer than within e-learning where users may be required to interact with complex systems. These systems can provide new educational opportunities, for example educational contouring software which can provide direct user feedback for trainees [12]. As complexity increases however, so too does the opportunity to lose users due to issues with system design.
Usability has been shown to be a key factor (alongside perceived usefulness) influencing our acceptance of information technology [13,14], which predicts actual use.
E-learning has been extensively studied within the corporate field where e-learning courses increasingly replace traditional instructor-led courses [15]. While e-learning courses are often cheaper and more convenient, they have also been shown to have higher attrition rates than traditional courses; one reason for this could be poor usability of the resources [16]. Sandars agrees with this view, when he argues that poor usability could be an explanation for the findings of a 2008 metaanalysis that demonstrates that while e-learning in healthcare is superior to no intervention, it is no more effective than traditional learning methods [8,17].
It is easy to assume that an educational resource we have created is usable; we have, after all, been carefully developing it for weeks or months and its intricacies are second nature. Barnum observes that "From the moment you know enough to talk about a product […] you know too much to be able to tell if the product would be usable for a person who doesn't know what you know." [10] It is therefore essential that we not only consider usability during the design of an educational resource, but that we formally test it.

How to Test Usability
Sandars describes four main dimensions that we should consider when assessing usability of e-resources: the learner, technological aspects, instructional design aspects, the context [8]. It is essential that, at least in some stages, a usability assessment involves the intended enduser and that ideally it is assessed in the context in which it will finally be used. 'Technological aspects' refers to factors such as the ease of navigation, consistency of layout and clarity of the visual design. Instructional design includes the content itself, the interactivity and judicious use of multimedia.
There are multiple different methods of usability testing described in the literature. Table 1 summarises those most encountered within medical education, briefly describes the benefits and limitations of each and provides an example study which can be consulted for further reference. It is relatively straightforward to survey many individuals with a questionnaire; it is considerably more resource intensive to perform multiple think aloud studies, cognitive walkthroughs or heuristic evaluations. Nielson and Landauer analysed 11 usability studies using either heuristic evaluators (i.e. usability experts) or end-user evaluators (e.g. patients or clinicians), compared the number of evaluators with the number of usability problems identified, and developed a model to determine how many testers were required [18]. They showed that for a small project the optimum cost-benefit analysis requires only four evaluators. Five evaluators are generally accepted to be able to identify ~85% of usability issues [19]. Later authors have however stressed the importance of context and appropriate sampling in defining the numbers to be studied [20].

Methods
A literature review was carried out to assess the reported use of usability testing in the radiation oncology education literature. PRISMA guidelines were followed [30]. Inclusion criteria were articles which described completed or planned usability assessment of an educational resource within radiation oncology. The search was initially carried out on 25/6/2022 and all results up to this date were included. Fig. 1 outlines the methodology:

Results
Three articles were identified as relevant from abstract screening and subsequently excluded at review. One was an analysis of pre-existing web resources for patients with prostate cancer [31]. While the tool they used to assess these websites had been previously tested for usability, the websites themselves were not explicitly assessed. A second article addresses usability specifically in the context of patients with lower health literacy levels [32]. They provide recommendations for enhancing usability in a practical sense, e.g. by providing audio material alongside visual, but do not cover how to carry out usability testing. A third study observed browsing patterns of visitors seeking radiologyrelated information on a hospital website to develop a model which could then by applied elsewhere to improve browsing experience [33]. This was excluded as it is not specifically about the development of an educational resource.
The fifteen items selected for full review were read by two reviewers (HLK, SLD) to identify the educational tool being developed, the intended audience, the usability assessment method used, the number of testers and the point in the design process at which usability was assessed. The results are summarised in Table 2.
Most examples of usability testing in the radiation oncology medical education literature describe the design of a patient information resource (13/15, 87%), although there are also examples of usability assessment use in designing resources for healthcare professionals (4/ 15, 27%). There are two examples of papers which do both: Juraskova et al. [21] describe the design of educational modules for both clinicians and patients/caregivers, and the study by Raith et al. [42] describes two separate augmented reality protocols for patients and radiographers respectively.
Direct observation was used in 8/15 (60%) studies. 3/15 (20%) describe the use of expert heuristic evaluation. Two of these studies are among those which describe more extensive usability testing, including multiple testing modalities and iterative testing throughout the design process. 6/15 (40%) studies describe a think aloud. These involve a range of 8 to 18 participants in any single round of testing, and some studies describe more than one round. The formality of analysis of the think aloud varies; some papers describe extensive transcription and thematic analysis, whereas others describe drawing general learning points.
10/15 (67%) studies use a survey as part of their usability assessment. Of these, 5/10 (50%) were not formally validated usability questionnaires, but asked questions explicitly designed to probe usability. Of those which used validated studies, three (60%) used the SUS (one of which was modified) and one (20%) used SUMI. One study mentions a future plan to use UMUX-LITE but has not yet done so.
Although most studies assessed usability on the intended target audience this was not universal; there was one example of a study where clinicians alone were used to assess the usability of a resource being designed for patients. Raith et al., [42] were unable to assess their augmented reality resource on patients due to the restraints of the covid-19 pandemic.

Discussion
In the introduction to this article, we described five methods of usability testing. These can be divided into 'expert-led' testing (heuristic evaluation, cognitive walkthrough) and 'user-led' testing (think aloud, semi-structured interview, questionnaire). Examples of all five types of testing exist in the current radiation oncology education literature. There are notably far more cases of user-led testing.
Usability experts may come with a cost -much of the seminal work on usability comes out of software design where usability experts to undertake heuristic evaluations are readily available, which is likely to be a luxury unavailable in healthcare. This scarcity may be addressed by collaboration with computer science departments within or across academic institutions.
In one of only three studies employing expert-led testing, Ankolekar et al. describe the development and validation of a patient-led decision aid for prostate cancer [29]. They detail an extensive process of usability testing involving multiple cycles of testing, re-design and re-testing. Over five rounds of testing, they employ questionnaires, heuristic evaluation and think-aloud methods. They explain that "Our development process spanned over two years and involved 58 participants, resulting in >100 h of interview material and feedback that needed to be processed, analyzed and incorporated in successive rounds." While such extensive testing clearly has the potential to produce a high-quality educational resource, the time and financial commitment required may prove a disincentive for others to carry out similar work.
Other studies describe less extensive usability testing from which the authors are nevertheless able to make changes to their resource. Bigelow et al. describe two cycles of questionnaire-based testing and demonstrate an improvement in usability between cycle 1 and cycle 2 following changes to the design, language and graphics of their resource [40]. We would therefore argue that current literature suggests that it is both valuable and feasible to carry out usability studies within radiation oncology education on a variety of different scales suited to the specific aims of the resource.
Most of the studies which carry out only one or two rounds of usability testing employ user-led testing. Many of the advantages of this are intuitive; testing on the group that will ultimately be using the resource makes logical sense. Additionally, most healthcare professionals involved in the design of an educational resource will have easy and free (limited by necessary ethical approval) access to the end users, be they trainees or patients.
Within user-based usability testing a scale of resource and timeintensity exists. Formal think-aloud testing is an extensive process involving scriptwriting, taping of users, transcription of tapes, coding by  [29] independent coders and thematic analysis. Hopmans et al. provide an example of how this process can be shortened while still providing valuable insights [36]. In an initial round of think-aloud testing they ask 18 participants to navigate through their website and then ask a series of probing questions. They transcribe all the interviews, then three are selected for independent coding and analysis by two separate researchers; the remainder are analysed by one researcher only. This is one example of how a potentially intensive analysis process can be shortened while still providing useful insights. The article does not describe how the three transcriptions are selected; it would be important to ensure this is done at random. Finally, questionnaires are a relatively easy method of usability testing. Most frequently used in our review is the SUS, which is a wellvalidated usability survey that is freely available online. As healthcare practitioners we are generally used to seeking survey style feedback on our educational resources, so it is fairly straightforward to add in some usability-focussed questions. If longer surveys like UTAUT are felt to be too arduous and add excessively to survey burden, then the SUS or UMUX-LITE are shorter alternatives.
Several studies in our review test usability on 'cancer patients'. We would agree that it is essential in health education to test resources on the target audience, however it is worth considering that unless these testers are at the beginning of their cancer journey, they may in fact have a higher level of knowledge than intended. In this context, healthy volunteers may provide a reasonable alternative, with the caveat that volunteers are likely to be interested and engaged in healthcare and may therefore have a higher level of health literacy than the general population.
Our systematic review identified only two studies describing usability testing on an educational resource for radiation oncology trainees. This might simply represent the fact that there are fewer such resources being regularly created. The two articles identified in our review [26,37] both describe design of resources to help with contouring. This is clearly a field where usability is of crucial importance as factors like ease of navigation, ability to concurrently view atlases and contouring software and similarity to trainees' local contouring software is likely to have a large impact on their engagement with the resource. We would encourage anyone designing such a resource to consider undertaking and reporting usability testing. As a result of the inclusion criteria, all articles included in our review include a description of planned or completed usability testing. Only a proportion of these articles describe whether and how the results of the usability testing benefited ongoing resource development. An example of this being done well is Nguyen et al.; Table 1 in their paper describes the method, results and insights of serial rounds of usability testing [24]. We would suggest that a description of the changes made (and, if possible, repeat usability testing to demonstrate an improvement), would enhance any paper reporting usability as it would help identify common areas of difficulty.

Limitations and possible future work
A limitation of this review is that due to the search criteria, it only picks up studies which have specifically mentioned 'usability'. It is possible that usability may be assessed but not formally described, or that usability is being assessed in educational resources that are never formally published (or are published only in abstract form). We would encourage more radiotherapy researchers to publish their usability data and lessons, as these may help prevent others from repeating similar mistakes. The literature search was limited to only two databases, which may also have limited the number of results.
A future review might identify a more specific area of radiation oncology educational material and assess all educational resources published within this field, to determine what proportion of them report usability testing. This would give a better idea of how widespread usability testing is. This is not possible to assess from our review, which does not include educational resources which are not usability tested.

Conclusion
In this article, we have discussed the rationale for carrying out usability testing in the design of educational resources and described the main methods for doing so. We go on to report the results of a literature review of the current use of usability testing within radiation oncology.
Current practice demonstrates that there is a balance to be achieved between the resource intensity of usability testing and the potential improvements to an educational resource. We would encourage all educationalists designing resources for either patients or trainees to consider how usability testing might reasonably be incorporated in their own design process. The ideal method(s) depends on the aim of the resource and certainly anyone aiming to design a durable and farreaching resource should consider multiple rounds and methodologies of usability testing.
We hope we have also provided the necessary tools and information to show that even in simpler more local projects, it is feasible to carry out some basic usability testing to maximise the impact of a resource.  [43].

Declaration of Competing Interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.