{'fraction_non_alphanumeric': 0.029864310112500392, 'fraction_numerical': 0.01767415624706214, 'mean_word_length': 5.364579178300758, 'pattern_counts': {'":': 1, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Addressing the"Leaky Pipeline": A Review and Categorisation of Actions to Recruit and Retain Women in Computing Education

Gender imbalance in computing education is a well-known issue around the world. The term"leaky pipeline"is often used to describe the lack of retention of women before they progress to senior roles. Numerous initiatives have targeted the problem of the leaky pipeline in recent decades. This paper provides a comprehensive review of initiatives related to techniques used to boost recruitment and retention of women in undergraduate computing and related courses in higher education. The primary aim was to identify interventions or initiatives (which we called"actions") that have shown some effectiveness. A secondary objective was to structure our findings as a categorisation, in order to enable future action discussion, comparison and planning. A particular challenge faced in a significant portion of the work was the lack of evaluation: i.e. the assessment of the direct relationship between the initiatives and the outcomes on retention or recruitment. The actions were categorised into four groups: Policy, Pedagogy, Influence and Support and Promotion and Engagement. Policy actions need support and potentially structural change at institution level. Pedagogy actions are initiatives related to the teaching of computing courses. The Influence and Support category includes actions associated with ways to influence women to choose computing and once enrolled to support and encourage them to stay. Finally, Promotion and Engagement actions are initiatives to promote computing based courses and involve engagement and outreach activities. We present our categorisation, identifying the literature related to actions under each category and subcategory. We discuss the challenges with evaluating the direct impact of actions and outline how this work leads towards the next phase of our work - a toolkit of actions to promote retention and recruitment of women in computing undergraduate courses.


INTRODUCTION
Despite a significant amount of research and practical initiatives that have taken place to stimulate gender balance in computing fields 1 , the problem still persists. In 2021 only 19.1% of ICT specialists working in the EU were women [1]. With regard to tertiary education,there are less than men in ICT disciplines. For Ireland, the latest available report for new entrants in 2017/18 [2] shows that on average nationally just 14.8% of ICT related course enrolments were women. A recent UK report [3] indicates that just 19% of computer science students were women. EU statistics show a similar pattern. In 2018 girls and women accounted for only 17% in upper secondary and tertiary ICT education on average across all countries [4].
Without enough women in computing education there will be limited chances to increase women's participation in relevant careers further down the pipeline. The metaphor "leaky pipeline" is commonly used to describe the lack of retention of women in STEM fields before they continue to more senior roles [5].
Some of the frequently mentioned reasons for the leaky pipeline include stereotypes about computing being a male domain [6], cultural influence [7], a lack of a sense of belonging [8] and family and caring matters [9].
Anderson et al. [10] discuss that these reasons are not universal and do not provide a solid foundation for extensive replicable initiatives for recruitment and retention of women in computer science. Nonetheless, there has been a large number of success stories reported in different parts of the world [11,7,12,13,14,15]. Hence, we find it important to collect and discuss effective initiatives that worked in a range of settings. This paper aims to produce a comprehensive review and categorisation of initiatives used around the world. It focuses on those that showed impact on recruitment and retention of female students in technology related courses. A significant challenge faced in tackling the problem of gender imbalance lies in the assessment of initiatives and the interpretation of results. We discuss the common techniques and measures used in evaluation. We also note any observed limitations of positive impact on certain groups of students.
The rest of this paper is structured as follows: Section 2 outlines our sources of literature, and our methodology. Section 3 outlines our proposed categorisation of actions, which is presented visually, and discusses the actions in detail. Section 4 discusses the challenges associated with the measurement of impact and discusses the methods that are currently used. Section 5 concludes the paper.

APPROACH
We focused our review on the technical higher education stage and successful recruitment and retention techniques relevant to gender minorities, primarily women, were targeted. We identified the relevant sources by using online search engines and digital databases, by conducting manual searches and by inspecting grey literature. Additionally, a search in languages other than English was performed, including German, Spanish, Ukrainian and Russian.
We used online search engines such as Google Search, Google Scholar, Microsoft Bing, Twitter Search, multi-databases platforms such as EBSCOhost, ProQuest and TU Dublin library catalogue. Databases such as ACM Digital Library and Emerald Insight were searched as well. Keyword combinations in relevant languages such as, "recruitment of women in computing", "effective techniques to recruit and retain women in STEM higher education", "women in technology initiatives", "recruitment and retention in computer science", "female gender recruitment in technology", "interventions to attract women into computing", "female student recruitment in computing", "initiatives for women in computing" were dealt with.
Manual searches were then done to look for additional relevant materials such as browsing through earlier or later volumes of relevant journals, and searching for additional publications of researchers specialising in the field.
We identified three different sources of literature among over 350 relevant publications. First are research papers from journals and academic conferences, as well as academic reports. We included academically published studies and interventions, which focused on addressing the problem of gender imbalance in computer science education. This group offered the largest pool of literature as compared to the other two. Nonetheless, many sources focused on theoretical discussions or research based recommendations, as opposed to evidence based studies.
Second, given that many gender balance in STEM actions are not formally reported in peer-reviewed publications, we considered material outside of the academic collections to identify practical advice and initiatives. These included submission papers for international awards [16]; web portals of initiatives aiming to increase gender diversity in computing, such as Ada in Norway [17], Czechitas in Czechia [18], INGENIC in Ireland [19], Ladies in Informatics in Slovenia [20] or MIT Women's Technology Program in the United States.
The third source covers governmental/institutional guidance -e.g., reports by international organisations and recommended strategies. Depending on their focus, authors dealt with the improvement of existing practices at a policy level [12,15]; teaching level [21,22,23]; support level by role models/mentors [7,24]; or used outreach strategies to attract students into the field [25,26].
Two categories of approaches on how to improve gender balance in computer science education have emerged from the literature: Frameworks and Actions. Frameworks offer a structure or strategy at a high level to promote and improve gender balance. Often they are presented as a process or step-by-step guide which typically involves key tasks such as self-assessment, planning, roles and responsibilities, recruitment and monitoring. They can offer practical advice on strategies and can include case studies. They can also be called workbooks, models, toolkits, toolboxes, or tools by their authors [27,28,29,30,31,32]. Actions are practical initiatives that can be implemented and evaluated individually. We focus on actions with some support in the literature for having had successful impact.
The following section introduces our categorisation of actions and discusses initiatives that have demonstrated effectiveness in either encouraging women into technology-focused courses or assisting in the retention of women on these courses. In limited cases, relevant initiatives have been considered from disciplines such as general STEM [33,34,35]; engineering [36,37,38,39]; or where women were not the focus but impact was significant [40,41,42,43,26,44,45].

CATEGORISATION OF ACTIONS
Our categorisation of actions, displayed in Fig. 1, consists of four groups: Policy, Pedagogy, Influence & Support, and Promotion & Engagement. Within each category, grouping or subcategories of actions were identified. Note that some actions could belong to multiple categories or subcategories or require support by stakeholders from other categories. Policy actions require support and potentially structural change and resources at organisation level. This can be at a department or school level within a higher level institution, and not necessarily just at the higher institution level. For example, to set a enrolment quota for women, or to recruit female faculty staff. Pedagogy related actions are initiatives that are related to the teaching of computer science and technology in terms of curriculum or module delivery. These types of actions could potentially be undertaken at a module level by an academic within a school or department and do not necessarily require policy changes. Influence & Support actions are associated with ways to influence female students to choose computing at third level and once enrolled to support and encourage them to stay in the field. Promotion & Engagement actions are initiatives to promote computer science and technology based courses and can involve engagement and outreach with external stakeholders such as industry, communities, and schools. Table 1 shows the papers with actions identified as having some level of impact in the recruitment and retention of women in computing course grouped by category and subcategory. Table 1: Papers which has shown some positive impact from actions to increase recruitment and retention of women in computing based courses grouped under category and sub-category and identifying the kind of evaluation used to measure the level of impact.

Policy
Policy actions can involve change to processes or policies at institutional or departmental level. These will generally require high level of support from leadership posts and may require significant resources to be assigned in order to implement.
A significant number of actions in this category are associated with Admission Procedures. Some examples include changing the entry requirements to remove the requirement for computing or programming background [7] and facilitating students moving courses more easily [76,46]. More radical actions have been successful in Norway where Norway University of Science and Technology (NTNU) instigated a quota for students and permitted reduced points for female candidates in 2007 [12]. There are women-only computer science courses emerging now [47,15].
There are also examples of actions that can be beneficial for female Staff Recruitment including adjusting job descriptions to make them more appealing for women to apply, or targeting specific individuals though personal contact and setting targets for hiring [39]. Some positive experience has been reported after appointing a dedicated champion/lead of the project [13,48].
The need for Cultural Change and the appropriate management of this change at institutional level are identified as important steps towards gender equality in technical disciplines [7,13,39]. Practical examples are the recognition of outstanding female students with awards [36]; organisations for faculty members and students together that provide institutional support and funding for initiatives [7]; and providing a guaranteed family friendly study schedule for students with children or carer's responsibilities [15].

Pedagogy
Pedagogical practices include actions that deal with the curriculum and teaching activities, and can be typically undertaken by the academic staff. In some cases support from a higher level may be needed. Some relatively simple Curriculum Based changes have been effective such as revising module sequences to balance practical and theory modules evenly in year one [13], or a non-obvious hardware module that ends with an installation party [47]. Hybrid courses including modules from other disciplines such as business [15], biology [49], a foreign language [13], or giving options to choose from many combined majors [50] have shown good success at being more attractive to female students. Frieze & Quesenberry [7] report from their experience at Carnegie Mellon University that curriculum change needs to benefit all, and not only women, when implemented.
Some actions involving game development collected positive feedback including a gender neutral interactive adventure game authoring for high school students [51], a 2 hour workshop to introduce women to creative IT including game development [53], and creating video games that are personal and meaningful to women and tell their own stories [52]. There is evidence that game degrees contributed to increased enrolment for all students [26]. Previous experience seems to be particularly important for women when deciding in favour of creative technical subjects [53].
Induction is known to be an important event for new students particularly to stress the value of incorporating the key message of participation and attendance as part of the induction [13]. A mandatory induction social event is recommended for new students as a possible benefit to later retention [77]. Focusing on Delivery Techniques also has had positive impact. The flipped classroom model is promoted as a promising approach for teaching computer science [43] and it has been used successfully in a lot of research into encouraging female students in computer science [21,54,14]. It has also been shown that the flipped classroom model positively influences the academic accomplishments of female students [14] and the retention of women in computing classes [54]. Conversely, it does not seem to benefit men as much as women [54,14] as well as less motivated students in general [43]. In some cases it was helpful to female students to be provided with e-learning courses repeating the material from the education plan [56] or provide all the study material upfront and a video recording after the class [23]. Active learning strategies are commonly mentioned as beneficial for female students alongside the flipped classroom teaching model. Specific techniques that had impact include pair programming [21,22,59,61], live coding [21,55], practice-based learning [38], particularly in a game environment [60], and peer instruction [21,23,54]. Approaches such as project based work [35,25] and encouraging a culture of discussion [35] have also shown impact. Gutica [44] found that students were engaged and motivated, demonstrated improved communication and team work skills through use of a large agile software development project. Additionally, agile software development methods are, according to Frieze et al. [78], a better match to women's style of management. Visual programming has shown to be beneficial for increasing female participation in computer science [51,57,62]. It has also been especially effective for retention in introductory courses [69].
Keeping female students with similar programming experience together in a class and not mixing novices with more experienced programmers was found to have a strong impact on the students' perception of difficulty [40]. The level of programming experience was determined using a mandatory assessment as part of the entry procedures. Other authors support the idea to split a class [49,23] or a practical lab [21] by experience as part of their overall efforts to improve participation and retention among women.
Personalisation is discussed by a number of scholars as effective tools to encourage the retention of female students. These include, for instance, sending personalised supportive emails with exam results which increased the retention of top performing students [63]; although sending the bottom-performing students emails with resources that might assist them had no effect. Knowing and addressing all students by name in class [47]; keeping focus on less experienced learners [23]; following up with individual students on any lack of attendance [13]; designing bonus assignments where appropriate [23] or modifying class content [23] are other practical examples of personalised actions focused on female students. To reduce the level of isolation for some students, seating can be shuffled randomly at the beginning of each class [21], or each female student placed in groups with at least one other woman [13].
There is evidence that in-class actions can encourage female students' participation. These include designing problem-based lessons that challenge and interest students [35]; encouraging a culture of discussion [23] or seminars where students of all genders can reflect on their own experiences [58].
Teaching Assistants (TAs) play their part in increasing students' confidence. The use of undergraduate students [13,33] and postgraduate students [58] as TAs have been part of the efforts that helped female retention. It was shown that a diverse TA team in terms of gender and race could help in influencing the retention of women [21]. In addition to providing help and increasing students' confidence, female lab assistants and TAs also serve as role models to younger female students [58].

Influence & Support
The gendered nature of computer science stereotypes can deter women from becoming interested in the field [6]. There are some simple ways to counteract this such as avoiding objects that enforce the stereotypical computer science environment (Star Trek posters, video games) [79]. There is a lot of evidence in the literature that role models and mentoring have a strong impact in this area. Both are discussed below.
Role Models have been shown to have a positive impact on encouraging and inspiring women to pursue computer science at different stages throughout the academic lifecycle. At recruitment, female students or faculty members can serve as ambassadors for annual fairs [64] or school visits [46]. Female computer science students from a UK university reported that their enablers were often male family members from the industry, as well as teachers in school [65]. Breda et al. [34] observed in a study with top performing female school students in France that the most effective role models were industry experts who did not emphasize gender disparities but instead delivered a positive image of their fields. Gutica [22] found that involving undergraduate role models of all genders for school students can improve the confidence of women in coding. University students have also been engaged to become digital ambassadors on social media [55]. Regular interaction with role models can be supportive for existing students as well, for instance in the form of informal meetings at the faculty [7,58]. Increasing female faculty provides more female role models for students also [39].
To target future students of computer science, Mentoring programs for secondary schools that link with existing CS students can be organised [58]. At college level, mentoring by more senior students helps to overcome programming difficulties [55,64]. A focus of mentoring the more inexperienced learners in their projects is also recommended [49]. Peer mentoring is discussed by a number of researchers as helpful to female students [66,25]. Matching students and mentors based on common interests outside of education has also shown success [24]. While female mentors can help with confidence, motivation and increased retention [37], some authors suggest that engaging both female and male mentors can be effective [80,24]. There are examples of mentoring networks with male mentors alongside the female in sub-fields, such as programming languages research [81] and music information retrieval [82].

Promotion & Engagement
This category includes actions related to advertisement and promotion of computer science to prospective female students and engagement with external stakeholders such as schools, other universities and industry.
Various strategies have been used for Advertising computer science courses with the aim of encouraging more female applicants. These include a national cinema campaign [12]; brochures [12], in particular with a focus on gender neutral design and presentation [64]; a dedicated career guidance portal [56]; online advertisement [39] and promotion through videos, articles and examples of projects which are designed to explain software development [55]. Promoting state of the art equipment in the computer labs has been effective [26] as has promoting good gender ratios [39]. However, the chicken and egg problem with gender ratios has been acknowledged and it is easier to attract women as female enrolment rises [39].
A range of outreach Events are recommended as effective ways to encourage women applicants into computing. Face-to-face information events, such as educational fairs in secondary schools [55], open days which focus specifically on the science [58], or women's days which focus on attracting female applicants [12,15,67]. There are many examples of outreach introductory classes to teach programming and coding to school girls [42,55,49,69,71]. A mid-year female technology camp is one of the most successful outreach activities at the University of Copenhagen, with 30% student application rate afterwards [55]. Other examples include summer camps with activities such as animation, virtual reality or robotics, which increased enrolment [68,70]. There are various way to encourage participation at the events. Students were asked to bring a friend they are comfortable to work with [22], some have mandated 50/50 gender balance [13] and others created an all-girls' environment in a summer school dedicated to computer science [71]. Others selected students to attend based on their mathematical abilities and motivation letter [48]. Workshops can include research activities as well [58]. There are also examples of weekend workshops for more mature women who might be willing to change careers [56]. The use of female faculty and students is important in outreach activities [58,55] Other types of events including external engagement which have helped in contributing to increased female student retention include Grace Hopper Celebration of Women in Computing [83], career days with guest speakers from industry [24] and industry networking events to facilitate the students meeting businesses [48].
Contests are another way of reaching out to prospective students [72,73,39]. Massachusetts Institute of Technology encourages female students to participate in the annual math competition and offers the winner the world's largest remuneration [39]. An innovative way to attract talented math students from secondary schools is a one day competition that takes place on a school day during school hours [72]. While this activity was well received by students, findings show that it also improved the perception and awareness of computer science amongst school teachers. Rewards can also be successful. To motivate students to enrol and to stay Tupper et al. [24] discuss reimbursement of course fees from a scholarship fund upon successful completion of technical modules.
Engagement with Industry has also demonstrated impact. Some examples of actions are inviting industry partners to university course teaching [47,41], co-organising workshops [80] or coding clubs supervised by industry partners [56]. Industry work placement has been shown to increase enterprise and employability skills and commercial awareness for IT students [45]. There is evidence that work experience is attractive to female students and can encourage "sticking with the programme" [75]. Some different approaches to placement which have had impact include international placements [13] and the dual programme of the women-only course at Hochschule Bremen [47], where students are taking internships throughout their course. Fryling et al. [74] found that a higher percentage of women undertook internships and had good retention and recruitment from non-computing majors.

DISCUSSION
One of the challenges with any initiative to encourage more female participation in computing is measuring the impact and knowing what actually works and results in good impact. A number of researchers recognise the challenge of assessing the direct relationship between the efforts taken and the outcomes on recruitment or retention [48,13,46]. It is difficult to directly relate a once-off event, even with positive feedback, to later recruitment or retention success. There are only a limited number of studies that include a control group and these studies tend to focus on one particular intervention or action [60,57,14,69]. It is also difficult to determine which action produced most impact where multiple actions have been implemented [21,13]. A number of researchers focus on measuring feedback, rather than impact. The level of satisfaction is a frequently mentioned form of feedback. It can be, for instance, satisfaction in the course of study and supervision [58], appreciation for the newly implemented way of teaching [44], or positive feedback from an event [13]. The usefulness of measuring satisfaction really depends on the significance of the question being answered by measuring satisfaction levels.
Most evaluation of impact is one of two types, empirical evaluations and assessment from experience.
Actions with quantitative empirical evaluation, which formed the majority of our findings, include preplanned once-off interventions or longitudinal studies which collected data and measured performance in some way. Those actions whose impact was based on experience were assessed more qualitatively. They could occur in teaching and learning events and be reported by observers or participants later. In some cases evidence was collected through interviews [12,39], focus groups [23,69], and during observation of students [44]. Anecdotal evidence has also been reported [49,44,69]. The primary way to collect the data is through surveys and questionnaires [21,13,15,24,21,38].
A variety of performance measures have been reported when evaluating the impact of actions. The most obvious ones are to directly measure female student enrolment rates [64,46] and retention rates [21,48,54,13]. However, significant changes to these statistics are unlikely in the short term, so a number of studies measure something more tangible in the short term such as students' perceived confidence in applying the skills learned [22,74,25] or their motivation [62,37] or sense of belonging [21,62]. Table 1 groups the papers based on evaluation types.
Studies have also reported that students' interest in computer science and their perception of computing changed following interventions [51,72,74]. Student performance before and after the intervention is also measured [21,41].
Most of these initiatives were implemented and measured over a short period of time but there are some longitudinal studies and observations too [48,54]. The importance of longitudinal data collection in order to observe trends in computer science participation is stressed [84], however a challenge with longitudinal studies is the effort and resources required to undertake them.
While most efforts discussed in this paper reported some positive impact, some researchers report some ineffective interventions. These include, for instance, open days that did not provide a clear return on investment [85], women only labs that were not attractive to female students [12] and an introductory computing course for business students that did not create a desired outcome [10]. There were also costly efforts to redefine the image computer science to be perceived as feminine which were not successful [12]. The message given in promotional materials was about the need for skills, as they claim, only women could offer to the computing industry -e.g., people skills or usability focus, as opposed to pure technical details. However, these unsuccessful efforts do not necessarily mean unsuccessful actions as these actions could potentially be effective in different situations and settings. The factors that impact on female participation are still not clearly understood [10]. Culture also plays a role and to increase the chance of success it is recommended to design or modify initiatives with respect to the local context [46]. Additionally, initiatives are often implemented and evaluated by the same people. To avoid negative publicity, unsuccessful results might stay in the shadows without being published [86]. In some rare cases an external evaluator was involved in the project [68,24]. The evaluator conducted the end of project survey but also monitored and reported on the progress of the initiative and organised yearly student focus group meetings, which facilitated changes based on students' recommendations [24].
To address the challenge with evaluation, a framework for the assessment of STEM projects for secondary school and university students has been developed [87,88]. Strategic steps, followed by practical recommendations, are outlined within four main phases: process design, decision on questions and answers for measurement of success, questionnaire development, and data evaluation.
There is evidence in the literature that the diversity of the student group should be considered when planning actions. Some actions have been shown to be successful only with top performing students [34,63] or with motivated students [43]. Others show negative impact on male students [54,14]. In terms of encouraging women to pursue creative IT courses, e.g., game development, there is evidence that women tend to choose subjects they have previously had experience with [53]. It is therefore important to get them exposed early on. Finally, research shows that once the initiatives stop, the numbers of women can drop again [48]. Hence, it is crucial to maintain continuous efforts to improve female participation in computing disciplines.

CONCLUSIONS
While there appears to be no universal cure to address the "leaky pipeline", this research provides a comprehensive review of actions taken in a range of environments to enhance recruitment and retention of women in computing disciplines. Actions have been collated from a broad range of sources and categorised into four groups which are discussed in detail. Categorisation of gender balance actions is critical, providing technical higher education organisations and indeed any stakeholders in STEM with a common language and understanding around action types and scale which will facilitate discussion and comparison of gender balance actions. Our review also examines practices around measuring the impact of gender balance actions. We identify the challenges of evaluating the impact of these actions. This work leads to our next project, the development of a toolkit of actions, with a mechanism for evaluation, to promote the recruitment and retention of women in undergraduate computing disciplines.