{'fraction_non_alphanumeric': 0.03018605989808184, 'fraction_numerical': 0.016303519731829957, 'mean_word_length': 4.805779437782583, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
TRAP: A Predictive Framework for Trail Running Assessment of Performance

Trail running is an endurance sport in which athletes face severe physical challenges. Due to the growing number of participants, the organization of limited staff, equipment, and medical support in these races now plays a key role. Monitoring runner's performance is a difficult task that requires knowledge of the terrain and of the runner's ability. In the past, choices were solely based on the organizers' experience without reliance on data. However, this approach is neither scalable nor transferable. Instead, we propose a firm statistical methodology to perform this task, both before and during the race. Our proposed framework, Trail Running Assessment of Performance (TRAP), studies (1) the the assessment of the runner's ability to reach the next checkpoint, (2) the prediction of the runner's expected passage time at the next checkpoint, and (3) corresponding prediction intervals for the passage time. To obtain data on the ability of runners, we introduce a Python package, ScrapITRA, to access the race history of runners from the International Trail Running Association (ITRA). We apply our methodology, using the ITRA data along with checkpoint and terrain-level information, to the``holy grail'' of ultra-trail running, the Ultra-Trail du Mont-Blanc (UTMB) race, demonstrating the predictive power of our methodology.


Introduction
What is trail running? According to the Cambridge Dictionary, trail running is the sport or activity of running along trails. More generally, the term may also include the activity of running on trails possibly with sections of paved road, although this is sometimes referred to as mountain running. The term is often used interchangeably with cross country (xc) running, although the latter typically refers to competitive running. Last, trail running differs from ultra-running, that is running for distances longer than a marathon (42.195 km); indeed, ultra-trail running is only a special case of both ultra-running and trail running. However, these definitions turn out to be rather technical and specific when compared to the broader viewpoints of race organizers, sponsors, and media outlets. According to the International Trail Running Association (ITRA), the largest trail running organization worldwide, trail running is a sport that takes place in a natural environment and enhances a shared sense of community among runners. Other definitions of trail running emphasize the runner's connection with nature, respect for each other, challenge, risk, and freedom. Trail running, which in 2015 has been recognized by the International Association of Athletics Federations (IAAF) as a track and field sport, embodies all these definitions.
Although all these terms may sound familiar today, they would have definitely sounded less familiar ten years ago. Indeed, the world of trail running has experienced an exponential growth in terms of the number of athletes, races, news and media coverage, and interest in the sport industry only in the last decade. Despite the recent surge in popularity, trail running has been practiced for decades. One of the most famous trail running competitions, the Western States Endurance Run, a 161 km race with 5500 m of elevation gain in California, was run for the first time in 1974; at that time, it was only known as a horse race and a participant decided to run instead of riding his horse. In 2019, race organizers have used a lottery system to select approximately 300 runners out of a pool of over 5000 applicants. Similarly, Ultra-Trail du Mont-Blanc (UTMB), with 171 km and 10000 m of elevation gain crossing France, Italy, and Switzerland, started with only a few participants in 2003 and now manages admissions through a convoluted lottery system. The cases of Western States and Ultra-Trail du Mont-Blanc are not isolated: obtaining admission to the most well known races is becoming increasingly harder over time. However, the number and the spectrum of characteristics of trail running races is also becoming wider. Figure 1a shows the number of races registered in the ITRA circuit from 2010 to 2018. The number of races recorded on the website grew 16 times in only 8 years, from 275 races in 2010 to 4439 in 2018. Although this estimate is affected by selection bias, as the participation to the ITRA circuit has also been growing over time, it still provides some insight into the rise in popularity of trail running. The stronger presence of ITRA in the European continent is noticeable in Figure 1b: the country with the highest number of races in the 2010-2018 period is France, followed by the United States, Spain, Italy, and China.
The current popularity of trail running is likely due to a variety of factors, among which the extreme nature of this sport. Indeed, most of the famous races, such as UTMB, Western States, Hardrock 100, Lavaredo Ultra-Trail (LUT), Tor des Géants (TOR), Marathon des Sables, and Ultra-Trail Mount Fuji are all ultra-trails. Qualification for these races requires runners to have already completed races of comparable difficulty. For instance, in order to register for UTMB runners must have scored 10 ITRA points (an index adopted by ITRA) in at most 2 races. Still, even among these highly skilled runners, the fraction of dropouts in these races is large. Table 1 shows that, for some of the races mentioned above, this percentage ranges from 20% to 40%, which means that 1 or 2 out of 5 runners do not conclude the race. Why such a large fraction of runners drops out? Intuitively, dropouts are proportional to the difficulty of the race in terms of finishing time, elevation gain, distance, and environment. For example, longer time spent in the mountains is linked to a higher likelihood of incidents: runners are more likely to suffer from dehydration, hypothermia, injuries, and sleep deprivation due to fatigue. We explore these relationships in Section 5. Although these famous races get more visibility in the media, most races feature shorter distances, as shown in Figure 1a. In these cases, some of the risks (e.g. sleep deprivation) are mitigated, but it is still important to fully evaluate the threats imposed by the environment (e.g. weather) in order to minimize the likelihood of tragic events and manage the organization of resources along the trail.
The organization of trail running races requires extensive knowledge of the terrain and of the runners' skills. For instance, organizers need to detect runners that are experiencing difficulties and consequently either contact them or take action. Moreover, they need to properly allocate medical staff and equipment, volunteers, food and drinks at aid stations, buses. The current approach is problematic for (at least) the following three reasons. First, knowledge of the terrain and of the runner's skills is either crowdsourced from several individuals with different areas of expertise or even obtained from just a handful of organizers. Second, this knowledge is transferred across races via heuristics. Third, runners' ability is typically estimated, again via heuristics, solely based on past (if available) and current performance only during the race. Such an approach is neither scalable nor transferable: knowledge does not improve with data and cannot be transferred races and individuals.
While we understand that the trail-running setting may sometime require knowledge that goes beyond what is coded and available in the data, we recognize the importance of introducing a data-driven benchmark that serves as an aid in the organization of these races. To our knowledge, such a benchmark does not currently exist. Our main contributions are: • the implementation of a Python package, ScrapITRA, for scraping the data available on the website of the International Trail Running Association (ITRA). Through ScrapITRA, a data set of over 1.7 million runners in more than 15 thousand races can be accessed. This rich data set is, to our knowledge, the largest data set on trail running ever collected for research purposes. The package is available on Github at https://github.com/ricfog/ScrapITRA.
• a novel framework, Trail Running Assessment of Performance (TRAP), for assessing runners' performance both before and during the race. Leveraging statistical machine learning tools, our methodology targets three quantities related to the runner's (a) expected passage time; (b) probability of dropping out; (c) prediction interval for passage time.
These quantities are computed for the following checkpoint (i.e. where runners have their bibs scanned) based on the runner's position. However, the framework is readily extensible to multiple checkpoints ahead at arbitrary positions. We present the TRAP framework through a case study of the Ultra-Trail du Mont-Blanc (UTMB) race for the editions 2015, 2016, 2017, and 2018.
The paper is structured as follows. Section 2 contains related work. Section 3 contains description and results of the ScrapITRA package. Section 4 contains exploratory data analysis for our case study, UTMB. Section 5 contains description and application of the TRAP methodology on UTMB with the various model results presented in Section 6. Section 7 contains final discussion and directions for future work.
In the following, we report all statistics with standard deviations, when they are raw counts in the data, or with confidence intervals, when they represent estimates. Both standard deviations and confidence intervals are reported inside parentheses. We omit standard deviations and confidence intervals when their rounded magnitude and length respectively are smaller than 1. We also occasionally omit these measure when they are not relevant to our discussion; for example, as we simply list the runners' nationalities in the UTMB race, we choose not to report confidence intervals.

Related work
The literature on trail running inherits results from two large bodies of works on ultra-trail running and track (or road) running. However, most of these studies focus on the medical aspects of these activities and therefore their goal clearly differs from the purpose of our paper. Although this literature would deserve an extensive review, we limit this section to the papers whose settings and scopes are closest to ours.
Researchers have examined to a great extent the role of pacing in running, that is how athletes allocate their own energy during the race. We start by reviewing results on marathons. Multiple studies report that men are more likely to slow their pace compared to women, even with several different definitions of pacing. (March et al., 2011) study the effect of age, sex, and finish time on marathon pacing, which they define as the ratio of the mean speed in the last 9.7 km and in the first 32.5 km. They find that older, faster, and female runners are more likely to keep the pace steady compared to younger, slower, and male runners respectively. Using a similar definition of pacing, (Trubee et al., 2014) analyse the 2007 and 2009 editions of the Chicago marathons and show that age, sex, finish times, and heat stress are significant predictors of pacing; in particular, ceteris paribus, heat stress leads to greater variability in pacing. (Haney Jr and Mercer, 2011) use data of Las Vegas and San Diego marathons uploaded on Garmin Connect and show that pace, defined as the ratio of the standard variation and the mean of the total duration, had less variability for faster rather than for slow runners. (Santos-Lozano et al., 2014) analyse a large data set containing the results of several editions of the New York City marathon and they find that both men and women try to avoid fast starts in order to keep their pace constant in the second part of the race. (Deaner et al., 2015) report that, after controlling for a variety of benchmark features, men are more likely than women to slow their pace, defined as the ratio of finish times of the second and first half marathons, for 14 marathons in the US during 2011. Using a data set of one million participants in several marathons, (Krawczyk and Wilamowski, 2017) find that men, youngest, and oldest participants tend to slow more in the second half. This finding partially matches those of (March et al., 2011) and (Trubee et al., 2014) mentioned above. (Smyth, 2018) perform a large-scale analysis of over 1.7 m recreational runners and find that women tend to pace their race more evenly than men. Pacing also seems to be tied to optimism. (Krawczyk and Wilamowski, 2017) analyse predicted and true finish times for the 2012 Warsaw marathon, and they find that optimism is associated with more slowdown in the second half of the marathon. (Hubble and Zhao, 2016) use data from the 2013 Houston Marathon to show that men are more overconfident in their future performance compared to women and their pace decreases more substantially in later stages of the marathon. The body of work on longer distance races is less rich, but nonetheless contains partial answers to some of the questions raised in the marathon setting. For example, using data from the 1995 IAU World-Championship, a 100 km race organized by the International Association of Ultrarunners (IAU), (Lambert et al., 2004) find that all participants decreased their speed during the race, with faster runners showing a smaller and delayed decrease. The effect of pacing on performance in ultramarathons is also found and not found by (Hoffman, 2014) and (Kerhervé et al., 2016) respectively, with only the first study reporting the presence of an effect. (Bartolucci and Murphy, 2015) fit a finite mixture latent trajectory model on runners' performance in the 2013 IAU World-Championship, a 24-hour continuous race. Interestingly, their results show that runners are more likely drop out in the middle of the race; we observe a similar pattern in our case study on UTMB. They also observe that runners' speed gradually decreases only to increase towards the end of the race, similarly to what was reported by (Lambert et al., 2004).
Gender gaps in performance have also been analysed. (Peter et al., 2014) use 35 years of several 24-hour ultramarathons held around the world to study gender gaps in performance. The authors conclude that, although the gap between the top men and women has been decreasing over the years, it is unlikely that the gap will be closed in the near future. In addition, they find that ultramarathoners typically achieve their best performance after the age of 35. ) reach a similar conclusions on an analysis of 50 years of ultramarathons from 50 km to 1000 km long. (Coast et al., 2004) analyse world records for races from 100 m to 200 km long and they conclude that the gender gap is larger for longer events; however, they warn that this result may be confounded by the fact that only a few women participate in long distance races.
Many studies are focused on the detection of significant predictors of finish times; however, most of them belong to the medical literature. Here we report only a few of these studies. (Ely et al., 2007) study the impact of weather on performance. Using longitudinal marathon and weather data for seven cities, they find that an increase in temperature from 5 to 25 degrees Celsius leads to lower levels of performance in the marathon. Similarly,  regress finish times of half marathons on percent of body fat and running speed during training with linear regression to test and possibly improve the coefficients of these equations proposed in the literature Rüst et al., 2011;Friedrich et al., 2014). In a review of the literature on equations for marathon prediction times, (Keogh et al., 2019) identify 36 studies with 114 different equations.
Our study is close to the body of work on prediction of finish times, but focuses on arbitrary distances. In addition, our target measures of interest have not been discussed yet in the literature. Our work is, to our knowledge, novel both in terms of the setting (trail running) and the methodology (TRAP). However, we are aware of a service provided by a proprietary software, LiveTrail, similar to part of the framework presented here. LiveTrail offers the necessary technology and equipment for race organizers and supporters to monitor runners' performance during the race. Elite runners are typically given GPS devices that allows real-time tracking of their position, while all other runners get their bibs scanned only at checkpoints. After the scanning, LiveTrail outputs a prediction of the expected passage time for the next aid station. In section 5 we present our open-source algorithms for this task. We believe that this part of the technology is easy to implement through our framework and should be available to race organizers at no costs. We also find that expected passage time is often not informative in the context of trail running, in which prediction tasks are difficult. Moreover, besides the estimate being unreliable, expected time is not the measure of central interest. Indeed, • supporters (e.g. family, friends, or sponsors) need to be present at the following aid station to help the runner. Therefore they are interested in a minimum passage time.
• race organizers and physicians need to be alerted if a runner has suffered from a serious injury and cannot reach the next checkpoint. Therefore they are interested in a maximum passage time.
For this reason, we extend the framework by proposing upper and lower bounds for the passage times via prediction intervals discussed in Section 5.

ScrapITRA
In this section we describe ScrapITRA, a Python package for scraping data from the website of the International Trail Running Association (ITRA). The structure of this section is as follows. First, we provide minimal background on ITRA, then we describe the package. Second, we present some exploratory analysis on one of the data sets that we scraped using the package.
The organization. ITRA is the world's largest trail running organization. Funded in July 2013, ITRA is nonprofit and its central mission is the promotion of trail running as a sport. Its rankings are considered the benchmark for the evaluation of races (e.g. difficulty) and athletes (performance) in the trail running community. However, ITRA is not the only organization in this space. Notably, UltraSignup is another organization with a large community in North America. ITRA and UltraSignup differ in their ranking systems, in the services offered to their members, and in the way memberships are managed. For instance, ITRA requires race organizers to pay for membership through a flat fee whose cost ranges from 100 to 550 Euros per year depending on the size of the organization but independently of the number of races. Through the membership, ITRA provides an evaluation of the race's difficulty and of runners' performances. All data is recorded in their database; for timing services, many races in the ITRA circuit use the LiveTrail software described in Section 2. UltraSignup, instead, charges a fee for each runner but provides a free timing software. We focus on ITRA because, besides being the largest organization, its circuit of races also includes the case-study of this paper. The main features of ITRA are: (1) the evaluation of the race's difficulty and the system of "points"; (2) the "performance", or ranking, system.
We start with (1). Each race in the ITRA circuit is assigned "ITRA points", a score proportional to its difficulty. The assignment of the number of points is based on a measure called "km-effort". This metric corresponds to the sum of the distance (km) and the hundredth of the positive elevation gain (m), minus penalties proportional to the number of refreshment points and laps. The km-effort difficulty of the race is then converted into points through a standard table, whose categories correspond to the ones used in Figure 1a. The maximum number of points is capped at 6. For instance, UTMB is approximately 171 km long with 10000 m of elevation gain and no laps, therefore the km-effort measure is around 171 + 100 = 271 which translates into 6 ITRA points. Every race finisher gains the corresponding points and can use them to qualify for future -and harder -races. Indeed, most races above 50 km require participants to have a minimum number of points to qualify. As an example, only to enter the lottery of UTMB runners need 10 ITRA points obtained in maximum two races; this means that they need either both a ≥ 210 and a 115 − 154 km-effort race or two 155 − 209 km-effort races. The points can only be gained through races in the ITRA circuit.
ITRA has adopted a scale, called the "ITRA performance index", that allows for comparison of athletes' performances across races (2). Every race finisher gains an ITRA score, which is a function of the km-effort, the technical difficulty of the race, and the hypothetical best finish time in that race. 1 Multiple scores are then aggregated into the athlete's general and categories indexes; the categories breakdown follows the legend of Figure 1a, but in terms of km-effort. Both indexes are calculated as the mean of the 5 highest ITRA scores weighted decreasingly by time and up to 36 months before. The index ranges from 0 to 1000 points, with elites and amateurs scoring 800 and around 500/600 respectively. Despite the drawback of being proprietary knowledge, the ITRA index has become the benchmark for the evaluation of athletes' performances in trail running.
We are interested in both races results and evaluations, and athletes' performance and scores. However, the ITRA website lacks an API. This means that one can download data only by manually searching and typing this information on the interface of their website, and then manually coping it into the desired format. We aim to make the data set more easily accessible through ScrapITRA.
The package. ScrapITRA is a Python package that leverages Selenium WebDriver and Beautiful Soup (Richardson, 2007). Selenium is a framework for testing web applications, while Beautiful Soup is a Python package for parsing HTML and XML documents. In ScrapITRA we employ the two packages sequentially. First, we dynamically navigate through the website using Selenium WebDriver, either on the local or on a remote machine. Then we scrape the static webpage through Beautiful Soup. For instance, in order to obtain the list of finishers for the Western States Enurance Run of 2018 in csv format, the following operations need to be performed: (1) go onto the webpage https://itra.run/page/328/Results.html; (2) select "Personalised" period and insert the correct dates; (3) select the correct race; (4) download the table the appears on the webpage and transform it into csv format. Between all these operations, it is always necessary to wait for the webpage to load completely. We have automatized these actions through ScrapITRA.
We now describe the functions in ScrapITRA: · get_browser(webpage, head=True, path_driver=None): this function initializes Selenium WebDriver and accesses the indicated webpage, with the options of showing (head=False) or hiding (default) the web browser. The WebDriver used in this package is Mozilla's Geckodriver. Its location is specified through path_driver or left unspecified if the driver is placed in the relative path Drivers/firefox_mac. The driver can also be used on a remote machine without root privileges via X11. · login(username, password, browser): using the specified browser initialized by the function get_browser, users that are members of ITRA can log into their accounts on the website. The membership provides access to the scores of individual races. · get_races(browser, start_date, end_date, path=None): this function returns a list containing information and results of all finishers together with their demographics for competitions that took place sometime between the specified start and end dates. The information about the race contains name, location, distance, elevation gain, and date. The information about runners includes name, finish time, ranking, sex, bib number, and nationality. The output is a list. · get_runner(browser, first_name=None, last_name=None, nationality=None): specifying first name, last name, and nationality (or one of them), a list of webpages for the runners that match the criteria is returned. If none of these values is specified, then the returned list is empty. · scrape_history(webpage, browser): for an individual webpage, two lists, one containing results for all past races and another one only relative to the best performances, are returned. The package also contains a script named write_races that allows for execution of the first three functions listed above through a one-line input on the command line.
The race data set. We used the get_races function to download results of all races available on the ITRA website from 2010 to the end of 2018. The downloaded data set contains over 2.9 million individual results 2 in more than 15 thousand different races. Figures 1a and 1b are based on this data set. We now present a few other findings from our exploratory data analysis on this ITRA data set. We describe, in sequence, results regarding races and runners. While the following data analysis is by no means extensive, it serves the purpose of presenting the main features of the data set.
We start with the information on races. We find a relationship between the race's distance, elevation, and period of the year. Figure 2a shows the histogram distribution of races across months for the years from 2010 until 2018 for the five countries in the data set that have at least 500 races each. Every bar represents the ratio of the numbers of races in a given month and in the corresponding year. We notice some degree of heterogeneity among the distributions. Among these five countries, only in France, Italy, and Spain the majority of races is held in the months from April to October. In Figure 1b, which shows the relationship between distance and elevation for ITRA races, we observe that competitions in these countries feature high elevation gains and therefore are probably held on mountain trails. For instance, races in Italy feature, on average, 56 m/km of elevation gain thanks to the fact that they are mainly held in the Alps. Consequently, the lower number of races in winter is likely due to factors such as low temperatures and snow on trails. Interestingly, for these three countries there appears to be a sizeable decrease in the number of races in the month of August; this effect may be explained by the high temperatures or by workers' vacations. The distributions of races across months for the US and China have different shapes. In the US, races are more evenly distributed across spring, summer, and fall. Figure 1b shows that races in this country are typically longer but have less elevation gain, with only 32 m/km. The absence of steep ascents, and consequently the presence of trails that are more flat, may allow for an easier organization of races in all seasons. Differently, in China most races are held either in spring or fall, and only a few of them are in the summer. Our explanation of these existing patterns between period and elevation gains focuses on the geographical location rather than on organizers' or runners' preferences. However, this explanation can not be ruled out. Indeed, in Figure 1b we notice that both the US and China have only a few races with distance between 100 and 160 km (100 miles). This effect may effectively reflect runners' preferences for the 100 miles distance.
We now present two results at the runner-level data. First, female participation appears to be decreasing with distance. Figure 3a shows the proportion of female runners, among all runners, in races belonging to the indicated binned distances. For example, 33% ([33% − 34%]) of all runners in trail races with distance in the range 0-24 km is female, but this proportion drops to only 22% ([21% − 22%]) for the distance bin 25-44 km. The fraction of women slightly decreases also for longer distances. Indeed, in the ITRA data set 21% of all runners are women, and 28% of them is in races belonging to the 0-24 km, compared to only 15% of all men running these races. The pattern persists also with respect to elevation and elevation gain per distance. Overall, this suggests that the proportion of Second, the performance gap between elite and average runners appears to be increasing with distance. Figure 3b shows the distribution of relative finish times, computed as the ratio of the difference between individual finish time and best time in the corresponding race and the difference between worst and best time in the same race, across distance categories. As distance increases, the shift of the mass towards the right means that the time performance of most runners in races becomes closer the performance of the worst runner in the race. For example, if we consider races in the categories 0-24 km and 75-114 km, half of the runners arrive in approximately 35% and 56% of the time gap between the best and worst participants respectively. This result may be due to either (1) the fact that most runners slow their pace with distance, while elites do not; or (2) the large fraction of dropouts due to the presence of time barriers in races with longer distances; or (3) selection bias because runners in longer races are more trained, e.g. ITRA points are required. The literature supporting the first effect in other settings has been discussed in Section 2. While we can not check the validity of the three hypotheses in this data set because we do not observe dropouts and partial times, we investigate them in our case study of UTMB in Section 4. For instance, we observe that also the distribution of arrival times for UTMB is skewed to the left, similarly to the pattern highlighted in Figure 3b in the last category.

Exploratory Analysis of Ultra-Trail du Mont-Blanc
Ultra-Trail du Mont-Blanc (UTMB) is an ultramarathon 3 held in France every year during the last week of August. UTMB is part of the Ultra-Trail World Tour, an international circuit of ultra-trail running races. In this race, participants run along the Tour du Mont Blanc hiking path: starting from Chamonix, they cross the borders of Italy and Switzerland, and run back to Chamonix making a complete tour of the Mont Blanc. Since its first edition in 2003, the race has been modified over the years and the last seven editions (2013-2019) featured a distance of approximately 170 km with over 10000 meters of elevation gain; the profile for the 2018 edition is shown in Figure 4. While elite runners typically complete the race in about 20 hours, most runners cross the finish line in more than 40 hours, right before the time barrier of 46 and a half hours. The participation is limited to approximately 2500 runners and it requires a minimum of 10 ITRA points scored in at most two races. Admission is based on a lottery system. Thanks to its extreme nature and the spectacular views, UTMB is seen by many runners as the pinnacle of a career in trail running. We present the race through an exploratory data analysis in order to lay the groundwork for the TRAP predictive framework in Section 5.
The following analysis is based on three separate data sets: (1) the ITRA data set described in Section 3; (2) a second data set, downloaded from Kaggle, containing the results of UTMB for the editions 2015-2017; (3) a third data set with the results of UTMB for the 2018 edition, that we scraped from the race's website. Sources (2) and (3) contain information on passage times at checkpoints throughout the race. Instead, we use source (1) to leverage information on the runner's past race history. Indeed, since UTMB requires ITRA points, all runners competing in the race need to have run at least two races in the ITRA circuit; this ensures that we have prior records for all of them. 4 The following analysis is based on the editions of UTMB 2015UTMB , 2016UTMB , 2017UTMB , and 2018. We consider this time window because the path of the course was slightly modified in 2017 and this makes the design of a predictive framework both more interesting and challenging.
The total number of participants in our UTMB data set is 8689, with approximately 2600 runners per edition. For each edition, we dropped almost 400 participants because we could not automatically match their names with records from ITRA. The analyses in both this section and in section 5 are based on the reduced data set. However, as we show later, the ITRA data set is not significantly informative of the runners' performance in the race, and therefore this information could be omitted without compromising the quality of the predictions in the TRAP framework.
How long does it take for participants to complete the race? The start of the race is at 18:00 on Friday and the time barrier is at 16:30 on Sunday, which translates into a maximum allowed time of 46.5 hours to complete the race. Figure 5b shows the distribution of finish times for the years 2015-2018 through stacked density histograms. While the winner typically crosses the finish line in Chamonix in 20 hours, only 5-7% and 39-43% of the runners complete the race in less than 30 and 40 hours respectively. The mean arrival time is approximately 40 hours and it does not differ significantly across age categories and genders. This means that most athletes complete the race with finishing times close to the time barrier and consequently spend two nights running out on the trails. Besides the final time barrier in Chamonix, there are multiple others located at some of the checkpoints,  highlighted in red in Figure 4. If a runner does not reach the checkpoint before the time barrier has passed, they are forced to withdraw from the race. Time barriers allow course organizers to concentrate resources and better monitor runners along the path.
How often is runners' performance recorded? As mentioned above, the path of the race was modified in 2017 and therefore the 2015 and 2016 editions differ from those of 2017 and 2018 in the positions of some of the checkpoints. For this reason, we restrict the checkpoints considered from all the checkpoints shown in Figure 4 down to only the 24 listed on the vertical axis of Figure 5a. While such a choice makes the prediction tasks of Section 5 harder, it simplifies the presentation of the exploratory data analysis in the current section. Figure 5a shows the density estimates for the runners' passage times at the 24 checkpoints. Each passage time is calculated as the difference between cumulative passage times (i.e. since the beginning of the race) at two consecutive checkpoints. For example, it takes approximately 2 hours for most runners to arrive to checkpoint 1 (Delevret) from the start of the race in Chamonix, and from 3 to 4 hours to reach 18 (La Giete) from 17 (Champex Lac). These estimates are computed considering only all runners still in the race, or equivalently those that have not dropped out yet. There are at least three notable patterns in Figure 5a. First, since the four editions of the race differ in the 8th (Lac Combal), 20th (Les Tseppes), and 22nd (Col des Montets) checkpoints, the corresponding partial times do not match. Second, despite different environmental conditions (e.g. weather) and dropout rates (37%, 42%, 33%, 30% for years 2015-2018 respectively) across the editions, the distributions of runners' passage times remain similar for the checkpoints whose positions did not change. Third, mean and skewness of the distributions greatly vary across checkpoints. For instance, most runners run the 8 km distance between checkpoints 3 (Les Contamines) and 4 in less than 2 hours, but it may take them from less than 2 up to 5 hours to cover the 14 km distance separating checkpoints 16 (La Fouly) and 17 (Champex Lac). As we show later, this variability is a function of the distance between checkpoints, cumulative distance from the start, and elevation gain.
Can runners access food and drinks? UTMB is run in semi-autonomy, which means that runners are required to bring a certain amount of mandatory gear (e.g. jacket, cellphone, food, water), food, and drinks with them, but they can also get extra resources provided by the race organization along the course path. While drinks are present at all checkpoints, food is available only at checkpoints 2, 3,4,6,8,11,14,16,17,19,21,24 as in the mapping of Figure 5a. We call these checkpoints aid stations. Beds are available at checkpoints 6, 11, 17, 24. Runners can meet supporters (e.g. family members, friends, or sponsors) and receive additional gear or food at checkpoints 3, 11, 17, 19, and 21. Runners can also send personal gear to checkpoint 11 prior to the start to the race. Buses provided by the organization take runners that have withdrawn from the race back to Chamonix and are available at checkpoints 2, 3, 6, 8, 11, 14, 16, 17, 19, and 21. Medical personnel is present at most of the aid stations and, in case of emergencies, runners can always request medical help by calling the organization.
How likely is it for runners to complete the race? Dropout rates of UTMB are consistently high across all editions considered, ranging between 30% and 42%. Not surprisingly, Figure 6a shows that dropouts are concentrated in a few small segments of the course: we observe large numbers of withdrawals between the checkpoints 3-4 (Les Contamines-La Balme, only for 2015/16), 8-9 (Lac Combal-Mont Favre, only for 2015/16), 11-12 (Courmayeur-Bertone), 14-15 (Arnouvaz-Col Ferret), and 16-17 (La Fouly-Champex Lac). Although we do not have access to the exact locations and reasons behind the individual withdrawals, we can infer with some degree of certainty that most of them actually occur at checkpoints. Indeed, such large rates of dropouts may be explained by runners' fatigue or non serious injuries (e.g. blisters, ankle sprains), i.e. injuries that do not prevent them from walking and reaching the checkpoint. We note that all these checkpoints correspond to aid stations at which both food and transportation provided by race organizers are available, that are easily accessible by supporters, and for which the distance to complete the race is still relatively long. Consequently, runners that are tired may have strong incentives to stop at these aid stations rather than on the trail; this explanation is supported by the finding that a dropout is more likely if the runner has been decreasing pace, as reported in Section 5. At the same time, a small fraction of the dropouts is often due to serious injuries; these events occur on the trail and require medical help. We also note that the total fraction of dropouts is slightly higher for women compared to men, with 42% ([36%, 47%]) and 35% ([33%, 37%]) respectively, and lower for younger runners (SE category) compared to older runners (V category), with 29% ([26%, 32%]) and 39% ([37%, 41%]). This finding is interesting, as it shows that younger athletes complete the race more often despite likely being less experienced. Last, among the countries with at least 300 runners in these editions (France, Italy, Japan, Spain, UK, US), dropout rates appear to be highest for Italy with 40% ([35%, 46%]) and lowest for Spain with 28% ([28%, 35%]). At the checkpoint level, we do not observe any significant difference in the locations of the dropouts across sexes and age categories.  Figure 6: (a) Cumulative proportion of athletes still in the race at the indicated checkpoints out of all athletes out of all athletes that started the race. (b) Fraction of runners, out of all the runners that completed the race, that gain at least one position in the total ranking, by category.
How does runners' pace vary throughout the race? Differently from the marathon setting, the uneven landscape of UTMB does not allow us to compare pace across different segments of the race. Indeed, each segment between checkpoints features unique distance, elevation, and difficulty, among several other characteristics; not all these variables are recorded in the data (e.g. difficulty) and consequently any analysis of pace across checkpoints requires assumptions on such variables. For this reason, instead of analysing (absolute) pace within groups, we focus on how (relative) pace compares across groups. In Figure 6b, we quantify these changes across age categories in terms of the fraction of runners that gain at least one position in the general ranking, out of all the runners that complete the race. A similar pattern can be observed analysing rankings based on partial times at checkpoints, but we do not report the related plots here. There are two notable patterns in Figure 6b. First, the fraction of runners that gain at least one position in the ranking is larger than 50% across almost every checkpoint for all categories. This means that runners are more likely to gain, rather than to lose, positions in the ranking. Since such a gain needs to be offset by the loss of multiple positions by a lower fraction of runners, this implies that a small proportion of the runners may substantially decrease their pace. Second, younger runners (SE category) surprisingly appear to be more likely to gain at least one position towards the end of the race compared to older runners (V1 and V2 categories). We also notice that performance, in terms of the speed at checkpoints, of runners in the SE category increases more along the race than for the other two categories. We also investigate the relationships across genders and nationalities, but we do not find any significant pattern.

TRAP framework
Our TRAP framework focuses on two random variables of interest for each runner i ∈ [n] at checkpoint t in a particular trail race with T total checkpoints: 1. Y i,t ∈ R : passage time for runner i at checkpoint t, and 2. D i,t ∈ {0, 1} : whether runner i drops out of the race at checkpoint t.
Given X i,t , a data structure representing the information about runner i from the start of the race through checkpoint t, we introduce models for predicting three key quantities: : corresponding 95% prediction interval for the checkpoint passage time; 3. P (D i,t+1 |X i,t ) : a probability mass function describing the likelihood that runner i drops out of the race at checkpoint t + 1.
While our framework can be easily extended to predict multiple checkpoints ahead, we focus on predicting runner's performance at the next checkpoint. We proceed as follows. In Subsection 5.1 we provide an overview of the different types of features used in modeling. In Subsection 5.2 we describe the modeling approach. In Subsection 5.3 we describe models validation. In Section 6 we summarise the results.

Features
We constructed features for our data structure X i,t that can be clustered in three groups, based on checkpoint-level (Table 2), runner-level (Table 3), and lag information (Table 4). Checkpoints characteristics, e.g. distance from starting point, altitude, food, and medical assistance, may vary across races and editions. To account for this heterogeneity, we manually encoded checkpoint-level information from tables and figures (e.g. Figure 4) available online. A summary of these features is listed in Table 2.
Runner-level information was scraped from the UTMB and ITRA websites using our package ScrapITRA. The scraped data included demographics such as gender, nationality, and age category, together with information on past race history. Based on this set of information, we constructed predictors that range from for the runner's performance in previous races to features that characterize the types of races the runner took part in. The features that fall under this category are detailed in Table 3.
When predicting runner's performance throughout a race, one checkpoint at a time, there might exist a temporal dependency between the considered features and passage times. For example, one might expect that a runner underperforming in previous checkpoints will continue to underperform in upcoming checkpoints as well. For this reason, we include features based on lagged versions of the passage time: (lag 1) Y i,t and (lag 2) Y i,t−1 when predicting the result at the next t + 1 checkpoint (both Y i,t+1 and D i,t+1 ). Additionally, we constructed several types of interaction variables between checkpoint-level information and the lagged passage time as presented in Table 4. We show in Section 6 that the inclusion of the lagged features greatly contributes to the predictive power of the TRAP models.

Models
Given the potential features considered for the data structure X i,t , the models in the TRAP framework must be able to address the following conditions: (1) be high-dimensional, (2) account for non-linear relationships, and (3) account for interactions. We considered many different types of variables in Subsection 5.1, resulting in a high-dimensional problem with the potential for non-linear relationships and interactions, e.g. types of runners perform better at certain types of checkpoints. Thus, we considered models that capture these aspects of the data.
For modeling both the expected passage time E[Y i,t+1 |X i,t ] and probability of dropping out P (D i,t+1 |X i,t ), we established a baseline intercept-only model without any of the features described in Section 5.1. The intercept-only model serves as a reference point for the model evaluation process described in Subsection 5.3.
Next, we considered the LASSO regression model (Tibshirani, 1996) to handle the high-dimensional nature of the problem, while providing fast and interpretable results. We use the glmnet implementation (Friedman et al., 2010) in the R programming language (R Core Team, 2017). For the expected passage time model we used the default mean-squared error (MSE) measure to choose the regularization parameter with cross-validation (CV), while for the probability of dropping out model we used logistic LASSO regression with the area under the curve (AUC) measure. For both models we chose the one standard error regularization penalty.  Year of last ITRA race time_last_race Time (months) since last ITRA race In order to account for potential non-linear relationships and interactions, we additionally explored the usage of tree-based models. We considered both random forests (Breiman, 2001), via the ranger package (Wright and Ziegler, 2017), and gradient boosted trees via XGBoost in R (Chen and Guestrin, 2016;Chen et al., 2019). Both tree-based models were trained using the default MSE loss for the expected passage time model. For the probability of dropping out, as with the LASSO, the AUC measure was used for training the XGBoost model. While for random forests we train probability forests as described in Malley et al. (2012). We found one hundred trees for both random forests and XGBoost models to yield the best results, where the boosted trees were limited to a maximum depth of three, based on CV results.
We concurrently selected the modeling framework for the prediction intervals based on which expected passage time model was selected according to the criteria described in Subsection 5.3. Based on the results in Section 6, we generated 95% prediction intervals with random forests using the quantile regression framework described by Meinshausen (2006), as implemented in the ranger package via the quantreg option. Table 5 summarizes the different models considered. Of course with more explicit feature engineering beyond what is listed in Subsection 5.1 the LASSO can also account for non-linear interactions, but for our purposes we compare its performance to the tree-based models as the simple, linear model reference.

Model Validation
To ensure that our selected models performs well in more than one race, we used leave-oneyear-out (LOYO) CV (e.g. train on all runner and race checkpoints in years 2015 through 2017 and test on 2018). To evaluate the models we used root mean-squared error (RMSE) and area under the curve (AUC) for the expected passage time and probability of dropping out models respectively. We evaluated the models with two criteria: (1) overall performance measures, and (2) the performance measures across all checkpoints so the models did not overfit to certain checkpoints.
In addition to evaluating the types of models in Section 5.2, we also evaluated the types of features included Subsection 5.1, separated in four groups beginning with checkpoint-level information and step-wise including more: Due to the inclusion of the lag information from two checkpoints prior, we only compared both model and feature performance starting at the third checkpoint in each year.

Results
This section walks through the results and analysis of the models in our TRAP framework. Figure 7a displays the overall LOYO CV RMSE for each type of expected passage time model considered from Section 5.2 along with the features considered in Section 5.1. All of the possible model-feature combinations vastly outperform the intercept-only model, but we additionally see that the tree-based approaches appear to display a better performance than the LASSO for each set of features. In particular, the drop in RMSE from including both checkpoint and runner-level information is greater for the tree-based models indicating the advantage from accounting for potential interactions in the data. We additionally see that both sets of lag variables indicate an improvement in model performance, with a smaller improvement from including features that were based on two checkpoints prior. Figure 7b displays the checkpoint level LOYO CV RMSE for each model type using the full set of features (Checkpoint + Runner + Lag 1 & 2) in addition to the reference baseline model. We clearly see that all three models vastly outperform the intercept-only approach and that unsurprisingly they each result in higher RMSE values for the checkpoints closer to the end of the race likely due to fewer runners remaining. Based on the results seen in Figures 7a and 7b we determine random forest and XGBoost models yield comparable results that are better than the baseline and LASSO overall and across checkpoints. However, due to its simpler extension, we use the quantile regression forest implementation of random forests in the ranger package for generating the 95% prediction intervals in Section 6.4.

Expected passage time model comparison
6.2 Probability of dropping out model comparison Figure 8a displays the LOYO CV receiver operating characteristic (ROC) curves for each model-feature combination considered for the probability of dropping out model. Table 6 displays the corresponding overall AUC for the LOYO CV results. In comparison to the expected passage time results in Subsection 6.1, including the runner level information with the checkpoints does not appear to lead to an improvement in model performance. However, as before, all combinations outperform the intercept-only baseline and inclusion of the lag features (back to two checkpoints prior) leads to the best holdout performance across the different models considered.   Figure 9: Top ten variables by importance for (a) random forest expected passage time model and (b) XGBoost probability of dropping out model. three models outperform the intercept-only and display a decay in performance for later checkpoints. We determine that the XGBoost probability of dropping out model performs best given its overall holdout AUC measure and performance across all checkpoints. We did not observed any notable differences between the holdout years for both the expected passage time and probability of dropping out models.

Analysis of Feature Importance
To provide more context with regards to the models we found to yield the best performance, Figures 9a and 9b display the top ten features by importance for the random forest expected passage time and XGBoost probability of dropping out models respectively. Unsurprisingly, the most important variable for predicting the expected passage time at the checkpoint is the interaction between gain in elevation from the last checkpoint and the ratio of time passed from the passage at the last checkpoint and from the second to last checkpoint. The second most important variable is the distance from the last checkpoint. 8 out of 10 of the features listed are interactions between lagged and checkpoint-level variables. Interestingly, the most important variable for predicting the probability of dropping out is an interaction between the increase in altitude and a relative 1-lagged response, indicating an increase in the likelihood of dropping out if the runner was slower in the previous checkpoint when compared to the runner who passed that checkpoint first and now has to face a high increase in altitude. This variable is followed naturally by the cumulative distance traveled, reflecting the increasing likelihood of dropping out as the runner continues in the race. In 95% prediction interval?

Prediction Interval Examples
No Yes Figure 10: Examples of performance for two randomly selected runners from each edition of UTMB in the data set, from 2015 to 2018, that completed the race. Each runner's actual passage time is displayed with black points along with 95% prediction intervals indicated with gray bands.
corresponds to the observed passage time by the runner, while the gray bands indicate the 95% prediction intervals. The color of the point indicates whether or not the the point was inside the prediction interval. While we notice examples of passage times outside of the prediction intervals, we see that they are not drastically off. Additionally, we can see differences between the pairs of runners for each year, with certain checkpoints varying in the size of the interval. For example, we see a clear difference in prediction interval and then performance for the two runners in 2018 following checkpoint 16. We explored the usage of features constructed based on these intervals (e.g. observed passage time within 95% prediction interval for previous checkpoint, cumulative in or out of interval, etc.), but did not find an improvement in model performance relative to the usage of the lagged information.

Discussion
In this work we have presented a Python package, ScrapITRA, and a predictive framework, TRAP. Our contributions serve two distinct but complementary purposes. On the one hand, ScrapITRA simplifies access to race-and runner-level information recorded on the ITRA website. We envision the simplified access to the novel ITRA data set offered by ScrapITRA, together with our seminal work, to encourage research on trail running. On the other hand, TRAP is informative in the phase of course design and assessment of runners' performance happening before and during the race respectively. Design entails decisions regarding the location of checkpoints and aid stations, evaluation of the segments and of race difficulty, and consequent placement of race volunteers and medical staff. Thanks to ScrapITRA, information about the runners' past race history can be incorporated in the design phase through the TRAP framework prior to the start of the race. At the same time, assessment of performance plays a fundamental role throughout the race. Likelihood of dropouts and prediction intervals inform race organizers of runners that are experiencing difficulties and may need medical help. Expected passage times and prediction intervals are useful to supporters and race volunteers to understand the time range in which runners will reach the following checkpoint. Together, ScrapITRA and TRAP represent a transferable and scalable framework: the model needs not be retrained for new races and the accuracy of its predictions increases with data.
Despite the successful results presented in Sections 3 and 5, our framework suffers from limitations related to the matching of runners within and across the data sets. First of all, matching of runners' names is a problematic issue. Indeed, in the ITRA data set the same name could refer to multiple runners. Since we used names and nationalities to aggregate race records, this could have caused misestimation of the race history for a tiny fraction of runners. This overcounting issue could be partially solved by matching runners also on the age category. Alternatively, scraping runner-level information from the runners' webpages, instead of constructing it from race-level data as we did, would completely solve this issue. We did not implement this strategy due to the large size of the ITRA data set. Matching of runners between UTMB and ITRA data sets is also complicated by mispellings. The fraction of participants that we could not match and consequently dropped from our analysis of UTMB is likely explained by names mispellings in one of the two data sets. Given the information currently available, we leave this problem for future work with record linkage methodology. Last, we recognize that ITRA data may not be only partially informative of the runners' past experience, particularly for runners from countries such as the US, where only a fraction of all races is recorded in ITRA.
This work opens several future directions of research. On the ITRA side, scraping runner-level data from the runners' webpages would provide information about age and ITRA performance scores. It would also help researchers circumnavigate the forementioned problem of duplicate names. Analysis on the ITRA data set could focus on runners' performance over time or differences across nationalities and age categories. On the race-level side, as it is the case for UTMB, an interesting line of research is represented by the analysis of performance leveraging data sources that provide finer grid information. For instance, Strava and Garmin Connect data contain a (almost) time continuous stream of information on runner's position and pace. Such data would allow researchers to investigate runners' pacing strategies in a manner similar to sports with player-tracking data, e.g. evaluating continuous-time movements in basketball (Sicilia et al., 2019) and American football (Yurko et al., 2019). An extension of the TRAP methodology could incorporate this filtration F(X i,t ) of information via recurrent neural networks, and provide improved, continuous-time insights into the segments difficulties. Another improvement of TRAP would be possible if data regarding the reason behind the dropout was available, i.e. why the runner withdrew from the race. The disentanglement of dropouts caused by serious medical issues from the rest is of central interest to race organizers and medical staff. While predictions of these events would likely suffer from a high degree of uncertainty, they might still be a useful target to consider.