{'fraction_non_alphanumeric': 0.02394089591321425, 'fraction_numerical': 0.012157486205929113, 'mean_word_length': 4.854014598540146, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 1, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Automatic Recognition of Public Transport Trips from Mobile Device Sensor Data and Transport Infrastructure Information

Automatic detection of public transport (PT) usage has important applications for intelligent transport systems. It is crucial for understanding the commuting habits of passengers at large and over longer periods of time. It also enables compilation of door-to-door trip chains, which in turn can assist public transport providers in improved optimisation of their transport networks. In addition, predictions of future trips based on past activities can be used to assist passengers with targeted information. This article documents a dataset compiled from a day of active commuting by a small group of people using different means of PT in the Helsinki region. Mobility data was collected by two means: (a) manually written details of each PT trip during the day, and (b) measurements using sensors of travellers' mobile devices. The manual log is used to cross-check and verify the results derived from automatic measurements. The mobile client application used for our data collection provides a fully automated measurement service and implements a set of algorithms for decreasing battery consumption. The live locations of some of the public transport vehicles in the region were made available by the local transport provider and sampled with a 30-second interval. The stopping times of local trains at stations during the day were retrieved from the railway operator. The static timetable information of all the PT vehicles operating in the area is made available by the transport provider, and linked to our dataset. The challenge is to correctly detect as many manually logged trips as possible by using the automatically collected data. This paper includes an analysis of challenges due to missing or partially sampled information in the data, and initial results from automatic recognition using a set of algorithms. Improvement of correct recognitions is left as an ongoing challenge.


Introduction
Automatic detection of the door-to-door trip chains of people has a multitude of applications. For infrastructure planners and public transport providers knowledge of the true origins, destinations and volumes of commuters gives a much better understanding of requirements for the road and transport networks than disconnected counts of cars or people at points of observation of the current network by using loops, cameras, ticket systems or manual passenger counting campaigns.
To assist users of public transport with relevant information about opportunities and problems it is vital to be able to generate a prediction of the next destination, time of travel and the means of public transport the person is going to use. For many people the same trips are repeated with regular cycles, i.e. daily, on (certain) weekdays, weekly or monthly. Such travellers can be proactively given targeted information about disruptions in road traffic or public transport lines, which they frequently use, at their personal times of regular usage. Real-time recognition of the current means of public transport and a prediction of the likely destinations can also be used to assist connecting passengers.
Multiple smartphone-assisted ways for automatic detection of public transport usage can be envisaged: • Ticketing system: If coupled with the payment of the trip in systems where both vehicle entries and exits are registered, precise and correct information about the public transport part of trips can be obtained. Requires integration with the fare payment system.
• Radio beacon: Some transport providers may provide vehicle-installed radio transmitters, e.g. WiFi 1 or Bluetooth 2 , which can be detected and compared with a list of beacon identifiers to detect the proximity of a public transport vehicle.
• Live positioning: The live positions of public transport vehicles can be matched with the measured positions of the passenger, searching for a sequence of continuous matches with suitable accuracy.
• Static timetable: Processed information about a trip carried out by a person (start and end locations and times, route geometry) using a vehicle can be compared with the information in a static timetable.
In our test arrangement we did not have access to the ticketing system of the public transport provider. Additionally, the current policy in the Helsinki region does not require registration of vehicle exits, and one validation of a regional ticket allows up to 80 minutes of transfer without a new validation. Therefore ticket-based information would not have been accurate even if it were available. At the time of testing listed Wi-Fi beacons were only available on trams, which does not give an adequate coverage of the transport network. Live positioning of public transport vehicles was made available by the public transport provider for a part of their fleet. Static timetable information was available for all lines and departures.
The target was to create a dataset for testing and benchmarking algorithms for automatic recognition of public transportation trips. The dataset is composed of position and activity recognition samples of 8 researchers between 9 am and 4 pm EET+DST on August 26th 2016, manual bookkeeping of their trips and the related transport infrastructure data. Data collection for the limited period was pre-agreed with every campaign participant to enable publication of the dataset without privacy concerns.
Seven participants executed as many public transportation trips as possible during the designated time, especially emphasising travel by subway, as it has been the most challenging transportation mode for automatic recognition. The eighth participant logged some private car trips to provide comparison data, which should not match with any public transportation.
Due to the exceptional amount of travel per person during one day this dataset cannot be used as a source for studying the regular travel habits of public transportation users. It also doesn't contain repeatedly visited locations such as homes or offices. The challenge is to correctly recognise as many trips listed in the manual log as possible by using the other forms of data available. The dataset consists of the following tables: • Device data: samples from mobile device sensors.
• Filtered device data: selection of the perceived activity and exclusion of data points at times of staying still using our algorithms.
• Device models: phone models used by the participants.
• Manual log: manual trip bookkeeping entries of participants.
• Live position samples: public transport fleet positions.
• Static timetables: public transport timetables valid on the date of the experiment.
• Train stop times: measured train stop time information for the date of the experiment.
The complete dataset is available 3 in github 4 .

Background
Transportation mode estimation and classification using mobile phone sensors has been discussed e.g. in [3,5,13]. Automatic recognition of bus trips using mobile phone sensors has earlier been addressed by the Live+Gov 5 project. Their mobile client collected both hardware sensor (accelerometer, rotation vector, gyroscope, magnetic field) and processed activity detection data [6]. A project proprietary human activity recognition classifier was built and trained. Public transport recognition was attempted using public transport infrastructure data similar to our study (static timetables and live locations of public transport vehicles). The client software also supported user marking of activity, but users were shown the currently detected activity, and consequently they have generally registered their activities only in cases, where the detected activity was incorrect. Service line detection probability of 37% is reported [8].
In one study bus passengers with mobile phones have been used as voluntary crowdsourced sensors for reporting the live locations of buses [1]. In the Live+Gov project traffic jams were detected from irregularities in public transport fleet location data [6]. The specific context of parking has also been considered, estimating from mobile phone sensor data, whether a parking area has space available [9,11].
The present effort belongs to the TrafficSense 6 project, a part of the Aalto Energy Efficiency Research Programme 7 . The project aims to save time and energy in traffic by understanding regular travel habits of individuals and proactively assisting them with information available in the internet. Sustainability and business model of the approach have been considered in [4]. The current prototype service uses the methods outlined in this paper to discover the public transport usage of travellers and automatically and proactively relay targeted information about disruptions in the greater Helsinki area based on the individually detected lines and times of usage.
The current TrafficSense mobile client is coded in native Android 8 . It is available on Google Play 9 , but currently restricted to Finland, because the public transport recognition and disruption information functions are only available locally. Example screenshots are shown in Figure 1. The main map view a) shows ranked regular destinations (purple 4 and 5), car, bus and train trips as well as the current location of the user. A train trip is selected with departure and arrival times indicated. The "Edit/Confirm" button can be used by the user to confirm or edit the activity and the line name (currently "U"). The screenshot of the notification tray b) shows a public transport disruption bulletin.
The types, formats and sources of open traffic information available were reviewed in [10]. For information on public transport the resources from Helsinki Regional Transport 10 (HRT) were used. Their developer resources are currently provided and documented under the digitransit 11 umbrella.
The resources follow the SIRI (Service Interface for Real-time Information [10]) specifications 12 . Live locations of the fleet were sampled from their live data API 13 . The static timetable data is in GTFS (General Transit Feed Specification 14 ) format as specified by Google Inc.
Train stop time information is encoded as JSON (Javascript Object Notation 15 [2]). It has been fetched from the digitraffic 16 service 17 operated by the Finnish Transport Agency 18 (FTA). All other sampled data in the repository is made available in CSV (Comma-Separated Values [12]) format. The repository includes scripts for importing the CSV tables into a PostgreSQL 19 database, which is used internally by the TrafficSense project.

Manual bookkeeping by test participants
All test participants manually documented the details of their trips during the day. The information provided for each trip leg is shown in Table 1. Because the timestamps were manually recorded, their values are approximate. In total 103 trips were recorded.

Mobile client filtering algorithms
The client alternates between ACTIVE and SLEEP states as shown in Figure 2. The current default value for the sleep timer is 40 seconds. If the detected position changes by a distance longer than the accuracy of the position fix during a period of the perceived activity indicating STILL, the timer is restarted. This rule aims to prevent erroneous transitions to SLEEP state while the device is moving, but activity detection perceives it as being still. Such situations typically occur during smooth trips on rails, i.e. trains, trams and subways.
In ACTIVE state position is requested from the fused location provider 23 provided by Google Play Services with high accuracy and a 10 second interval. In SLEEP state position requests are dropped to no power priority, which means that position fixes are passed to our client only if requested by another application. Activity reports are always requested with a 10 second interval, but as a form of device power saving, during STILL detections activity recognition interval has been observed to increase up to 180 seconds. As a result, sometimes the client may need up to ≈ 200s of movement to wake up from SLEEP state.
In our data format each accepted position record is coupled with the latest activity information. The timestamp of the entry is the timestamp of the position, not the activity detection. Therefore the same detected activity may repeat over multiple points. The received position fixes ('points') are filtered as follows ( Figure 3): • If 60 minutes have passed since the last accepted point, any point is accepted (to provide a "ping" effect and record that the client was running).
• Accuracy 24 must be better than 1000m.  • If activity != last queued activity and activity is 'good' 25 , the point is accepted.
• If (activity == last queued activity) and (distance to last accepted point > accuracy), the point is accepted.
The dataset contains about 30% of the theoretical maximum 26 number of points.
The fused location provider used by the mobile client combines data from satellite, Wi-Fi and cellular positioning. Despite that, sometimes positioning errors occur. In a typical case the correct position alternates with a distant point, probably due to changing between different positioning methods within the fused location provider. An example of such a problem in shown in

Filtered device data
The dataset also includes a table of filtered device data. The filtering operation serves two main purposes: • Remove periods during which the terminal was not substantially moving.
• Find the most likely activity for each remaining data point.
Filtered device data is included in the published data set, because some of our recognition algorithms use it. A new candidate solution is welcome to base itself on the more complete device data instead, and implement other filtering approaches. The activity filtering algorithm has a clear impact on recognition results, as can be seen from the data shown in Section 5.1. The following parameters are included in device-data-filtered (corresponding descriptions are the same as in Table 2

List of device models
A list of the models of the eight smartphones used by the test participants is provided as a separate table. It is included, because some differences were observed in e.g. activity recognition performance between different devices. The table contains the following columns: 1. device id (integer, same as in Table 2) 2. model (string name of the model)

Public transport infrastructure information
Details of the information sourced from public transport infrastructure providers is described in this section.

Live positions of public transport vehicles
The live positions of the public transport fleet were obtained from HRT and sampled at 30 second intervals. The columns recorded into the dataset are shown in Table 3. The time period was restricted to the time of the trial. The maximum and minimum coordinates recorded by the participants were checked and the live vehicle position data was filtered (as a rectangle) to include only the area surrounding the locations sampled from the test participants. The resulting area is shown in Figure 5. The table length is 229,451 entries.

Static timetables
The static timetables from HRT are not included in the data repository, but the file is available through the following link: http://dev.hsl.fi/gtfs/hsl_20160825T125101Z.zip. It can be used e.g. with an OpenTripPlanner 27 (OTP) server to query for trips with matching start and end locations and start times. A more precise description for the specific task of querying timetable information can be found in the documentation for the digitransit 28 API. The data includes the static timetables for all the public transport vehicles (also local trains) used by the study participants.

Train stop times
The dataset also includes information about the recorded stop times of local trains at stations on the day of the study. The information 29 includes a JSON-format junat object, including the stopping times of trains at each station as described ( • Buses: 16, 67, 72, 550, 560 • Subways (line name in manual-log varies and cannot be used for comparison) Travelled line name specifiers not found in transit-live are: • Trains: E, I, K, P, U In terms of recorded trips, the 10 bus trips shown in Table 4 can therefore definitely not be found in transit live. The 28 trips in Table 5 can be confirmed as being discoverable and identifiable using transit live data, as they have been found with our algorithm. The 38 trips in Table 6 have an overlapping IN VEHICLE (sometimes also an ON BICYCLE) segment in device data filtered. Therefore these trips should be recognisable if the particular vehicle exists in live, but as shown in the recd type and recd line columns, they have not been properly recognised by the current algorithms. Logged trips matching multiple recorded segments are listed multiple times in the table. For performance gains in recognising trips using live data, these are expected to be the best candidates to look at.
Finally, the 13 trips in Table 7 have no overlapping IN VEHICLE segment from the mobile device samples. It is also noted that: • 3/13 trips (2 subway + 1 tram) have no overlapping mobile device samples at all • 10/13 trips overlap with WALKING (in 6/10 cases shadowing the whole trip) 30 Bus line name specifiers <100 can be re-used by the adjoining cities of Helsinki, Espoo and Vantaa. Bus 18 from Helsinki is included in transit live, but the test participant used bus 18 of Espoo.  • 12/13 trips (device id 5 and 6) were carried out with the same device model The device data sampled before and after a time segment with no data should still reveal that the mobile device has moved without recording positions, i.e. it has either erroneously been in SLEEP state or no position fixes have been received. Looking at the timestamps and locations recorded before and after the break period with geographical displacement, a set of candidate options could still be found from either the static timetable or live fleet position data. In downtown areas with many transport alternatives errors would be likely, but in less busy areas the set of alternatives will be smaller. Faulty recognitions changing car trips to public transport would still be very likely, so this type of recognition should only be attempted in scenarios where the penalty of false positive is not very high. This type of recognition has not been attempted in the current experiment.

Methods of automatic recognition
The collected live public transport vehicle locations and static timetables, in conjunction with the user location traces, were used to automatically recognise public transport trips taken by users. Trip legs consisting, after filtering, of a continuous sequence of IN VEHICLE activity were matched against the vehicle position traces and timetable data. The train stop times were not used.

Live vehicle location
The vehicle location data from Table 3 is compared with the user locations collected by the personal mobile devices as described in Section 4. Potential issues in matching include: • Missing vehicle or user data points For performance reasons the number of user location samples used for matching a trip leg was limited to 40. To counteract the location accuracy and vehicle length issues, a distance limit of 100 metres was used for collecting vehicle matches. A greater limit may cause more false positive matches to appear.
With the sample rate of thirty seconds in the collected vehicle positions, a vehicle traveling at 80 km/h would produce samples every 667 metres, much in excess of the above mentioned distance limit. This could cause false negatives with the user location samples falling in between the vehicle points in such a way that they are not matched. To prevent this, the position sequence of each vehicle is processed into linestrings, and user point distances calculated against those line geometries.
The vehicle location points for composing the linestrings for comparison are collected in a ±60second window around the timestamp of each user point sample. This allows for some clock difference, and sampling time difference.
For a vehicle to be accepted as a possible match, its linestring must be within the 100 metre distance limit for a minimum of 75% of the user location samples.
Each match within the permitted 100 metres' distance accumulates a score of 100 − d when the distance is d metres. The vehicle with the highest score wins.
The line type and name are set according to the matched vehicle. In case of the vehicle having multiple line names in the matches, the most frequently occurring name is used. On some lines, a vehicle can intentionally change line name when passing a certain stop. Also, some vehicles in the data set change line name intermittently to false values.
In addition to the method described above (subsequently referred to as "New live"), a prior implementation ("Old live") was evaluated. In the older implementation, four user trace sample points were used, and matched against vehicle location points in the surrounding time window. With the vehicle location sampling interval of 30 seconds, and the 100 metre distance criterion, this would be expected to cause otherwise optimal user trace point samples to potentially fall outside the matching radius once the vehicle speed exceeds 2 × 100m/30s ≈ 6.67m/s ≈ 24km/h. For trams in Helsinki having an average speed of 14.7 km/h (2013-2014) 31 , with dense stops the new and old methods should produce similar results. More differences can be expected on the subway, and on bus routes with highway segments, where speeds are higher and stops more sparse.

Matching with static timetables
Comparison with static timetables is based on searching for public transport plans of past trips based on the sampled IN VEHICLE segments. Firstly, a trip-plan query is sent to the OTP journey planner interface based on each candidate leg's start-time, first point (origin) and last point (destination). Secondly, the resulting PT plans are compared to the leg's start-time and end-time as well as the user location trace to identify whether the leg represents a PT ride. Potential issues in matching include: • Missing or inaccurate user location points • Inaccuracy in activity determination and filtered transition points • Clock differences 31 https://www.hsl.fi/sites/default/files/uploads/13_2015_raitioliikenteen_linjastosuunnitelma_netti. pdf • False positives when the route and timing of a car trip are similar to a segment of a public transport line Table 8 explains the constants and Table 9 the variables and formulas used in the process. Figure 6 illustrates an example public transport trip and highlights the following parameters showing possible extra parts of the planned trip compared to the original sampled trip: tW b, tW e, tP T b, tP T e, where t indicates "time", W indicates "walking" and P T indicates "public transport". The b denotes "at the beginning of a trip or leg", and e denotes "at the end of a trip or leg".  The data sampled from a mobile device can often have a fair amount of inaccuracy in location and activity detection. For this reason the filtered transition points starting and ending the vehicular trip leg can often appear between stops during the actual mass transit leg. Therefore almost all travel plans from the journey planner inevitably include walking sections in addition to the desired transit (vehicular) leg. The additional walking sections can appear at the beginning (tW b to walk from the detected start of vehicular leg to the public transport stop for boarding) and/or at the end of the trip leg (tW e to walk from the exit stop to the detected end of vehicular leg).

Selection of query parameters
The origin and destination parameters of our OTP query are set equal to the leg's start and end geolocations respectively. The third query parameter, trip start-time, is set based on the leg's starttime as follows.
The maximum allowable inaccuracy in transition points is dEmax = 500metres. Therefore, to account for inaccuracy in time and place of origin, in our query to the journey planner the earliest permissible start-time of the trip is adjusted back by tW b = 6.2minutes to allow 500 metres of walking at a speed of vW = 1.34m/s to the boarding stop. Figuratively speaking, in case of offset in recorded IN VEHICLE leg, we allow the traveller to catch the desired transit line according to actual timetable by walking to the nearest stop. If there is no delay and time and place of transition are completely accurate, the traveller would just wait 6.2 minutes at the boarding stop to take the transit line. Adjacent stops are assumed to be no more than one kilometre (2 × 500m) apart.
Correspondingly, to relax both the origin and the destination, a maximum of 2 × dEmax = 1, 000 metres of walking is requested for each resulting plan. The OTP query requests for three PT plans for each candidate vehicular leg.

Filtering and validation of query response
Out of the three returned plans the best match, if good enough, is chosen. Plans that match the following criteria can be discarded : • Plans having a total duration of more than tEP T = 3minutes shorter than the user-recorded vehicle leg, thereby assuming that the transit vehicle must travel closely according to schedule to avoid false positive matches.
• Plans having a total duration of more than ∆dt = 18minutes longer than the user-recorded vehicle leg. This difference denoted by ∆dt includes the walk times for dealing with location inaccuracy discussed above (∆tW = 6.2min × 2 = 12.4min for the whole trip), and the time assumed for a mass transit vehicle to travel between adjacent stops (maximum 1000m for the whole trip at minimum speed of vP T = 3m/s, equal to ∆tP T = 5.6minutes).
• Plans where the duration of the included transit leg (tP T ) differs by more than ∆tP T = 5.6minutes from the user-recorded vehicular leg (tV ), based on assumed time of travel of the vehicle between adjacent sparse stops.
• Plans where the start of the included transit leg differs from that of the recorded vehicular leg by more than 5.8 minutes. This value considers a public transport vehicle traveling between adjacent stops at the beginning of the trip (tP T b = 2.8min, for 500m) plus an assumed tEP T = 3min deviation of the transit line from its schedule.
The plans returned by the journey planner interface include location point sequences of the planned trips. As illustrated in Figure 7, the location point sequences are also matched against the recorded user trace to verify that the route of the public transport line matches the recorded user trace. For comparison the recorded user trace is sampled at no less than 100 metre intervals, and leading and trailing points may be ignored based on the assumed inaccuracy of origin and destination. A minimum of 70% of sample points must match a plan point within 100 metres, and no more than four adjacent sample points may fall outside 100 metres of the plan points for the plan to qualify. Out of the qualifying plans, the one with the closest start-time to that of the recorded leg wins.
Our current algorithm only validates plans containing a single vehicular leg. However, quick transfers between vehicles may not have been detected as separate vehicular legs by the activity detection and filtering of the mobile device. Allowing trip plans with transfers, and splitting the reference vehicular leg accordingly, could produce improved detection for such cases. An example of the fusion of a BUS leg and a SUBWAY leg to a single IN VEHICLE leg can be found from device 6 at 13:59 to 14:25, where a short transfer has occurred between 14:17 and 14:19.

Recognition results
Compared with the 103 manually logged trips, 86 IN VEHICLE segments were recognised by our filtering algorithm, 85 of them overlapping at least a part of a logged trip.
The 28 trips recognised with correct line name from transit live using the new live algorithm were detailed in Table 5. The statistics for all the recognition methods are collected to Table 10. The static approach yielded the highest number of matched trips, but with one false line type and two false bus line names. Live recognition performance suffered from the absence of many buses and all trains in the data, but especially subway trip detection was significantly improved with the new live algorithm compared with the old one. Looking at the combined results but without requiring the line name to match, 60% of bus and tram trips, 44% of train trips and 43% of subway trips were recognised. If the correct line name is required, bus recognition success drops to 47%. Table 10: Trip matching statistics for all recognition methods compared to the manually logged trips ("line name" = matching line name, "line type" = matching line type).
in trains and underground scenarios, measurements can be intermittent or completely missing. Other problems in positioning sometimes result in locations hopping between the correct position and a single distant point. Live locations were not available from all public transport vehicles at the time of the trial.
Altogether 103 trips are logged in the dataset, 97 of them carried out using public transportation. Combining matches from the static timetable and live data 60% of bus and tram trips, and 43-44% of train and subway trips were recognised with the correct vehicle type. Recognition of correct line names was otherwise on the same level, but for buses the recognition result dropped to 47%. The joint combined public transport recognition reached a level of 48% for the correct line type.
The currently achieved results are approximate, but adequate for purposes, where exact recognition of every trip is not necessary. For e.g. public transport disruption information filtering the current recognition level would be sufficient, because the likelihood of a correct recognition increases fast for frequent trips on the same public transport line and the penalty for false positive recognitions is not very high. For purposes where the passenger can review past trips individually the current level of performance can be frustrating. For fare payment processing it would be unacceptable. Improvements in activity recognition accuracy of the mobile device and better vehicle coverage of live transit data would both contribute to better recognition probability. With the current power saving algorithms the mobile device power consumption is reasonable but clearly noticeable, especially when the device is constantly in motion, causing positioning to be requested with high accuracy. Without radical improvements in battery and / or positioning technologies power consumption should be decreased rather than increased, which sets limitations to future improvements in the client sampling. One approach for testing would be to make more use of radio beacons in public transport vehicles. While in proximity of an identified radio beacon, the high accuracy positioning of the mobile device could be switched off and the system would rely on the positioning of the public transport vehicle.

Acknowledgements
Supported by the TrafficSense project in the Aalto Energy Efficiency Programme funded by Aalto University.

Licensing
The data in the referenced github repository is licensed under the Creative Commons BY 4.0 licence 32 . The data extracted from the test group 33 is licensed by Aalto University. Train data 34 is obtained from Digitraffic offered by the Finnish Transport Agency. The live data on public transportation 35 is compiled from a service offered by Helsinki Regional Transport. Map data from the city of Helsinki 36 is authored by "Helsinki, kiinteistöviraston kaupunkimittausosasto". 32 http://creativecommons.org/licenses/by/4.0/ 33 device data, device data filtered, device models, manual log 34 commuterTrains.json, trainStations.json 35 transit live 36 in the subway station entrance images offered in the repository