{'fraction_non_alphanumeric': 0.03785564670972247, 'fraction_numerical': 0.011040321172979577, 'mean_word_length': 4.4214572983203215, 'pattern_counts': {'":': 0, '<': 6, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Introducing the Partitioned Equivalence Test: Artificial Intelligence in Automatic Passenger Counting Validation

Automatic passenger counting (APC) in public transport has been introduced in the 1970s and has been rapidly emerging in recent years. APC systems, like all other measurement devices, are susceptible to error, which is treated as random noise and is required to not exceed certain bounds. The demand for very low errors is especially fueld by applications like revenue sharing, which is in the billions, annually. As a result, both the requirements as well as the costs heavily increased. In this work, we address the latter problem and present a solution to increase the efficiency of initial or recurrent (e.g. yearly or more frequent) APC validation. Our new approach, the partitioned equivalence test, is an extension to this widely used statistic hypothesis test and guarantees the same bounded, low user risk while reducing effort. This can be used to either cut costs or to extend validation without cost increase. It involves a pre-classification step, which itsself can be arbitrary, so we evaluated several use cases: entirely manual and algorithmic, artificial intelligence assisted workflows. For former, by restructuring the evaluation of manual counts, our new statistical test can be used as a drop-in replacement for existing test procedures. The largest savings, however, result from latter algorithmic use cases: Due to the user risk being as bounded as in the original equivalence test, no additional requirements are introduced. Algorithms are allowed to be failable and thus, our test does not require the availability of general artificial intelligence. All in all, automatic passenger counting as well as the equivalence test itself can both benefit from our new extension.


Introduction
Assessment of passenger counts is of paramount importance for public transport agencies in order to plan, manage and evaluate their transit service. Over the past three decades, automatic passenger counting (APC) systems have played an increasingly important role in determining the number of passengers in local public transport. They are used in the daily monitoring of operations, in long-term demand planning, as well as in revenue sharing within transport associations around the world. For more details and an overview of APC development and current practice, see Siebert and Ellenberger (2019). Revenue magnitudes in the billions are common in public transport (Armstrong and Meissner, 2010), e.g. in the year 2018, total ticket revenues in Germany alone have been 12.95 billion euros (Wagner, 2019), while APC systems are deployed worldwide. In many cases, a passenger count is the de facto standard for public tenders or the acquisition of subsidies. The counting quality of existing APC systems on the market has been continuously improved by technical developments in combination with increased requirements. Nowadays, APC systems are expected to have a maximal systematic error or bias of 1%. This aspect of APC validation is referred to as an accuracy of 99%. For transport associations in Germany, but also internationally, validation is typically regulated by the VDV, recommendation 457 (Köhler et al, 2018): In 2018, a criterion based on the t-test was replaced by an equivalence test, which takes the user error into account and limits it to 5%. This tightening of the requirements compared to the previous test criterion has led to up to four times larger sample sizes in the testing of the measurement accuracy, which also quadruples the costs that arise primarily from the manual inspection of the counting situations by comparison counting personnel. The already high cost pressure on the comparison counting continues to increase and the need for solutions to reduce costs increases alongside. Since the hourly wages of the reference counters remain the same or increase in perspective, technical and regulatory solutions are necessary. In order to increase validation efficiency and reduce costs, we have identified the following levels during our recent years of research (compare Table 1 for real world numbers): Efficiency Level 0: Manual ride checkers that stay in the vehicle during its entire journey and count boarding and alighting passengers (and other count objects) at doors.
Efficiency Level 1: Perform all manual counting on recorded (and automatically cut) (3D-)videos, which modern APC systems can acquire directly from the sensors, compare Table 1. Having a pool of (possibly unseen) videos available is a requirement for Efficiency Level 2 and 3.
Efficiency Level 2: Increase efficiency in the evaluation of video material, e.g. by using application-specific software that integrates comparison counting and video viewing, compare Figure 1.
Efficiency Level 3: Reduction of the video volume to be evaluated manually with verifiably equal validation quality (manufacturer and user risk) as the current method based on the equivalence test.
For latter, Efficiency Level 3, the reduction of the manually evaluated video volume, an adapted mathematicalstatistical formalism is required, which is the subject of this manuscript: We introduce and discuss the concept of this so-called Partitioned Equivalence Test in the second and present a mathematical formalization third section. We show how to perform a sample size calculation for our new method, analyse and optimize costs in the fourth section, evaluate real world data in the fifth section and close with some concluding remarks and future prospects in the last section. In the following we assume that a single (3D-)video corresponds to a door opening phase (DOP), i.e. it shows a view from inside the vehicle that allows to see an entire indoor door area (compare Figure 1) and Figure 2) from door opening to door closing: 1. Door opening: initially, the door is (almost) completetly closed, so that no passenger (or countable object) might pass.
2. All the events that happen while the door is open, e.g. boarding and alighting passengers.
3. Door closing: at the end of the video, the door is (almost) completely closed again so that no passenger (or countable object) might pass.  Table 1: Savings w.r.t. different video material granularity for an urban railway transport system. The original video data (38:43h) has been recorded with a GoPro connected to an external battery and represents the original ride checker approach. From the actual timetable, real arrival and departure times were known, so the original video could automatically be cut into smaller pieces accordingly. From an APC system installed in the vehicle, the door opening phases were known as well and utilized in a similar way. Stop door events is a granularity in between vehicle stops and door opening phases: all videos with no door opening have been removed from the vehicle stop set, which is around 50%, since in most cases only one side of the vehicle doors open at stops. Specialized video software (e.g. VisualCount, compare Figure 1) savings are 70% according to our studies for the reason that users fast-forward through idle sequences. Manual and algorithmic partitioned equivalence tests have been accounted for with 20% resp. 50% savings. All in all, Efficiency Level 3 reduces the manual effort by around 99% over Efficiency Level 0.
A commonly used unit for (automatic) passenger counting so far has been the Stop Door Event (SDE), which corresponds to a video of an entire door during an entire stop: e.g. if a door (e.g. door 2) opens and closes again 3 times during a stop, 3 DOP and one SDE is generated. However, nowadays APC systems allow to record DOP, e.g. by accessing the corresponding door opening signal, which reduces the amount of video data to be manually evaluated considerably and upfront, compare Table 1. From an analytical point of view, the standard deviation can be estimated more reliably, since door opening phases are the smaller statistical unit than stop door events. In the following we therefore use door opening phases. The sample size calculation may still be carried out on stop door events e.g. for comparison purposes, see Section 4.

Partitioned Equivalence Test Concept
As mentioned in Section 1, in order to achieve a reduction in certification costs beyond Efficiency Level 2 (the use of more integrated hard-and software solutions) a further reduction in video volume to be evaluated manually is necessary.
But how can this reduction be achieved methodically? Here, we initially had the idea of using an additional algorithm to pre-classify the videos according to the certainty or safeness of a correct count by the APC system, so that a manual count is only necessary on material with a considered highly uncertain or unsafe. The challenge Figure 1: Screenshot of VisualCount: a specialized video software to perform manual counts, which has been used to obtain our count data, compare Section 5. The height information from 3D videos can be used to distinguish between children and adults, e.g. by using a critical height of 1.20 meters. To theses standards, the small person in the center of the image above would be considered as an adult, while the one to the left as a child. VisualCount yields a 70% speedup over the use of a standard video player with a spreadsheet software (like Microsoft Excel), which reduces the additional costs for video evaluation from 300% (the quadrupling of costs by accounting for the user risk in the VDV 457 v2.1 vs. the old version v2.0) to approximately 33%. It runs entirely in the browser, thus can operate without a server by using videos from the users filesystem, which increases data privacy. A cloudbased operation would be possible as well. Performance-critial parts are implemented in C/C++ and compiled to JavaScript or WebAssembly with Emscripten (Zakai, 2011), with filesystem-call passthrough. This allows to determine durations, frame counts as well as single-frame random access to hundreds of gigabytes of video data not only using browser-native (like H.264 or AV1), but also general standard (e.g. MJPEG via FFmpeg) or specialized, custom file formats (e.g. multiple, losslessly compressed 3D video formats) without noticeable delay, eliminating almost all wait times. To ensure a seamless integration, we reverse engineered a proprietary video file format from one of our suppliers, who provided the full documentation after a technical demonstration. A gamecontroller is used to navigate through the video with playback or rewind speeds corresponding to the pressure applied to the analog buttons to obtain both slow motion for crowds as well as high speeds (10x and more) to skip through long idle timespans. Rates of 60 frames per second can be reached within the browser so that even in fastest fast forward mode, no boarding or alighting passenger is accidently skipped. A specialized video software (such as VisualCount) is required to reach Efficiency Level 2 and above. would have been to formulate rules for the general approval of such a classification system and to develop an algorithm that is powerful enough to meet the formulated requirements. However, as it turned out in the course of the project, this approach has unmanageable implications: it either requires the construction of a kind of superior, infallible algorithm, which can, with unlimited certainty, identify the incorrectly counted videos. Alternatively, if that algorithm had been fallible, the ground truth would have had to be redefined: currently, a count value is considered correct if it was generated by (at least) two manual counters and verified a third manual counter -the supervisor -in case the first two counters differ (Köhler et al, 2018). An attempt to redefine this current ground truth would have raised both ethical as well as technical questions (Lake et al, 2017) and would definitely have lead to unfruitful, never ending discussions in the foreseeable future. Facing these challenges, we finally changed our approach to include a sample in the material that the algorithm has classified as not necessary to be manually viewed or safe. Here the question arises whether savings can be achieved in this way at all, but such a procedure is at least technically feasible by today's standards and also methodically sound. We have therefore continued our investigations.
For the implementation of Efficiency Level 3 (reduction of the video volume to be evaluated manually with verifiably equal validation quality, compare Section 1), a mathematical-statistical formalism is required which satisfies the following requirements: 1. Automatic passenger counting and the VDV 457 influence the worldwide distribution of revenue in public transport, which means that all changes, especially in the statistical inference, are critical. This results in the following fundamental requirements: (a) The cost savings should be relevant enough to make the adaptation of an existing validation procedure legitimate.
(b) For a well-founded decision, an analytical derivation of the new statistical method must be possible.
(c) The cost savings should be achievable without changing the definition of the ground truth.
2. Compared to the equivalence test, the new test should not place any additional requirements on the statistical distribution of counting errors.
3. If the parameters of the new test are selected so that the entire sample is counted, the new test should correspond to the previous test, i.e. the equivalence test.
4. As in the equivalence test, it should always be ensured that the user risk is not greater than a specified limit (currently max. 5%) and a specialized sample size estimation should lead to a controllable adaptation of the manufacturer risk.
5. It would be beneficial if no special software is required to determine the result of the test, i.e. an evaluation itsself should be possible with commonly used spreadsheet software.
We have created a test procedure that statisfied all the above mentioned requirements, the Partitioned Equivalence Test. The idea is as follows: First, divide (or partition) the video material to be evaluated into two parts: one part comprises the unsafe door opening phases, i.e. all videos where a miscount of the APC is suspected. This entire so-called unsafe partition is counted manually according to the current procedure, i.e. by (at least) two persons and a supervisor count in case the first two differ. In the other part, the so-called safe partition, there is only a relatively small manual sample count. The partitioned equivalence test is now able to merge the comparison counts of the two partitions again and to create a common confidence interval, on which a regular equivalence test can be carried out. The partitioned equivalence test is performed in four steps, compare

Statistical Model
As a starting point we take the model as in (Siebert and Ellenberger, 2019): Let Ω 0 = {ω i }, i = 1, . . . , ∞ be the statistical population of door opening phases (DOP), which are used to summarize all boarding and alighting passengers at a single vehicle (bus, tram, train) door during a door opening of that door. Further, let Ω = {ω ij }, i j ∈ {1, . . . , ∞}, j ∈ {1, . . . , n} be a sample, which consists of n either randomly or structurally selected door opening phases (e.g. by a given sampling plan). We use the notation n = n e when the sample size was planned according to the equivalence test. Let M i , i ∈ {i, . . . ,n} be the manual count, K i , i ∈ {i, . . . , n} be the automatic count of boarding passengers made by the APC system and . . , n} the differences among them. For the partitioned version of the equivalence test yet to be introduced, it does not suffice to obtain the manual count by ride checkers, (3D-)videos of door opening phases are mandatory: We require a supply of (possibly unseen) footage, which can subsequently be used for counting through multiple manual sightings of single videos, favourably 3D depth data or 2D videos with additional, lower resolution 3D information. 3D depth data allows to estimate the persons heights, which is used to distinguish between adults and children and which is especially relevant for revenue sharing, since tickets for children are typically sold at a reduced price. To create the ground truth counts, only manual counts may be used: at least two with an additional supervisor count in case of differences. Alighting passengers (or other objects) are counted as well and results apply analogously, but without loss of generality we only consider the boarding passengers. Let M = 1 n n i=1 M i be the average manual boarding passenger count. We consider the random variables which we call relative differences being the differences of the automatically and manually counted boarding passengers relative to the average of the manually counted boarding passengers. The average D : is the actual systematic error of an APC system, since it can systematically discriminate participants of the revenue sharing system. It could also be referred to as bias of the measurement device (see e.g. Nielsen et al, 2014) or as statistical distortion (see e.g. Köhler et al, 2018) To test for equivalence, one wants to show that observed differences are within certain bounds, as opposed to complete equality. We here use the hypotheses and error types as commonly defined for equivalence testing, also sometimes referred to as the two one-sided tests (TOSTs) procedure (Schuirmann, 1987). As an alternative approach, the equivalence test can be derived directly from the two-tailed t-test under certain simple assumptions, i.e., that the parameters are induced (or simply exchanged) from those of the t-test (Siebert and Ellenberger, 2019).
Thus, the hypotheses are (Julious, 2004) H 0 : There is a (relevant) systematic APC measurement error(|µ Ω | ≥ ∆) (2) H 1 : There is no (relevant) systematic APC measurement error (|µ Ω | < ∆) . ( We define ∆ to be the equivalence margin and the relevant errors for the equivalence test with α referring to (half) the risk of the user and β to the risk of the device manufacturer. We will consider two-sided 1 − 2α confidence intervals where α is commonly chosen to be 2.5%. The test criterion to be evaluated is Sample size estimation for an equivalence test defined this way is given by Julious (2004) as: To develop the partitioned equivalence test as outlined, one uses an already existing classification into safe and unsafe door opening phases (DOP) on the total sample n rec of recored videos (n e ≤ n rec ), which results in two partitions. This classifcation is given by W i , i = 1,..,n into safe (W i = s) and unsafe (W i = u) door opening phases (DOP). To improve readability, we use the character placeholders s and u but any indicator will suffice. Further, let p s , p u = 1-p s be the likelihood p s of a DOP to be classified as safe and the (counter) probability p u of a DOP to be classified as unsafe. Then W i is accordingly Bernoulli distributed: W i ∼ P W = Bin(1, p s ). Let N s , N u = N−N s be the total number of safe and unsafe DOP and p s = N s /n be the frequency of safe DOP. Analogously, let µ s , ν s , µ u , ν u be the corresponding parameters for µ and ν on the respective partitions of the safe DOP and the unsafe DOP, explicitly µ s = 1 Ns Ns k=1 µ sk , µ u = 1 Nu Nu k=1 µ uk and ν 2 s = 1 Ns Ns k=1 ν 2 sk and ν 2 u = 1 Nu Nu k=1 ν 2 uk . Let q be the so-called counted quota of the safe partition and 1 − q the fraction not to be counted, i.e. skipped.
Different procedures, the so-called use cases, to obtain such partitions are described when discussing application cases in Section 5. In this context, useful additional information, such as video data and stop characteristics, can be used in a meaningful way to achieve a classification W i , i = 1,..,n. This implies that, in the following, conditional distributions (and parameters) on safe DOP and on unsafe DOP are to be considered, e.g. for M i |W i . A critical part of the use cases is the cost control, which is introduced in Section 4.3: taking into account basic costs c s0 , as well as manual counting costs c sZ for safe DOP and the combined costs for unsafe DOP c u .
For a schematic representation, compare Figure 4 of the current as well as the newly introduced parameters. The alighting passengers are not explicitly mentioned, as they are handled analogously to the boarding passengers. Our general statistical model is thus based on two strata that are defined by the two partitions. Since the classification into a safe and an unsafe partition can be entirely arbitrary, our model used stems from a general mixture model. A two dimensional mixture model can described as a hierarchical model consisting of the random classification as mixture component, see e.g. McLachlan and Basford (1988) or for Gaussian mixture models (Reynolds, 1993). Methods described in the statistical literature usually consider estimating the parameters of the mixture components. In our case, the classification is known and estimation of this component is thus not required. The estimation here is only the weighted recombination of both partitions. The case of combining data from different partitions (i.e. sources) resembles a fixed-effects meta analysis, but is different in two aspects. First, weights are not determined solely by sample size or standard error but rather by an artificial weighting scheme that is induced to avoid any overrepresentation that might be introduced by any (free choice) q. Secondly, the weights are not fixed but are dependent on the randomness of the classification, i.e. to be considered a random variable. In the following two sections we will introduce methods that address both aspects.

Estimation of the expected manual count
As a first step for all further calculations with the partitioned equivalence test, it is necessary to (indirectly) estimate the mean number of boarding passengers in the total sample M because in the safe partition the ground truth is only determined, i.e. manually counted, for a proportion q. Thus, in addition to the information of the measurement error X i , the non-comparison counted data also lacks the information of the actual M i . The average boarding number is needed for the definition of D i and has to be estimated, since the mean value M is not available. We here use the leave-q-out estimator (see Appendix A for details). With we obtain the estimator for the average number of boarding passengers. This value can now be used instead of M when calculating D i : and in the case of a full manual count q = 100%, the definition of D i is identical to the definition of the regular equivalence test because M q=100% = M .

Estimation of APC bias
Analogously to the estimation of the expected manual counts and with the the expected values of safe µ s and unsafe µ u DOP, a similar result regarding the expected value holds (see Appendix A for details with the random variables X uk being the relative differences D uk ): Methods are now needed to obtain a range-preserving confidence interval if only a portion specified in advance (for example 20%) of the safe DOP is to be counted. A random selection of q ·N s safe DOP is formed which are to be included in the final evaluation -i.e. for which the actual ground truth must be determined. The mean value serves as the estimator for µ u .
and for µ s a similar estimator can be obtained, which, however, only uses the quota q. The Z j indicate herby, if the DOP was randomly selected (=1) or not (=0): From the different estimators one can now generate some kind of composite estimator for µ: All values D uj where Z j is zero are therefore no longer needed for calculations. Then and thus D q is an unbiased estimator which can be calculated on a reduced dataset. Using the results of Appendix A, holds for the variance of the estimator. It should be noted that the variability of the uncertainty classification (as a random variable) (µ s -µ u ) 2 ·p s ·(1-p s ) may account for a substantial part of the variance if the expected values µ s and µ u should be very different. If the uncertainty is assumed to be fixed, this part would not be taken into account, which would lead to too narrow confidence intervals. This yields the (asymptotic) confidence interval: where the unknown parameters p s , µ s , µ u , ν 2 s , ν 2 u can be replaced by the empirical variance estimators as plugin estimators. Details on variance estimation are described in the following section.

Sample size calculation
For the sample size calculation, we introduce a type II error adjustment, a minimal (relative) standard deviation and consider costs.

Type II error adjustments
The required sample size calculation for the equivalence test in the case without partitioning can be obtained from Siebert and Ellenberger (2019). With the partitioned equivalence test, however, the challenge arises that a reduction of the sample size (initially) increases the manufacturer risk. In order to compensate for this increased risk, an adjustment of the sample size is necessary, which we call the recorded size. The sample size calculation of the partitioned equivalence test can be directly derived from that of the previous conventional equivalence test, with the previous standard error ν/ √ n replaced by the one determined in equation (18). This results in the following formula: The following applies to the variance on the total sample in case of partitioning (see Equation 18 summed over the full sample in the case q = 1) which allows the following simplified representation. Compared with the calculations for the previous equivalence test, this results in i.e. since 0 < q ≤ 1 and thus (1/q − 1) ≥ 0 and p s ≥ 0 it can always be viewed as multiplication of n e by a factor ≥ 1 and thus is an (apparent) enlargement of the sample. In fact, the partitioned equivalence test initially only increases the recorded sample, i.e. more comparative video footage, of which only a part is manually counted. The added value of first recording more and then omitting material again during the evaluation is that a higher proportion of unsafe events can be sighted compared to the original equivalence test. This property enables the partitioned equivalence test, after optimizing the costs, to make a more precise statement about the systematic error (or bias) of the counting error with less overall effort than the original equivalence test.

Minimal Standard Deviation
At very small sample sizes, it is difficult to reliably estimate any parameters, including the standard deviation, which is a problem known for example as small-sample bias (Hummel et al, 2005). This affects the equivalence test in general, but special attention must be paid to the partitioned equivalence test in particular, since sample sizes in the safe partition can be very small and errors very rare. To better understand the implications for practice, we ran simulations with real world APC system count errors (for more details, see Appendix C) and in some scenarios the user risk ≤ α indeed cannot be ensured for inappropriately small sample sizes. However, this is more of a hypothetical problem, since for very small sample sizes the overall chance to pass the test is below 15%, independent of the actual error D of the APC system (even for D = 0) and this can therefore not be made a sustainable business model for any APC manufacturer. From an authority's perspective, this is still undesirable because the chance of approving an APC system due to a poorly designed validation is greater than the user risk α implies. For any (partitioned) equivalence test, a minimal (relative) standard deviation ν min can be introduced as a restriction, which solves this problem: ifν < ν min is encountered anywhere it is replaced by ν min (in our casê ν sq < ν min orν u < ν min ). Surprisingly, this allows the partitioned equivalence test to operate even more theorycompliant than the original equivalence test. For the latter, introducing a low ν min like ν min = 3% has almost no effect, since there is no separtation into safe and unsafe videos and thus ν ≤ ν min only very rarely occurs.

Cost Management
To minimize costs (see also Appendix B), we introduce c s0 , the basic costs of a safe DOP. These include, for example, marginal costs of video recording, marginal costs of data preparation and execution of the additional algorithms, as well as costs due to time delays in carrying out further comparison procedures. Further, let c sZ be the costs incurred for a manual comparative counting of safe DOP so that it can be considered as a ground truth. The video data must be viewed by at least two human counters and by a supervisor in case of conflicts. These so-called counting costs are mainly composed of the personnel costs of the counters and the costs due to time delays in the testing process. Finally, c u is the (average) combined cost of the unsafe DOP, i.e. consisting out of to basic costs and the counting costs, since both are always carried out here. With the definition of the above partial costs, the total project cost of a validation process can be approximated by the following function: Cost(n,q) = n rec · p u ·c u + p s ·(c s0 + q·c sZ ) .
This approximation takes into account taking into multiplicity in which the partial costs occur in dependence of the total recorded sample n rec and the proportions of the subgroups. If n rec is already determined (for some reason) and greater than n from Equation (22), the optimal quota q 0 can be determined by solving that equation: In practice, however, n rec is to be determined and depends on q 0 . This can be done numerically by running a simple loop over possible q and checking the cost function. However, it can also be done analytically: With the model from Section 3 and using the sample size formula (22) we can optimise the total costs in relation to q: q 0 := Opt(q) = argmin q {Cost(n(q),q)} = argmin q n(q) · p u ·c u + p s ·(c s0 + q·c sZ ) = argmin q z 1-α/2 +z 1-β/2 2 · p s ·ν 2 = argmin q (z 1-α/2 +z 1-β/2 ) 2 p s ν 2 The derivative of the function f (q) = (q + a) · (1/q + b) = b · q + a/q + const to the variable q results in f (q) = b + a ·(−1/q 2 ) and with f (q min ) ! = 0 it follows, that for all q ≥ 0% and q ≤ 100% all non-marginal minima must hold q min = + a/b. Since costs diverge to infinity for q towards 0, the minimum of f (q) is min a/b, 100% and it follows

Application
The equivalence test as currently in use is summarized in Procedure 1 and our new partitioned equivalence test in Procedure 2.

Use Cases
Since the classification function itself can be completely arbitrary and due to the large amount of possible partitions (e.g. in the case of a recommended sample size of 6147 there are 2 6141 = 2.7 · 10 1850 possible partitions), we introduce use cases: 1. The original equivalence test: either all videos are classified as safe or all videos are classified as unsafe. In both cases, the partitioned equivalence test reduces to the original equivalence test.

2.
A simple classification, a so-called rule of thumb: this use case can e.g. depend on the automatic counts only, like sorting videos by their passengers per minute count and creating the partitions according to whether a certain threshold has been surpassed or not. By this method, door opening phases more prone to overcrowding are considered to be less safe.
3. Classification by using the first manual count: since in VDV 457 two manual counts plus a supervisor as a tie-breaker is required, the first manual count can already be used to classify whether a video is safe. Like in the second use case, videos are sorted according to their difference in manual and automatic count and split into safe and unsafe partitions using a certain threshold.
4. Classification by algorithms/artificial intelligence: the videos are processed by an additional algorithm, which produces an estimate of how difficult or unsafe it considers the video to be, which are handled like in the use cases before.
(a) Only use the safeness estimate and ignore the count of the APC system entirely. We take a look at this use case to determine whether there is an algorithm independent intrinsic video or scene difficulty. In case the additional algorithm and the APC system's algorithm are related, we expect greater savings.
(b) The additional algorithm is capable to create a count as well. Use the difference to the APC system's count and the safeness estimate as a tie-breaker if the count delta is zero.
5. Combined Classification: the methods above can be combined to yield better savings than the individual use cases themselves.
(a) Use cases 2 and 3: first, classify by a rule of thumb, then do a first manual count according to that classification. This approach is still entirely manual.
(b) Use cases 3 and 4b: as the case before, but with an algorithm instead of a rule of thumb. This approach has higher requirements, but may yield greater savings as well.
For an evaluation of the use cases performance, see Figure 5. APC: Neural APC 20190826_102009_16280 Figure 5: Possible savings for different use cases and APC systems. Red bars indicate an entirely manual workflow with only sensor counts necessary (e.g. from the APC system vendor) while blue bars indicate that the videos from the door opening phases were processed by an additional (possibly non APC system vendor) algorithm which yields both additional counts as well as a self-estimated confidence of correctness. In our case, this is an ensemble of 9 Neural APC (Jahn, 2019) agents. For the manual workflows, the savings are between 15% and 25% with the first manual count (use cases 3 and 5a) being slightly more effective than the rule of thumb (use case 2) solely. Still, costs can already be significantly reduced by reorganizing the current validation workflow and using the partitioned equivalence test. Using only a second algorithm (use case 4a) and interpreting its confidence estimation as a general scene difficulty shows mixed results: when the algorithm and the estimator share a common background, which is the case for the Neural APC 20190826 102009 16280 and the ensemble of the remaining agents, the savings are around 35%, while otherwise, they are typically below 20%, which is a little less than purely manual obtainable savings (red bars). However, when combining the sensor counts with the second algorithm and confidence, around 50% savings can be reached in average (use case 4b and use case 5b). For the simulation, passenger counts on a combined (2D and 3D) dataset of 11530 videos have been used, with at least three manual counts per video (six-eyes principle). The costs are stated in euro cent, compare Appendix B.

Discussion and Suggested Values
As can be seen in Figure 5, basically two use cases remain: the entirely manual use case 5a and use case 4b. Use case 2, the rule of thumb, is not effective enough and use case 3, the first manual count can often significantly profit from a pre-classification, turning it into use case 5a. For the algorithmic partitioned equivalence test, combined cases do not yield a lot of improvement, yet complicating the process.
Overall, as optimal parameters, p s = 90% is a common finding and ν s /ν = 35% can be assumed. For q, in purely manual methods, q = 35% seems suiteable, while q = 17.5% proved viable for the algorithmically assisted use cases (compare figures in Appendix C).

Conclusion
Our investigations have shown that a seamless connection of the partitioned equivalence test to current requirements from VDV 457 (and the equivalence test itsself) is possible. Here the standard benefits considerably from the completed changeover from the t-test to the equivalence test. The partitioned equivalence test allows statistically robust samples to be realized at significantly reduced costs when compared to the classic equivalence test. Not only can costs be reduced, but also more manual tests can be carried out with the same monetary investment, thus increasing quality. Measured by the volume of funds to be distributed by revenue sharing, shortcomings in passenger counting validation are not justifiable in economic terms anyway. The partitioned equivalence test is not exclusively limited to the use of algorithms, as shown by the classification by the first manual counting, a rule of thumb or a combination of both. Basically, any information available about the counting behavior of the sensors can be used to reduce costs -in a statistically robust way and thus without increased risk for the user. There is also no vendor lock-in, as all relevant calculations can be performed with simple spreadsheet formulas without specialized software. In the medium to long term, the partitioned equivalence test lays the foundation for a deeper integration between manual and automatic counting to raise count quality to a new level. This will not only benefit revenue distribution, but also forecasting and real-time in-vehicle passenger counts. The new test will be an incentive for transport companies to invest in the corresponding IT infrastructure. In other industries, e.g. the tech industry, hybrid systems, in which people and algorithms work together, are already common, e.g. in fraud detection. We hope that the partitioned equivalence test will not only help automatic passenger counting catch up with state-of-the-art technologies, but will even make it a technological pioneer other fields can profit from.

A Confidence intervals for sums with random weights
In the following we use the notation of Section 2.2. Let X = (X i ) i=1,..,n the counting errors of an APC system, independent, identically distributed with probability measure P X , E(X 1 ) = µ and Var(X 1 ) = ν. The indicator W i represents the (random) classification into safe and unsafe door opening phases (DOP). Then, W i is Bernoulli(p s )distributed, in the sense that the outcome is safe with likelihood p s and unsafe with likelihood p u =1-p s (W i ∼ P W ∼ Bin(1, p s )). The total number of safe DOP is N s = W i = 1 {Wi=s} which is the sum of all W i and p s = N s /n. We now consider the random variables X i |W i , i.e. the distribution of errors in the case that a DOP is safe resp. unsafe. The associated probability measure of the conditional distribution X|W always exists, since W is integer and further the assumption P(W i =s)>0 holds, since otherwise it would be trivial. Note that X i and W i are not needed to be stochastically independent. Let E(X i |W i =u) = µ u and E(X i |W i =s) = µ s . Then for the expected value the following holds: Further, let now be i s (k) for k=1,..,N s the assignment in ascending order to the (random) variables of the safe DOP, such that for i = i s (k) the notation X sk := X i , W sk := W i , etc. is well defined and i u (k) for k=1,..,N u the analogous assignment to the variables of the unsafe DOP (X uk ,W uk , etc.). Methods are now required to construct a range-preserving confidence interval if only a previously specified proportion q 0 (e.g. 50%) of the safe DOP to be manually counted. First an adjustment (upwards) of the pre-specified q 0 takes place, such that the number of safe DOP to be counted is integer q := q 0 ·N s /N s .
A random sample of q·N s safe DOP can be implemented with Z=(Z k ) 1,..,Ns ∈ {0,1}, such that Ns k=1 Z k = q·N s holds. The resulting (Z k ) 1,..,Ns are by design independent of all measured variables. For i = 1,..,n let with i = i s (k) the weights be defined C i = Z k /q for the safe DOP and with i = i u (k) the corresponding C i = 1 for the unsafe DOP.
The estimator for µ u will be the mean value and for µ s one can state a similar estimator, which we will refer to as leave (1-q) out estimator: An estimator for µ can now be constructed from the individual estimators as follows: All values X uk where Z k is zero are not needed. These are the values that are not needed for the calculation, i.e. can be discarded. Then, = 1 n · (p s ·n · µ s ·E(Z k )/q + p u ·n · µ u ) = p s ·µ s + p u ·µ u = µ and thus X q is an unbiased estimator which can be calculated on a reduced data set. For the variance of the estimator the following holds: ·δ and union of the sums of the safe and unsafe DOP it follows: Furthermore, for the variance of a product of independent random variables C and X the identity Var(C ·X) = E(C) 2 ·Var(X) + E(X) 2 ·Var(C) + Var(C)·Var(X) holds and thus Var(Z 1 /q·(X s1 -µ s )) = E(Z 1 /q) =1 2 ·Var(X s1 )+E(X s1 -µ s ) =0 2 ·Var(Z 1 /q)+Var(Z 1 /q)·Var(X s1 ) = Var(X s1 )/q , because Z 1 has a hypergeometric distribution and thus Var(Z 1 /q) = 1/q -1 holds. Thus, Var( X q ) = 1 n p s ·Var X s1 /q + (1-p s )·Var X u1 + (µ s -µ u ) 2 ·p s ·(1-p s ) .

B Practical guidance on parameters for cost functions
For calculations regarding the planning of an APC validation, a plausible cost function must be assumed: (a) Video recording time t {video, i} in hours (b) The acceleration factor r A/V = t labor /t video of the video corresponds to the ratio of working time t labor to recording time t video , i.e. r A/V < 1 corresponds to an acceleration, r A/V = 1, that viewing the footage takes just as long as its duration and r A/V > 1 corresponds to a slowdown compared to the video duration.
(c) The labor cost c labor , which is usually equal to the hourly wage of the manual comparison counters.
(d) Additional costs due to the second manual count and the supervisor: r S Then the following basic costs are derived from this: Empirical results with manual counts, which were performed with untrained personnel and with software optimized for the workflow, allow the following approximate values: This results in average counting costs of per door opening phase (DOP). If the recording costs are assumed to be zero, the total costs of the uncertain DOP are calculated as follows: while two cases must be distinguished here for the safe DOP. The first case represents the classification without manual counting: The second case when the first manual count is included: Cases in which only a partial count is performed are calculated analogously as partial totals over the respective DOP. For the use cases described in Section 5.1, different classification options result in safe and unsafe DOP. The better a classification succeeds, with simultaneously low costs, the greater is the overall savings potential of the process. A special approach here is the combined classification, e.g. consisting of cases 2 and 3: a rule of thumb and the first manual count. With such a combined classification, the costs incurred can also be determined in a simple manner. This is done by considering a more detailed partitioning: the safe DOP are those that have already been classified as safe using the first classification rule (e.g. rule of thumb) and those that were initially classified as unsafe, but were checked using the second classification rule (here the manual count) and were reclassified as safe due to lack of discrepancies in the counts. Theory (Lines) and Normal Distribution Simulation (Markers, Runs: 10000) Actual Standard Deviation = 15%, Zero Success Chance (Theory) for 7.5% Figure 6: Equivalence Test Success Theory and Simulation with a normal distribution. The overlap is almost perfect: at ν = 7.5%, the success chance is supposed to be zero analytically, from which the simulation deviates slightly. For a comparison with actual passenger counting system errors, see Figure 7, Figure 8 and Figure 9.
is important for appropriate allocation of costs. The counting costs incurred by the reclassified DOP for the initial manual (comparative) count are now added to the basic costs, since they may have been incurred without the ground truth for that DOP actually being finally determined. In contrast, this attribution does not happen for unsafe DOP, where both classification rules have provided an unsafe classification, since for these the ground truth must be determined in any case. Since full counting costs are therefore always incurred here, the sequence is irrelevant. In summary, the costs of the safe DOP are as follows: Ns Ns i=1 W reclass.,si · c (Z,si) (51) (1 − W reclass.,si + r S ) · c (Z,si) .
An application of this approach is shown in Figure 5.  : Test Success Simulation, comparing the equivalence test with the partitioned equivalence test, no ν min resp. ν min = 0%. The difference of D to a perfect normal distribution (compare Figure 6) is that it has a slightly higher chance of test success than it should have for low sample sizes. Even though success chances of around 10% to 15% across all D will not allow any APC manufacturer any kind of sustainable business, in order to ensure the bounded user risk promise of success chance ≤ α/2 = 2.5% for |D| ≥ ∆ = 1%, the introduction of ν min > 0 can close the theoretical gap as can be seen in Figure 8 and Figure 9.  Figure 7, even for low sample sizes, test success ≤ α/2 = 2.5% for |D| ≤ ∆ = 1% is ensured. To our surprise, the partitioned equivalence test can even be closer to the theoretical success probability than the original equivalence test: we use ν min for both tests, but only in the non-partitioned test, the threshold is never triggered, since safe and unsafe videos are mixed up with ν being around 10%. Also, the partitioned equivalence test has a larger n rec to draw videos from.  : Test Success Simulation, comparing the equivalence test with the partitioned equivalence test, ν min = 3%. As in Figure 8, test success ≤ α/2 = 2.5% for |D| ≤ ∆ = 1% is ensured. However, ν min = 5% penalizes the partitioned equivalence test much stronger than ν min = 3%, increasing costs or increasing the manufacturers risk. Therefore, we suggest ν min = 3%.