{'fraction_non_alphanumeric': 0.03393531633616619, 'fraction_numerical': 0.012551148882593642, 'mean_word_length': 4.551878549584972, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 12, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 14, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Online multiple hypothesis testing for reproducible research

Modern data analysis frequently involves large-scale hypothesis testing, which naturally gives rise to the problem of maintaining control of a suitable type I error rate, such as the false discovery rate (FDR). In many biomedical and technological applications, an additional complexity is that hypotheses are tested in an online manner, one-by-one over time. However, traditional procedures that control the FDR, such as the Benjamini-Hochberg procedure, assume that all p-values are available to be tested at a single time point. To address these challenges, a new field of methodology has developed over the past 15 years showing how to control error rates for online multiple hypothesis testing. In this framework, hypotheses arrive in a stream, and at each time point the analyst decides whether to reject the current hypothesis based both on the evidence against it, and on the previous rejection decisions. In this paper, we present a comprehensive exposition of the literature on online error rate control, with a review of key theory as well as a focus on applied examples. We also provide simulation results comparing different online testing algorithms and an up-to-date overview of the many methodological extensions that have been proposed.


Introduction
Large-scale hypothesis testing is now ubiquitous in a variety of biomedical and technological applications. For example, many major technology companies perform tens of thousands of randomised controlled experiments (known as A/B tests) each year to make data-driven decisions about how to improve products [Kohavi et al., 2020]. Meanwhile, in genomics it is now routine to test hundreds of thousands of genetic variants for an association with particular phenotypic trait(s). Even in the setting of randomised controlled trials (RCTs) in medicine, there is a growing push towards the use of "overarching" trial frameworks to allow the efficient testing of multiple experimental drugs for multiple patient subpopulations.
Performing a large number of hypothesis tests naturally gives rise to the problem of multiple comparisons [Tukey, 1953]: given a collection of multiple hypotheses to be tested, the goal is to distinguish which hypotheses are null and non-null, while controlling a suitable error rate (see Section 1.2). This error rate is generally formed around the probability of incorrectly classifying a null hypothesis as non-null. Typically, a p-value is calculated for each hypothesis and is then used to decide whether to reject the null hypothesis. Multiple hypothesis testing is one of the core problems in statistical inference, and has led to a wide range of procedures that can be used to correct for multiplicity and ensure that a suitable error rate is controlled. In contrast, uncorrected hypothesis testing contributes to serious concerns over reproducibility, publication bias and 'p-hacking' in scientific research [Ioannidis, 2005, Head et al., 2015.
Traditionally, multiple hypothesis testing is offline in nature, in the sense that a procedure for testing N hypotheses will receive all of the corresponding p-values (p 1 , . . . , p N ) at once.
Step-up and step-down multiple testing procedures (for example) require knowledge about all p-values in advance. However, this paradigm is often incompatible with modern data-driven decision-making processes, as demonstrated by our motivating examples in Section 1.1. Once the data is available to make a decision about a particular hypothesis, it can be desirable to take a corresponding action (e.g. to update a tech product) relatively quickly, and not to wait for the results of ongoing or future hypothesis tests. Linked with this, in many application areas it may not even be possible to know in advance how many tests in total will be performed.
Moreover, the repeated application of traditional offline multiple testing procedures as the family of hypotheses grows can lead to repeatedly changing past decisions, which is clearly undesirable from the practical perspective of wanting to act in a timely manner on test results.
What is needed therefore are procedures for online multiple hypothesis testing, which better take into account the nature of modern data analysis. This is defined as follows: A stream of hypotheses arrives online. At each step, the analyst must decide whether to reject the current null hypothesis without having access to the number of hypotheses (potentially infinite) or any future data, but solely based on the previous decisions and evidence against the current hypothesis.
Online hypothesis testing has a sequential nature, in the sense that individual hypotheses (or batches of hypotheses) are tested one after the other over time. However, this is distinct from the more traditional concept of sequential testing, which refers to the testing of a single hypothesis in a sequential manner. In sequential testing, the sample size for the experiment is not fixed in advance, and the accumulating data is evaluated as they are collected to allow the experiment to be stopped adaptively, such as when statistical significance is achieved. The framework of online multiple testing can be expanded to be "doubly sequential", where the inner sequential process is a single sequential test, and the outer sequential process refers to the multiple experiments that are performed to test different hypotheses.
We also wish to draw a distinction between online hypothesis testing and multi-armed bandit (MAB) testing. While both frameworks allow the comparison of multiple experimental arms over time, an MAB can be considered as a single experiment in which resources are iteratively allocated to the different arms in order to adaptively trade off certain costs and benefits, and this allocation depends on the previously observed outcomes on each arm. Again, the two testing frameworks can be combined within a doubly sequential framework where there is a sequence of MAB problems over time, see Yang et al. [2017]. Figure 1 gives a diagrammatic representation of online multiple testing, where different hypotheses (corresponding to experiments) are tested over time (corresponding to the collection of data samples). As discussed above, each experiment could itself be a sequential experiment or take the form of an MAB. Figure 1: An abstract online multiple testing framework. As time passes (left to right), new experiments testing different hypotheses are started and stopped, in a possibly indefinite manner. Each horizontal line represents a new experiment/hypothesis, and the length represents the number of samples collected. Decisions about each hypothesis must be made as soon as the corresponding experiment ends.
Since the framework was first proposed by Foster and Stine [2008], a variety of procedures that control error rates for online hypothesis testing have been developed [Aharoni and Rosset, 2014, Javanmard and Montanari, 2015, Ramdas et al., 2018. Our aim in this paper is to provide an expository overview of this literature on online error rate control, with a review of the underlying theory, key methods and applied examples.
In the rest of this section, we give motivating examples for online multiple testing and present formal definitions of error rate control. Section 2 describes the key procedures for online error rate control in detail. Section 3 presents a simulation study of the procedures, while Section 4 presents two case studies of applying online error rate control. We describe further methodological extensions as well as future directions in Section 5, and conclude with a discussion in Section 6.

A/B testing in tech companies
The development of web applications and services in the tech industry increasingly relies on the use of randomised controlled experiments known as A/B tests. There are a number of widely used platforms now available that streamline and handle the implementation of A/B tests.
A typical application is in the development of different versions of webpages. As described in Berman and Van den Bulte [2021], in this context there are two webpage variations (A and B). When an online user visits, the platform randomly assigns the visitor to one of the variations for the duration of the experiment. The platform records the actions that the visitor takes, where the monitored actions reflect the experimenter's goal(s), such as increasing visitor engagement (defined appropriately) or increasing revenue. One of the variations is designated as the baseline, and the performance of the other variation is compared to the baseline using suitable test statistics. If run correctly using anytime-valid p-values [Johari et al., 2021] and/or confidence sequences [Howard et al., 2021], the data can be continuously monitored by the experimenter, allowing a decision to be made at any stopping time.
Many tech companies run tens of thousands of A/B tests per year, as part of a continuous process of designing, delivering, monitoring and improving webpages and other web services.
However, there is reason to reduce the number of false alarms which result in making changes to web products that do no better (or even perform worse) than the current iteration, corresponding to an incorrect rejection of the null, particularly when such changes are potentially costly or disruptive to users. Hence, the framework of online error rate control provides a framework to do so while still allowing a large number of A/B tests to be performed in a flexible manner.

Data repositories
Public databases and shared data resources are becoming increasingly pervasive and important in modern biomedical research, particularly in the fields of genetics, molecular biology and routinely collected healthcare records. Some well-known examples include the 1000 Genomes Project [1000Genomes Project Consortium, 2015 and the Wellcome Trust Case Control Consortium [Wellcome Trust Case Control Consortium et al., 2007]. Another example is the International Mouse Phenotyping Consortium database [Koscielny et al., 2013, Dickinson et al., 2016, which we describe as one of our case studies in Section 4. Meanwhile, the increase in routinely collected healthcare data has allowed evaluation of different healthcare technologies currently used in practice through emulation of target trials [Dickerman et al., 2019].
Multiple testing naturally occurs in this setting in two ways. Firstly, such databases can be accessed by multiple independent researchers at different times. When a researcher or research group comes up with a new hypothesis, they can fetch the relevant data from a database and perform a statistical test. Secondly, in many databases the family of hypotheses to be tested grows over time as new data is continually added. In both of these scenarios, the number of hypotheses being tested will be unknown and potentially very large.
In order to control the number or proportion of false discoveries in this context, new procedures are required that allow a researcher to decide whether to reject a current hypothesis with minimal information about previous hypotheses, and without prior knowledge of even the number of hypotheses that are going to be tested in the future. This is precisely the online multiple testing framework described earlier.

Platform trials
A platform trial has a single master protocol that evaluates multiple treatments across one or more patient types, and allows a potentially large number of treatments to be added during the course of the trial [Saville and Berry, 2016]. Treatments are dropped from the trial after they have been formally tested for effectiveness. Such a trial could (in theory) 'perpetual' in that new treatments can continue to enter into the trial and be tested. In a platform trial, treatments are introduced at different time points by design. However, the trial investigators will wish to make a decision on whether a treatment is beneficial as soon as the data are ready, without waiting for results from the other treatment arms. Hence, the treatment effects are tested sequentially in an online manner, where the number of treatments to be tested in the future may be unknown. More formally, a platform trial generates a sequence of null hypotheses (H 1 , H 2 , H 3 , . . .) which are tested sequentially. Hypothesis H i tests the value of some parameter θ i , such as an estimate of the treatment difference compared to a control arm. Note that the control arm could also change during the trial, for example due to one of the previously evaluated experimental arms being confirmed as effective and becoming the new control.
The p-values generated from the platform trial described above will not be independent in general. Dependencies will primarily arise due to the shared control data that is re-used to test multiple hypotheses. A current example of a long-running platform trial is the STAMPEDE trial [James et al., 2008] for patients with locally advanced or metastatic prostate cancer, which we return to as a case study in Section 4.2.

Error rates
We now formally define some error rates of interest. The basic problem setup is as follows.
At each time step t = 1, 2, . . . the experimenter observes a p-value P t corresponding to a null hypothesis H t , and must make a decision whether to reject H t before the next time step. We assume that all p-values are valid, i.e. if the null hypothesis H t is true, then Pr{P t ≤ x} ≤ x for all x ∈ [0, 1]. At time t = 0, the experimenter fixes the level α at which a suitable error rate is meant to be controlled at all times.
A general testing procedure provides a sequence of test levels α t with decision rule . (1) At any time T , let R(T ) = T t=1 R t denote the number of rejections (also known as discoveries) made so far and V (T ) denote the total number of falsely rejected hypotheses (also known as false discoveries).
The false discovery proportion at time T is defined as The false discovery rate is then the expectation of the FDP: A commonly studied variant is the marginal FDR: Another related error rate is the false discovery exceedance (FDX), which is the probability the FDP exceeds a predefined threshold : An alternative error rate to those based on the FDP is the familywise error rate (FWER), which is more commonly considered in clinical trial contexts. The FWER is the probability of falsely rejecting any null hypothesis: The FWER and hence the FDR can be controlled at level α in a simple manner by using a Bonferonni-type correction, also known as alpha-spending. More precisely, we can choose significance levels α t for H t , such that ∞ t=1 α t = α. However, this method suffers from a low statistical power, with the probability of the null hypothesis H t being rejected tending to zero as t increases. This motivates the development of more sophisticated algorithms for online error rate control.
2 Online error rate control methodology 2.1 Generalised alpha investing (GAI) The first proposals for online error rate control were based on "alpha investing" by Foster and Stine [2008] and its generalization (GAI) [Aharoni and Rosset, 2014]. These have an intuitive interpretation, described next. Any GAI rule begins with an error budget, or alpha-wealth, which is allocated to the different hypothesis tests over time. That is, there is a price (or penalty) to be paid each time a hypothesis is tested. However, when a new discovery is made (i.e. a hypothesis is rejected) some of the alpha-wealth is earned back, which can be viewed as a return on the alpha-investment. Hence as long as discoveries continue to be made, hypotheses can be tested indefinitely without the test levels tending towards zero.
Formally, a GAI rule produces a series of test levels (α 1 , α 2 , α 3 , . . .) based on which it uses (1) to produce the corresponding decisions (R 1 , R 2 , R 3 , . . .). Of course, α t must be based only on R 1 , . . . , R t−1 . At each time point t, the rule updates the alpha-wealth W (t) as follows: In other words, the penalty φ t is the amount paid for testing a new hypothesis, and the payout (or reward) ψ t is the amount earned if a discovery is made at that time. Hence the initial wealth is W (0) = w 0 and it is updated via: The parameter w 0 and sequences α t , φ t , ψ t are all user-defined. The total wealth W (t) must always be non-negative, and hence φ t ≤ W (t − 1). Additionally, there are restrictions on α t , φ t , ψ t in order to control the FDR (or its variants), namely that when a rejection is made, the payout ψ t is capped. As is predominately the case in offline multiple testing, we often use monotone decision rules for α t considered as a function of (R 1 , . . . , R t ): ifR i ≥ R i for all Ramdas et al. [2017] defined a class of improved GAI algorithms, called GAI++, as follows.
We set w 0 so that 0 ≤ w 0 ≤ α and choose the reward ψ t to satisfy Theorem 2.1. If the null p-values are independent of all other p-values, any monotone GAI++

Algorithms for online FDR control: LORD, SAFFRON and ADDIS
Although an "algorithmic perspective" led to the GAI procedures initially used in the online testing literature, Ramdas et al. [2017] posited a "statistical perspective" to construct procedures, which is to keep an estimate of the FDP less than α. First, the oracle FDP is defined where H 0 denotes the set of true null hypotheses. If we can keep FDP * (t) ≤ α at all times t, then (depending on dependence assumption on the p-values) we can prove that mFDR(t) ≤ α or FDR(t) ≤ α. This technique has been used to derive the LORD, SAFFRON and ADDIS algo- for FDP * (t).

LORD
The LORD algorithm was conceptualised by Javanmard and Montanari [2018], and is an instance of a GAI. Following that, Ramdas et al. [2017] defined a simple upper bound of FDP * (t): and showed that LORD can be viewed as an algorithm that keeps FDP LORD (t) ≤ α. Exploiting this view, they derived a uniform improvement of LORD, termed LORD++, presented next.
Given an infinite non-increasing sequence of positive constants {γ t } ∞ t=1 that sums to one, the test levels α t for LORD++ are chosen as follows: where τ j denotes the time of the j-th rejection. The above formula may look daunting but it is interpretable. The first term is the fraction of the initial wealth w 0 that is used by the t-th test.
The other terms are the fractions of the earnings from rejections before t that are spent in the t-th round: LORD++ awards α − w 0 for the first rejection and α for every subsequent rejection, and on receiving this reward, the method immediately allocates that reward to future rounds according to the same schedule of constants {γ t }, shifted to start at the next instant. This rule ensures that LORD++ never spends more than it has earned, thus keeping FDP LORD (t) ≤ α.
The manner in which FDP LORD (t) is a simple upper bound on FDP * (t) is reminiscent of the seminal Benjamini-Hochberg (BH) procedure, which can be derived in a similar fashion. This leads us to view LORD++ as the online analog of the BH procedure.
Guarantees for LORD hold under different p-value dependencies, which we now formalise.
Define the filtration at time t as F t = σ(R 1 , . . . , R t ) and Finally, in terms of theoretical power guarantees for LORD, Chen and Arias-Castro [2021] considered the setting of a (generalised) Gaussian model and showed that LORD is asymptotically optimal, in particular by being as powerful as BH to first asymptotic order. et al. [2018] derived an adaptive version of LORD++ called SAFFRON, which is based on an estimate of the proportion of true null hypotheses. By not wasting its earnings on attempting to reject weaker signals (i.e. larger p-values), SAFFRON preserves alpha-wealth and hence can have a higher power than LORD++.

Ramdas
To this end, choose λ ∈ (0, 1) and define the candidate p-values as those that satisfy P t ≤ λ, since SAFFRON will never reject a p-value larger than λ. For the extension to a sequence Ramdas et al. [2018]. SAFFRON then proceeds as follows: 1. At each time t, define the number of candidates after the j-th rejection as Hence, SAFFRON starts off with alpha-wealth (1 − λ)w 0 and does not lose any of this wealth when testing candidate p-values. It gains an alpha-wealth of (1 − λ)α for each discovery after the first. SAFFRON can make more rejections than LORD if there is a significant fraction of non-nulls and the signals are strong.
Similarly to LORD, SAFFRON provably controls the mFDR at all times if the null p-values are conditionally super-uniformly distributed. SAFFRON provably controls the FDR at all times if the null p-values are independent of each other and of the non-nulls, and {α t } is a monotone function of (R 1 , . . . , R t−1 , C 1 , . . . , C t−1 ); see Ramdas et al. [2018] for full details.

ADDIS
ADDIS stands for an ADaptive algorithm that DIScards conservative nulls, and was proposed by Tian and Ramdas [2019]. ADDIS can invest alpha-wealth more effectively than LORD or SAFFRON by explicitly discarding the weakest signals (i.e. the largest p-values) in a principled way, which can lead to a higher power. More formally, in practice it is common to encounter Often nulls are uniformly conservative, which means that under the null, For example, for a one-dimensional exponential family with parameter θ, when the true parameter θ is strictly smaller than θ 0 , the uniformly most powerful test of H 0 : θ ≤ θ 0 versus H 1 : θ > θ 0 will give uniformly conservative nulls [Zhao et al., 2019]. Another setting is using always-valid p-values [Johari et al., 2021] in the context of continuous monitoring for A/B testing, which will always be conservative.
With R t , C t and C j+ defined as for SAFFRON, we now also define S t = 1{P t ≤ η} as the indicator of H t being selected for testing (i.e. not discarded). Hence η is the discarding threshold, which can also be extended to a sequence {η t } ∞ t=1 , see Tian and Ramdas [2019]. We also choose an infinite non-increasing sequence of positive constants {γ t } ∞ t=0 that sums to one. The testing levels for ADDIS are then given by α t = min{λ,α t }, wherê

Simulation studies
In this section, we compare the performance of the LORD, SAFFRON and ADDIS algorithms in terms of the FDR and statistical power. For reasons of space, we do not aim to give a comprehensive set of simulations but instead focus on a few key settings to give a flavour of typical results. In Section 3.3 we refer the reader to further simulation settings that have been published in the literature. First though, we briefly describe software implementation of algorithms for online error rate control.

Software: onlineFDR package
The onlineFDR package is an open-source R package that aims to provide a comprehensive and up-to-date implementation of algorithms for online error rate control. It is freely available via Bioconductor [Robertson et al., 2021]. The package implements the LORD, SAFFRON and ADDIS algorithms, as well as almost all of the algorithms corresponding to the further extensions of online error rate control methodology (see Section 5.1). In particular, it also provides functions for algorithms for online FWER and online FDX control. The package documentation provides a user-friendly introduction to the use of the package, and there is also a Shiny app available [Liou and Robertson, 2021] to allow users to explore algorithms for online FDR control in an interactive way without having to program. All results for the simulation studies and case studies in this paper were calculated using the package.

Testing with Gaussian observations
In order to examine the relative performance of the online FDR algorithms, we use a simple experimental setup of testing Gaussian means, with a total of T hypotheses. The null hypotheses take the form H t : µ t ≤ 0 which are tested against the alternative H t : µ t > 0 for t = 1, . . . , T .
We observe independent observations Z t ∼ N (µ t , 1) which are transformed to one-sided p-values P t = Φ(−Z t ), where Φ denotes the standard Gaussian CDF. The means µ t are set according to the following mixture model: where F 1 ∼ N (3, 1) and F 0 is defined as below.
We use the default settings for LORD, SAFFRON and ADDIS that have been suggested in the literature. For LORD, we choose the sequence γ t ∝ log(t∨2) t exp( √ log t) , which has been shown to maximise its power in the Gaussian setting [Javanmard and Montanari, 2018]. We also use this choice of γ t for alpha-spending, where {α t } ∞ t=1 is simply given by α t = αγ t . For SAFFRON we set λ = 0.5 and γ t ∝ 1 t 1.6 . Finally, for ADDIS we set λ = 0.25, η = 0.5 and γ t ∝ 1 (t+1) 1.6 . Figure 2 shows how the test levels {α t } ∞ t=1 (displayed on the log 10 scale) evolve over time for LORD, SAFFRON and ADDIS compared with uncorrected testing (where α t ≡ α) and alpha-spending. Here, T = 300, α = 0.05, π 1 = 0.5 and we choose F 0 ≡ 0.
All of the online FDR algorithms have higher test levels than alpha-spending (apart from Figure 2: Test levels for LORD, SAFFRON and ADDIS compared with uncorrected testing and alpha-spending. We set T = 300, α = 0.05 and the proportion of non-nulls π 1 = 0.5. LORD briefly early on in this particular experiment). The relative difference increases with t as the online algorithms 'earn back' wealth over time, which alpha-spending cannot do. Both SAFFRON and ADDIS have higher test levels than LORD, reflecting how they can more efficiently invest the alpha-wealth. In this setting, since µ t = 0 under the null, the nulls are exactly uniform and so ADDIS cannot take advantage of conservative nulls. Hence the testing levels of ADDIS are similar or slightly lower than those for SAFFRON. Finally, we see that SAFFRON has similar test levels as uncorrected testing, and both SAFFRON and ADDIS can even have test levels above the nominal α. Figure 3 compares the statistical power of LORD, SAFFRON and ADDIS compared with uncorrected testing and alpha-spending, as π 1 varies from 0.01 to 0.9. Here, we define power as where H 1 denotes the index set of the non-null hypotheses. We also include the standard Benjamini-Hochberg (BH) procedure as an additional comparison. We stress that BH is an offline procedure and so could not be used for online testing in practice. In our simulation, we set T = 1000, α = 0.05 and F 0 ∼ N (−0.5, 0.1). Results are based on averaging 10 4 simulation replicates.
Starting with alpha-spending, as expected the power is very low (< 0.2) for all π 1 . LORD has substantial power gains compared with alpha-spending (as long as π 1 is not close to zero) and this advantage increases with π 1 . However, LORD has substantially lower power than BH for all values of π 1 . As expected, SAFFRON performs better as the fraction of non-nulls π 1 increases, with a higher power than LORD for π 1 > 0.05, BH for π 1 > 0.5 and even uncorrected testing for π 1 > 0.7. Since F 0 ∼ N (−0.5, 0.1), almost all the means for the null hypotheses will be negative, i.e. we are in a setting with conservative nulls. Hence, as expected, ADDIS outperforms SAFFRON in terms of power (except for very high values of π 1 ). ADDIS also has a higher power than BH for π 1 > 0.2 and uncorrected testing for π 1 > 0.6.
In the Appendix ( Figure 5) we show the corresponding FDR for all of the algorithms considered. We see that uncorrected testing can have substantial inflation of the FDR, with the FDR Figure 3: Power of LORD, SAFFRON and ADDIS compared with uncorrected testing, the BH procedure and alpha-spending as the proportion of non-nulls π 1 varies. We set T = 1000 and α = 0.05. Results are based on 10 4 simulation replicates.
inflated above the nominal α = 0.05 level for π 1 < 0.3. The FDR reaches as high as 0.65 when π 1 = 0.01. All other algorithms control the FDR below the nominal 0.05 level, as expected.

Observations from other simulations
Here, we summarise a few take-home messages for LORD, SAFFRON and ADDIS from simulation results already found in the literature. Javanmard and Montanari [2018] investigated the effect of the ordering of the hypotheses for online testing rules, including LORD. In some applications, hypotheses can be ordered using side information, such that those that are most likely to be rejected come first. With this favourable ordering, the statistical power of LORD can substantially increase as long as π 1 is not too large (since ordering is less relevant in that case). Similar findings for LORD, SAFFRON and ADDIS in the context of platform trials can be found in , which also looked at the adversarial setting where hypotheses happen to be ordered so that those most likely to be rejected come last, resulting in a lower statistical power. Ramdas et al. [2018] considered the impact on LORD and SAFFRON of choosing sequences of the form γ t ∝ t −s , where the parameter s > 1 controls the 'aggressiveness' of the procedure (since the larger the value of s, the more the alpha-wealth is concentrated at small values of t).
For Gaussian alternatives, the simulation results suggested that less aggressive sequences are to be preferred in terms of increased power for SAFFRON and LORD. Meanwhile, Tian and Ramdas [2019] showed that ADDIS can match the power of SAFFRON when the nulls are not conservative (i.e. uniform nulls). The power advantage of ADDIS over LORD and SAFFRON increases the more conservative the nulls are, i.e. the more negative the means for the null hypotheses are (in the Gaussian setting).
The theory presented in Section 2 for provable FDR control requires null p-values to be independent of one another (with a weaker condition sufficing for mFDR control). Robertson and Wason [2018] explored the performance of online testing rules, including LORD and SAFFRON, when p-values are correlated with each other. In particular, the test statistics are assumed to come from a multivariate normal distribution where the covariance matrix has ones along the diagonals and off-diagonal entries equal to ±0.5. There was no evidence of FDR inflation when using LORD under a range of non-null distributions, showing empirically that it is robust to reasonable departures from independence. However, with a two-sided test under a Gaussian alternative, the SAFFRON procedure had an inflated FDR for smaller values of π 1 .
This inflation persisted and even increased as T increased from 100 to 1000.
More generally, the simulation studies in Robertson and Wason [2018] also highlighted the value of using 'bounded' versions of online testing algorithms. This requires setting an a-priori upper bound M on the number of hypotheses to be tested, so that the γ t ≡ 0 for t > M , which allows setting γ t ≡ 1/M for t ≤ M (for example). The bounded versions have a uniformly higher power than the versions presented in Sections 2 (with the default choices of γ t given in Section 3.2) which assume no upper bound on T , and empirically a substantial gain can be observed for small T (i.e. T < 100). Finally, another general observation is that the power advantages of online testing algorithms compared with alpha-spending increase as T increases.
Indeed, when T is small and π 1 is low, online testing algorithms may no longer be competitive in terms of power. We return to this issue in Section 5.2.

Case studies 4.1 IMPC dataset
Our first case study uses high-throughput phenotypic data from the International Mouse Phenotyping Consortium (IMPC) data repository, which aims to generate and phenotypically characterize knockout mutant strains for every protein-coding gene in the mouse [Koscielny et al., 2013]. The IMPC database is an example of a growing dataset mentioned in Section 1.1, since the family of hypotheses is constantly growing as new knockout mice lines are generated and phenotyping data is uploaded to the data repository.
We focus on the analysis of IMPC data performed by Karp et al. [2017], who looked at the influence of sex in mammalian phenotypic traits in both wildtype and mutants. As part of their analysis, Karp et al. analysed the role of sex as a modifier of the genotype effect (for continuous traits) using a two stage pipeline. Stage 1 tested the role of genotype using a likelihood ratio test comparing models (a) and (c). Similarly, stage 2 tested the role of sex using a likelihood ratio test comparing models (a) and (b).
(a) Y ∼ Genotype + Sex + Genotype * Sex + Weight + (1|Batch) The above procedure resulted in two sets of N = 172 328 distinct p-values. Note that these p-values will not be independent, due to positive and negative associations between different genes (caused for instance by linkage disequilibrium). In addition, multiple variables are being measured for the same gene, and these can be aspects of the same phenotype or be biologically correlated. Table 1 below shows the number of traits that had a statistically significant genotype effect or were classed as having a statistically significant sexual dimorphism (SD) using LORD, SAFFRON and ADDIS. As a comparison, we include the results from alpha-spending, BH and uncorrected testing. The 'Fixed' procedure is the fixed significance threshold of 0.0001 used in practice for the IMPC pipeline. Starting first with the results for the genotype data, the online testing algorithms make two to three times as many rejections as fixed testing. ADDIS and SAFFRON in turn make substantially more rejections than LORD, with an increase of almost 50% and 70%, respectively.

Genotype SD
ADDIS makes a similar number of rejections to BH, but SAFFRON makes noticeably more rejections than both BH and ADDIS for these data. For the SD data, again the online testing algorithms make substantially more rejections than fixed testing, but the relative increase is much less. ADDIS and SAFFRON make a very similar number of rejections, about 50% more than the number of rejections for LORD.

Platform trial: STAMPEDE
Our second case study is the ongoing STAMPEDE (Systemic Therapy for Advancing or Metastatic Prostate Cancer) platform trial, which evaluates the effect of systemic therapies for prostate cancer on overall survival [James et al., 2008]. The trial started with 5 experimental treatment arms (B -F), and compared these with the control arm A, which was standard-of-care (SOC) hormone therapy. Figure 4 shows a schematic of the treatment comparisons that have already been reported from STAMPEDE. Two additional experimental arms (G and H) were added to the trial in 2011 and 2013, respectively. Table 2 shows the reported p-values (unadjusted for multiplicity) when comparing the experimental arms with the control (arm A), as given in James et al. [2016James et al. [ , 2017, Mason et al.
[2017], Parker et al. [2018]. The dashed lines denote the four 'batches' present in the trial, where a batch corresponds to multiple hypotheses being available to be tested at the same time, as reflected in Figure 4.  Following , we apply the online testing algorithms to these observed  Table 3 shows which of the hypotheses corresponding to each experimental arm can be rejected at level α = 0.05, as well as the current significance level α 8 that would be used to test the next experimental treatment after the 7 already evaluated in the trial.
Uncorrected testing rejects the hypotheses corresponding to three experimental arms (C, E, G), and has by far the highest value of α 8 . Both SAFFRON and the BH procedure reject hypotheses C and G, and SAFFRON has a substantially higher value of α 8 than for the other online testing algorithms. ADDIS and alpha-spending only reject hypothesis G, and have similar α 8 . Finally, LORD does not reject any hypotheses and the value of α 8 is substantially lower than any of the other algorithms. For further discussion and results, see .

Algorithm
Rejections α 8 Uncorrected C, E, G 0.0500 Alpha-spending G 0.0025 BH C, G -ADDIS G 0.0016 SAFFRON C, G 0.0165 LORD -0.0002 Table 3: Hypotheses rejected and current significance level α 8 of different algorithms using the results of the STAMPEDE trial, with the ordering as in Table 2.
5 Extensions and future directions

Further extensions
Prior weights, penalty weights and decaying memory Ramdas et al. [2017] proposed a number of extensions that apply to the class of GAI++ algorithms, including LORD++. Firstly, they showed how to incorporate certain types of prior information about the different hypotheses as expressed through prior weights w t and penalty weights u t . Prior weights allow the experimenter to exploit domain knowledge about which hypotheses are more likely to be non-null. By assigning a higher prior weight w t > 1 to a hypothesis, the algorithm will have a higher chance of rejecting H t . Meanwhile, penalty weights express the different importance attached to the hypotheses being tested, with u t > 1 indicating a more impactful or important test. Importantly, both w t and u t are allowed to depend on past rejections in this framework. Ramdas et al. [2017] proposed doubly-weighted GAI++ rules that provably control the penalty-weighted FDR when using both prior and penalty weights under independence.
The second proposal dealt with problems of 'piggybacking' and 'alpha-death'. Piggybacking happens when a substantial number of rejections are made so that the online testing algorithms earn and accumulate enough alpha-wealth to reject later hypotheses at much less stringent thresholds (hence the later tests 'piggyback' on the success of earlier tests). This can lead to a spike in the FDR locally in time, even though the FDR over all time is controlled. Meanwhile, alpha-death occurs when there is a long stretch of null hypotheses, so that online testing algorithms make (almost) no rejections and lose nearly all of their alpha-wealth. Subsequently, the algorithm may have essentially no power, unless a non-null hypotheses with extremely strong signal (small p-value) is observed. Ramdas et al. [2017] proposed the decaying memory FDR (mem-FDR), which pays more attention to recent discoveries through a user-defined discount factor δ ∈ (0, 1] and thus smoothly forgets the past. They then proposed GAI++ rules that control the mem-FDR (which can also include penalty weights) under independence. In addition, they showed how to allow the algorithm to abstain from testing in order to recover from alpha-death.

Asynchronous and batched testing
In most of the literature mentioned so far, an implicit assumption is that each hypothesis test can only start when the previous test has finished (the synchronous setting). In reality, experimentation is "doubly sequential" like in Figure 1, where it is common to have hypothesis tests that overlap in time, where each test may itself be run sequentially (the asynchronous setting). One natural adjustment for asynchronous testing is to use an online FDR algorithm whenever each test finishes (that is, whichever test is the t-th one to finish, test it at level α t ). However, this would only assign α t at the end of a hypothesis test, which would not be appropriate for sequential hypothesis testing and multi-arm bandit approaches that typically require specification of the target type I error level in advance because it is an important component of their stopping rule. Hence, the testing levels must be specified at the start of a hypothesis test. The asynchronous setting also means that potentially arbitrary dependence between some p-values must be considered. Indeed, hypothesis tests that are being conducted concurrently are often likely to be dependent, since they may use the same or highly correlated data during their overlap.
To address these challenges of asynchronous testing, Zrnic et al. [2021] derived asynchronous versions of LORD and SAFFRON that output test levels α t dynamically at the beginning of the t-th test, such that, despite arbitrary local dependence and regardless of the decision times for each hypothesis, the mFDR is controlled at level α. These procedures achieve this goal both at all fixed times t, as well as certain adaptively chosen stopping times. Tian and Ramdas [2019] also showed how to derive asynchronous versions of ADDIS. Note that in order to account for the uncertainty about the tests in progress, the test levels assigned by asynchronous online procedures must be more conservative. Thus, there is a trade-off in that although asynchronous procedures take less time to perform a given number of tests, they can be less powerful than their synchronous counterparts. Zrnic et al. [2020] considered the related setting of online batched testing, where a potentially infinite number of batches of hypotheses are tested over time (see Section 4.2 for an example).
To this end, they introduced online, FDR-preserving versions of the most widely used offline algorithms, namely the BH procedure and Storey's improvement of the BH method [Storey, 2002]. These online "mini-batched" testing algorithms interpolate between online and offline methodology, thus trading off the best of both worlds. These algorithms control the FDR under independence (an algorithm valid under positive dependence was also derived), and have a higher power than the fully online testing algorithms. Further, since they consist of compositions of offline FDR algorithms, they imply FDR control over each constituent batch, and not just over the whole sequence of tests.

A Bayesian approach
Gang et al.
[2021] developed a new class of structure-adaptive sequential testing (SAST) rules for online FDR control, which instead of being based on p-values, are based upon estimates of the conditional local FDR (Clfdr; Cai and Sun [2009]), which can optimally adapt to important local structures in the data stream. This results in a novel alpha-investing framework that more precisely characterises the effects of rejection of hypotheses: rather than viewing each rejection as a gain of alpha-wealth, the Clfdr characterisation does not view all rejections as equal. Rejections with small Clfdr will lead to increased alpha-wealth whereas rejections with large Clfdr will lead to decreased alpha-wealth. SAST learns the optimal rejection thresholds adaptively and optimises the alpha-wealth allocation across different time periods. Gang et al. [2021] showed that SAST can achieve substantial power gain over existing methods, but it comes at the cost of only asymptotically controlling the FDR if the underlying Bayesian model is well-specified.
Online FWER control Tian and Ramdas [2021] focused on developing methods for online control of the FWER (see Section 1.2). Starting with the observation that only alpha-spending had previously been proposed for online FWER control, the authors first extended existing offline algorithms for FWER control, namely the Sidak method [Šidák, 1967] and the fallback procedure [Burman et al., 2009].
The online Sidak and online fallback procedures then control the FWER at all times t under independence and arbitrary dependence, respectively. However, although these online FWER control methods are guaranteed to be uniformly more powerful than alpha-spending, the improvements are usually minor in practice, except for extreme cases. Hence, Tian and Ramdas [2021] proposed the ADDIS-spending algorithm for online FWER control, which (like ADDIS does for online FDR control) benefits from adaptivity to the fraction of nulls, but also gains power by discarding conservative nulls (if they exist). ADDIS-spending controls the FWER when null p-values are uniformly conservative, and independent of each other and of the nonnulls.
Online FDX control and FDR at stopping times Xu and Ramdas [2022] focused on online control of the FDX (see Section 1.2). Prior to this work, the only online procedure that controlled the FDX was proposed by Javanmard and Montanari [2018], but this had very low power (no better than alpha-spending). Xu and Ramdas [2022] proposed the supLORD algorithm, which has a higher power and provably controls the FDX when the null p-values are conditionally superuniform. One feature of this algorithm is that it allows the user to choose the number of rejections after which FDX control begins, in exchange for more power. SupLORD is based on the GAI framework, and the authors also show how to dynamically chooses larger test levels α t when the wealth is large, allowing the algorithm to fully utilise its wealth and increases its power as a result. Finally, the authors show that supLORD also controls the mFDR and FDR at both fixed times and stopping times.
Hence, supLORD provides the first guarantee for online FDR control at stopping times (LORD, SAFFRON and ADDIS only control the mFDR at stopping times and not the FDR).

Retesting of hypotheses
One feature of online testing algorithms that has not been explicitly pointed out in the literature is the option of retesting hypotheses (i.e. using the same p-value again later in the testing sequence, when the alpha-wealth may be higher). Crucially however, the choice of whether to retest must be made without using knowledge of the p-value itself, but only that it was not rejected (e.g. that it is greater 0.01

Online control of the False Coverage Rate
Finally, Weinstein and Ramdas [2020] considered the problem of constructing confidence intervals (CIs) that are valid for online hypothesis testing. In particular, they focus on control of the false coverage rate (FCR), which is the expected ratio of number of constructed CIs that fail to cover their respective parameters to the total number of constructed CIs. In the online hypothesis testing framework they considered, at each step the investigator observes independent data that are informative about the parameter of interest θ t , and must immediately make a decision whether to report a CI for θ t or not. If a CI is reported for θ t , then the the aim is to ensure that that the CI for θ t has FCR ≤ α at all times T . For further details of the proposed algorithms and their theoretical guarantees, see Weinstein and Ramdas [2020].

Current shortcomings and future directions
Online testing for small numbers of hypotheses Online testing algorithms are most powerful in settings where there are a large number (i.e. T > 1000) of hypotheses that will eventually be tested. Thus the biggest advantage will likely be in settings such as A/B testing in large tech companies or in large-scale biological data repositories. However, while platform trials provide a framework for a trial to continue indefinitely in theory, in practice they will typically evaluate a maximum number of interventions in the low tens. Hence, there is a need for investigation of optimal online testing procedures when the maximum number of hypotheses to be tested is relatively low and when the correlation between hypotheses is known (e.g. because of a shared control arm). Separately, since many platform trials use an adaptive design, methods that explicitly combine group-sequential designs with online control of error rates would be useful (see Zehetmayer et al. [2021] for a recent proposal along these lines).

Managing incentives across sponsors or products
There are some additional challenges in using online control methods in platform trials or in the IT industry. If different sponsors (e.g. pharmaceutical companies) are supporting a platform trial, they might be reluctant to have their intervention be tested at a notably more stringent level than other sponsors: it may be difficult to reconcile the most powerful overall procedure not being acceptable to individual sponsors. Similarly, if a large IT company imposes that experiments run across various products must all be subjected to oversight in the form of a common online FDR controlling procedure that acts across products, then it may be hard to convince individual product teams that their tests must be subject to a level determined by the results of experiments by other groups.
Optimal choices of parameters for online algorithms Online error rate control under dependence One major shortcoming with almost all of the proposed online testing algorithms is their reliance on the assumption of independence of the null p-values for provable FDR control, which is unlikely to always be case in real data applications. However, in terms of online FDR control under dependence, there have only been limited proposals in the literature. Zrnic et al. [2021] showed that the LOND algorithm by Javanmard and Montanari [2015] controls the FDR under positive dependence. In the online setting, arbitrary dependence of p-values across all time is a rather pessimistic and unrealistic assumption, and thus in the asynchronous setting, Zrnic et al. [2021] introduced the concept of arbitrary local dependence and showed that online algorithms can be modified to control the FDR even with such dependence. See also Fisher [2021], who showed further results for control of the FDR under positive dependence in the minibatch setting. Finally, Zrnic et al. [2020] showed how to control the FDR in the online batched setting under positive dependence. A useful avenue of future work could explore how to construct powerful online FDR algorithms under different forms of dependence, including when the correlation structure is known or can be estimated.

Discussion
Online error rate control methodology provides a powerful and flexible framework for largescale hypothesis testing that takes into account the temporal nature of modern data analysis.
Over the past 15 years since this framework was first proposed, there have been many proposed improvements and extensions, which better reflect the nature of real-world data and expand the scope of potential applications. In particular, continuous progress has been made towards increasing the statistical power of online testing algorithms, so that they can match (and in some cases even exceed) the power of traditional offline algorithms. The issue of accounting for dependent p-values remains open, although progress has been made here too.
As the methodology becomes increasingly mature, the next natural step is to see application of online testing algorithms in practice. To this end, and as seen in Section 4, there have been a number of papers that are specifically focused on application examples, including in the context of growing data repositories [Robertson et al., 2019], platform trials  and RNAseq data [Liou et al., 2022]. Further work may be required to explore and solve practical challenges that may arise in different application settings. Finally, the provision of software and training will also be key to promoting the use of online error rate control in practice. The onlineFDR package we described earlier is a key step in that regard, but software tuned to specific applications may also be desirable.