{'fraction_non_alphanumeric': 0.04618096357226792, 'fraction_numerical': 0.013396004700352527, 'mean_word_length': 4.516104906262465, 'pattern_counts': {'":': 1, '<': 3, '<?xml version=': 0, '>': 7, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 7, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Gaussian Prepivoting for Finite Population Causal Inference

In finite population causal inference exact randomization tests can be constructed for sharp null hypotheses, i.e. hypotheses which fully impute the missing potential outcomes. Oftentimes inference is instead desired for the weak null that the sample average of the treatment effects takes on a particular value while leaving the subject-specific treatment effects unspecified. Without proper care, tests valid for sharp null hypotheses may be anti-conservative should only the weak null hold, creating the risk of misinterpretation when randomization tests are deployed in practice. We develop a general framework for unifying modes of inference for sharp and weak nulls, wherein a single procedure simultaneously delivers exact inference for sharp nulls and asymptotically valid inference for weak nulls. To do this, we employ randomization tests based upon prepivoted test statistics, wherein a test statistic is first transformed by a suitably constructed cumulative distribution function and its randomization distribution assuming the sharp null is then enumerated. For a large class of commonly employed test statistics, we show that prepivoting may be accomplished by employing the push-forward of a sample-based Gaussian measure based upon a suitably constructed covariance estimator. In essence, the approach enumerates the randomization distribution (assuming the sharp null) of a P-value for a large-sample test known to be valid under the weak null, and uses the resulting randomization distribution to perform inference. The versatility of the method is demonstrated through a host of examples, including rerandomized designs and regression-adjusted estimators in completely randomized designs.


Introduction
In finite population causal inference two distinct hypotheses of "no treatment effect" are commonly tested: Fisher's sharp null and Neyman's weak null. Fisher's sharp null of no effect refers to the null that the responses under treatment and under control are the same for all individuals in the study (Rosenbaum, 2002). The sharp null imputes the missing values of the potential outcomes for all individuals, in so doing facilitating the use of randomization tests to provide exact inference with randomization alone acting as the basis for inference (Fisher, 1935). Neyman's weak null instead specifies only that the average of the treatment effects for the individuals in the experiment equals zero, while allowing for heterogeneity in the unit-specific effects. The missing potential outcomes are no longer imputed under the weak null, such that the randomization distribution under the weak null remains unknown. Consequently, inference for the weak null has historically proceeded using asymptotically conservative analytical approximations to the limiting distribution of the treated-minus-control difference in means.

arXiv:2002.06654v4 [math.ST] 13 Jun 2021
While the exactness attained under the sharp null is appealing, randomization tests have been criticized for the seemingly restricted nature of the conclusions to which the researcher is entitled should the null be rejected (Caughey et al., 2017). While the researcher may suggest that the treatment effect is not zero for all individuals, generally one is not entitled to a statement of whether the treatment effect is positive or negative on average for the individuals in the study. To address this, a recent literature has emerged on how randomization tests may be modified to maintain asymptotic validity under the weak null hypothesis. The resulting methods provide a single testing procedure that is asymptotically valid for the weak null hypothesis, while maintaining exactness should the sharp null also be true (Ding, 2017;Loh et al., 2017;Ding and Dasgupta, 2018;Wu and Ding, 2018;Fogarty, 2020).
The existing literature attains this unified mode of inference largely on a case-by-case basis: for a given experimental design, a specially catered test statistic is constructed so that the corresponding randomization test under the sharp null maintains asymptotic validity under the weak null. In this work, we provide both general conditions under which the unification may be achieved and a general methodology for attaining it. The central idea is to leverage prepivoting, an idea introduced in Beran (1987Beran ( , 1988. For most commonly employed experimental designs and test statistics, the reference distribution generated by the prepivoted statistic under the assumption of the sharp null asymptotically stochastically dominates the true, but unknowable, randomization distribution under the weak null, yielding asymptotically conservative inference for the weak null while maintaining exactness under the sharp null hypothesis. As we demonstrate, prepivoting succeeds in many scenarios where other common resolutions such as studentization prove inadequate. At a high level, prepivoting takes a test statistic T 0 and composes it with a cumulative distribution functionF constructed from the observed data, forming the new test statistic T 1 =F (T 0 ). IfF were a consistent estimate of T 0 's limit distribution,F (T 0 ) would, through an asymptotic application of the probability integral transform, tend to a standard uniform. Under the weak null hypothesis, the true distribution function for common test statistics T 0 cannot generally be consistently estimated. Fortunately, as developed in §5 a distribution function for a random variable that asymptotically stochastically dominates T 0 may be constructed. For most common test statistics for the weak null hypothesis, under conditions outlined in §5 this dominating distribution function amounts to a suitable pushforward of a multivariate Gaussian measure constructed using a conservative covariance estimator. Using this estimated distribution function, T 1 =F (T 0 ) is instead stochastically dominated by a standard uniform in the limit. Observe that through this construction, the prepivoted test T 1 =F (T 0 ) is precisely one minus the large sample p-value for the test statistic T 0 leveraging the central limit theorem. Rather than using this p-value to reach a conclusion by comparing its value to the desired α, we instead use the reference distribution of this large-sample p-value enumerated over all possible randomizations assuming the sharp null holds. This reference distribution generally converges pointwise to the standard uniform distribution function for commonly used covariance estimators. As a result, inference is guaranteed to be asymptotically conservative under the weak null while maintaining exactness under the sharp null. The general takeaway is that rather than looking at the randomization distribution of a test statistic itself under the sharp null, one should instead enumerate the randomization distribution of one minus an asymptotically valid p-value to restore validity of Fisher randomization tests when only the weak null holds. In §2 we introduce notation for finite population causal inference and detail some standard assumptions. Section 3 defines the reference distribution assuming the truth of Fisher's sharp null and juxtaposes it with its true though unknowable randomization distribution under Neyman's weak null of no effect on average. After an overview of useful asymptotic results on completely randomized designs in §4, §5 introduces Gaussian prepivoting in the context of suitably constructed functions of treated-minus control difference in means. Section 6 provides examples of and insight into prepivoting using Gaussian measure. Section 7 extends these results to other asymptotically linear estimators including regression-adjusted estimators, while §8 provides simulation studies highlighting the benefits of Gaussian prepivoting.

Notation for finite population causal inference
While the developments in this work apply quite generally across common experimental designs and with two or more levels of the treatment, in this work we focus on completely randomized experiments and rerandomized experiments with two treatments; see the appendix for extensions to paired designs and to completely randomized designs with multi-valued treatments. Consider a collection of N individuals, where n 1 receive treatment and n 0 = N − n 1 receive the control. For the ith individual, the random variable Z i is the treatment indicator, taking the value 1 if the ith individual receives treatment and 0 otherwise. We assume that the stable unit treatment value assumption holds, such that there is no interference and that there are no hidden levels of the treatment (Rubin, 1980). The ith individual has two deterministic potential outcomes: y i (1), the d-dimensional outcome under treatment, and y i (0) the d-dimensional outcome under control. Furthermore, the ith unit has deterministic covariates x i ∈ R k . The jth coordinate of y i (z) is y ij (z), and the analogous statement holds for x ij . The random vector Z represents (Z 1 , . . . , Z N ) T ; likewise y(1) = (y 1 (1), . . . , y N (1)) T and y(0) = (y 1 (0), . . . , y N (0)) T . Under the finite population model the potential outcomes are viewed as fixed across randomizations, and the only randomness enters through Z, the treatment allocation. For a discussion of the finite population inference framework, we suggest Ding et al. (2017). The observed outcome-vector for individual i is y i (Z i ) and the collection of these is denoted y(Z). Causal inference with multiple outcomes is becoming increasingly common in modern applications ranging from drug repurposing studies to A/B tests assessing the impact of competing web page designs on various user engagement metrics. See Teixeira-Pinto et al. (2009) and Teixeira-Pinto and Mauri (2011) for concrete examples of causal inference with multiple endpoints in biomedical sciences, and see Ding et al. (2019) for a reference on the underlying mathematics of multivariate potential outcome models.
The vector of treatment effects for the ith individual is τ i = y i (1) − y i (0). The average treatment effect for the individuals in the experiment isτ = N −1 N i=1 τ i . As the two potential outcomes are not jointly observable, τ i is unknown for all individuals. Neyman's weak null of no treatment effect on average is H N :τ = 0, while Fisher's sharp null further stipulates H F : τ i = 0 (i = 1, . . . , N ) such that the treatment made no difference among any of the d outcomes measured. We implicitly define the alternative hypothesis as that which complements the null, so for H N the alternative is H A : τ = 0 and for H F the alternative is H A : ∃ i s.t. τ i = 0. Consequently, our tests are non-directional; this differs from the one-sided bounded alternatives tested by Caughey et al. (2017). Furthermore, the one-sided bounded alternatives of Caughey et al. (2017) bound each individual's treatment effect, whereas we are interested in unifying inference for both individual effects and aggregate effects.
For any matrix r ∈ R N ×d and any binary vector W with N i=1 W i = n 1 , we define the function (1 − W i )r i .
Using this notation, the observed treated-minus-control difference in means for the outcome variables isτ (y(Z), Z) and is often denoted byτ as shorthand. In general, "hats" are used to denote functions of observed quantities whose limiting properties will eventually be studied herein. Defineȳ(0) = N −1 N i=1 y i (0) andȳ(1) = N −1 N i=1 y i (1) to be the average potential outcomes for the N individuals in the study population. Likewise, we define the covariance matrices Σ y(z) = (N − 1) −1 N i=1 (y i (z) −ȳ(z))(y i (z) −ȳ(z)) T , z ∈ {0, 1}; To emphasize the distinction between functions of observed outcomes and functions of covariates, we define the function δ(x, W) with binary W such that The functionδ(x, W) is a special case ofτ (r, W). The observed difference in means for covariates isδ(x, Z), abbreviated asδ. The finite population mean of the covariates isx = N −1 N i=1 x i . The finite population covariance matrix for the covariates is Σ x , defined by simply replacing y i (z) with x i andȳ(z) withx in the definition of Σ y(z) . The finite population covariance between potential outcomes and covariates is Σ y(z)x for z = 0, 1, and the covariance between treatment effects and covariates is Σ τ x . Asymptotic arguments that follow will imagine a single sequence of finite populations of increasing size, with N → ∞. As a result, quantities such as Σ τ themselves vary as N → ∞ and should be denoted by Σ τ,N to reflect this. Generally the dependence is suppressed to reduce notational clutter; however, we do employ the notation Σ τ,∞ to denote the limiting value of Σ τ,N as N → ∞, and likewise for other finite population quantities. For more on the finite population model for causal inference, see Imbens and Rubin (2015) and Ding et al. (2017) among many.

Rerandomized designs and balance criterion
The set of all possible treatment assignments Z is denoted by Ω, and is determined by the experimental design. In completely randomized experiments, covariates are not used to inform the chosen treatment assignment and To mitigate the risk of significant covariate imbalance, Morgan and Rubin (2012) suggest instead building covariate balance into the treatment allocation process through rerandomization. The study is conducted by collecting covariate data for the study participants, determining a measure of imbalance and a threshold for deciding what imbalances are acceptable, and selecting a treatment allocation uniformly over the set of allocations satisfying the balance criterion (Li et al., 2018). Stringent balance criterion reduce the cardinality of Ω by eliminating undesirable assignments, with the hopes of improving precision as a consequence. Naturally, randomization inference must take into account that the allowable realizations of Z depend upon the condition that covariate balance is met.
A balance criterion is a Boolean-valued function φ(·), where φ( √ Nδ) = 1 is taken to mean that the treatment allocation Z which results in the particular realization ofδ under consideration satisfies appropriate covariate balance. We impose the following restriction on φ: mirror-symmetric about the origin (i.e. b ∈ M ⇔ −b ∈ M ) with non-empty interior.

Regularity conditions
We make the following assumptions about the structure of the finite populations and experimental designs as N goes to infinity. These assumptions are for the most part standard in the literature; see, for instance, Wu and Ding (2018). Assumption 1. The proportion n 1 /N limits to p ∈ (0, 1) as N → ∞. Assumption 2. All finite population means and covariances having limiting values for both the potential outcomes and the covariates. For instance, lim N →∞ȳ (z) =ȳ ∞ (z) for z ∈ {0, 1} and lim N →∞ Σ y(1) = Σ y(1),∞ .
Assumption 3. There exists some C < ∞ for which, for all z ∈ {0, 1}, all j = 1, .., d and all N , where y j (z) denotes the jth coordinate of y(z). Further, the above holds for the covariates with x ij replacing y ij (z) above for j = 1, .., k.
Assumption 3 is used to obtain finite population-inference strong laws of large numbers for mean and variance estimators. Such an assumption is made at times for mathematical convenience to simplify the analysis of certain random distributions and may hold under weaker assumptions. Assumption 3 is commonplace in the literature on finite population causal inference; see, for instance, Wu and Ding (2018); Lin (2013a); Freedman (2008a,b).

A technical note on the convergence of random measures
A random sequence of probability measuresμ N on S converges weakly in probability to a deterministic probability measure µ if S f dμ n p − → S f dµ for all continuous bounded functions f : S → R (Dümbgen and Del Conte-Zerial, 2013, Section 2). Aspects of the Portmanteau Theorem (van der Vaart and Wellner, 1996, Theorem 1.3.4) extend to weak convergence in probability of random measures (Dümbgen and Del Conte-Zerial, 2013;Crauel, 2002). Most importantly for our purposes is that if {F N } are random cumulative distribution functions and F is a fixed cumulative distribution function, then their associated measures converge weakly in probability if and only ifF N (t) p − → F (t) for all t which are continuity points of F ; we take this as the definition of weak convergence in probability for random cumulative distribution functions.

Randomization distributions
Consider a scalar test statistic T (y(Z), Z), a function of the observed responses and the treatment assignment received. The randomization distribution for the test statistic T is (1) R T is the true cumulative distribution function of T (y(Z), Z) with respect to the randomness in the treatment allocation Z, distributed uniformly over Ω. If we had access to R T under the null hypothesis in question, we could make direct use of it to provide inference that is exact in finite samples, proceeding without dependence on asymptotics. Under Fisher's sharp null hypothesis, R T is specified by the observed outcomes as y(Z) = y(w) for any w ∈ Ω. Unfortunately, the distribution is generally unknown under the weak null, as the weak null merely constrains the missing potential outcomes without determining them.

Randomizaton tests assuming the sharp null
In practice an experimenter draws a single realization of Z, in so doing only revealing the values of the potential outcomes corresponding to the observed assignment. Suppose that regardless of whether or not Fisher's sharp null hypothesis actually holds, the researcher considers use of the randomization distribution to which she or he would be entitled if the sharp null were true. This reference distribution takes the form While R T =P T under the sharp null, under the weak nullP T is a random distribution function as it varies with Z. Inference usingP T proceeds as though y(Z) would have been the observed response for any w ∈ Ω. As the true response y(w) under assignment w need not align with y(Z),P T does not actually reflect the true randomization distribution under the weak null. This gives rise to potentially anti-conservative inference shouldP T be used to test the weak null hypothesis.
For α ∈ (0, 1) define the Fisher randomization test of nominal level α by Under the sharp null, E{ϕ T (α)} ≤ α for any sample size asP T = R T . Throughout this paper, we examine the extent to which certain choices of test statistics entitle us to asymptotic Type I error control at α when ϕ T (α) is used to conduct inference but only the weak null holds. For a given test statistic T , we will often proceed by juxtaposing its true limiting behavior under the randomization distribution R T with the limiting behavior ofP T , the randomization distribution if we (incorrectly) assumed that the sharp null held.

Towards a unified mode of inference
Suppose that for a test statistic T (y(Z), Z) based upon the observed outcomes y(Z) and the treatment allocation Z, (a)P T converges weakly in probability to a fixed distribution P T,∞ as N → ∞; and (b) R T converges pointwise to a fixed distribution R T,∞ at all continuity points of R T,∞ . Formally, is asymptotically sharp-dominant, then inference based upon the reference distributionP T will be asymptotically conservative even if only H N holds (Wu and Ding, 2018, Proposition 4), satisfying lim sup E{ϕ T (α)} ≤ α as N → ∞ while maintaining exactness should the sharp null be true.
Many common test statistics are not asymptotically sharp-dominant over all elements of the weak null. For instance, with univariate potential outcomes and under a completely randomized design with imbalanced treated and control groups, the absolute difference in means T (y(Z), Z) = √ N |τ | is not generally asymptotically sharp-dominant as there exist sequences of potential outcomes satisfying the weak null such that lim inf E{ϕ T (α)} > α; see Ding (2017), Wu andDing (2018, Cor. 3), or Loh et al. (2017) for details. For this test statistic, simply studentizing by the usual standard error estimator ensures sharp dominance. However, studentization fails to generalize to other more complicated test statistics and complex experimental designs. Significant efforts have recovered appropriate studentization techniques for some test statistics, but each test statistic requires its own separate analysis (Wu and Ding, 2018). For some experimental designs, studentizing the difference in means is not sufficient to regain asymptotically valid inference even in the univariate case; we explore this topic in §5.2 and §8.1 in the context of rerandomization. In §5, we present a general method called Gaussian prepivoting which both recovers studentization when it alone would be sufficient, but also yields asymptotic sharp-dominance in circumstances where studentization would be insufficient. Before describing the method, we recall a few important results on the difference in means in completely randomized designs which underpin the success of Gaussian prepivoting.
4 Useful results for the difference-in-means in completely randomized designs 4.1 Asymptotic normality and conservative covariance estimation for the randomization distribution Consider the distribution of √ N (τ −τ ,δ) T in a completely randomized design. Under Assumptions 1, 2, and 3, a finite population central limit theorem applies , and √ N (τ −τ ,δ) T converges in distribution to a mean-zero multivariate Gaussian with covariance matrix V of the form While V δδ and V τ δ can be consistently estimated, V τ τ cannot be in the presence of effect heterogeneity due to its dependence on Σ τ , the covariance of the unobserved treatment effects. Consequently, one cannot consistently estimate the probability that √ N (τ −τ ) falls within a given region B. While consistent variance estimates are not available, there are several covariance estimatorsV τ τ (y(Z), Z) for V τ τ satisfyingV τ τ − V τ τ p → ∆ for some ∆ 0 under Assumptions 1 -3 in completely randomized designs. These estimators typically have the property that Σ τ τ = 0 implies consistency, rather than asymptotic conservativeness; see Ding et al. (2019) for more details. So while the matrix V cannot generally be consistently estimated, one can construct an estimate converging in probability to a matrixV of the formV As an illustration, consider the conventional covariance estimator for the difference in means in a two-sample problem, V τ τ = N Σ y(1) /n 1 +Σ y(0) /n 0 witĥ T and the analogous forΣ y(0) . Under both completely randomized experiments and rerandomized experiments with balance criterion satisfying Condition 1, this estimator satisfiesV τ τ − V τ τ p → Σ τ,∞ 0 under Assumptions 1 -3.

Limiting behavior of the reference distribution
Suppose we have a completely randomized design, and consider the random variable where Z and W are independent, identically distributed, and drawn uniformly from Ω andỹ i (Z i ) = y i (Z i ) − Z iτ , such thatỹ(Z) = y(Z) − Zτ T . Proposition 1. Subject to Assumptions 1, 2, and 3, under a completely randomized design the distribution of { √ Nτ (ỹ(Z), W), √ Nδ(x, W)} T | Z converges weakly in probability to a multivariate Gaussian measure, with mean zero and covarianceṼ of the form The proof of this statement is contained within the proof of Theorem 1 in Wu and Ding (2018) and is omitted. Under the sharp null,Ṽ = V as y i (1) = y i (0) for all i. Under the weak null however, whileṼ δδ = V δδ generallyṼ τ τ = V τ τ andṼ δτ = V δτ . The divergence between V andṼ can render randomization tests for the weak null hypothesis anti-conservative; examples are given in §5.2. We now describe how prepivoting may be used to guarantee asymptotic correctness when inference for the weak null hypothesis is conducted using a reference distribution generated under the sharp null.

Prepivoting with an estimated pushforward measure
Consider functions f η : R d → R subject to the following requirement: is jointly continuous in η and t.
We begin with statistics for H N of the form whereξ =ξ(ỹ(Z), Z) satisfies the following condition for some set Ξ: Condition 3. With W, Z independent and each uniformly distributed over Ω, for some ξ,ξ ∈ Ξ.
As will be shown in §5.2, several commonly encountered statistics for Neyman's null are of this form. A detailed discussion of Condition 2 is included in the appendix. Suppose further that one employs a covariance estimator V (ỹ(Z), Z) with the following property: Condition 4. With W, Z independent, both uniformly distributed over Ω, and for some ∆ 0, ∆ ∈ R d×d , As a concrete example satisfying Conditions 2-4, suppose that f η (t) = t T η −1 t T andξ(y(z), w) =V N eyman (y(z), w) withV N eyman denoting the usual Neyman variance estimator of Neyman (1990); numerous other examples are included in Section 5.2. Observe that when assuming the weak null for the purpose of testing,ỹ(Z) = y(Z) andτ −τ =τ . Gaussian prepivoting transforms the test statistic T (y(z), w) = fξ( √ Nτ (y(z), w)) into a new statistic of the form where γ (p) µ,Σ (B) is the p-dimensional Gaussian measure of a set B with mean parameter µ and covariance Σ, i.e.
For (A, B) T jointly multivariate normal with mean zero and covarianceV , A ∈ R d , B ∈ R k , G(y(Z), Z) represents the fξ-pushforward measure of A | φ(B) = 1 evaluated on the set (−∞, T (y(Z), Z)]. That is, G(y(Z), Z) treats fξ andV as fixed and computes the conditional probability that fξ(A) falls at or below the observed value for T (y(Z), Z) = fξ( √ Nτ ) given that ϕ(B) = 1. From the perspective of hypothesis testing, G(y(Z), Z) is 1 minus the large-sample p-value for T (y(Z), Z) leveraging the finite population central limit theorem and the estimated covariancê V .
We now describe how to use the prepivoted statistic G(y(Z), Z) to provide a single procedure that is both exact for H F and asymptotically conservative for H N . In order to provide a precise implementation of this, we give detailed pseudocode in Algorithm 1 and provide example code through the appendix. First, we compute the prepivoted test statistic G(y(Z), Z) given the observed data; this proceeds according to Equation 5 and is Step 1 of Algorithm 1. Next, we construct the reference distributionP G (·), the construction of which requires imputing counterfactual outcomes as if Fisher's sharp null held; this is Step 2 of Algorithm 1. Finally, the p-value for testing H N is computed and we reject the null when this lies below or at the nominal level α.
Algorithm 1: Inference for the weak null through Gaussian prepivoting Input: An observed treatment allocation z, with observed responses y(z), test statistic T (y(z), z) = fξ( √ Nτ obs ) and covariance estimatorV (y(z), z) Result: The p-value for the Gaussian prepivoted test statistic Step 1: The observed prepivoted statistic Compute fξ (y(z),z) (·);V (y(z), z). Compute Step 2: The reference distributionP G for w ∈ Ω do Compute fξ (y(z),w) (·);V (y(z), w). Compute Observe that 1 − g z defined within Algorithm 1 is the usual large-sample p-value based upon a Gaussian approximation and using the covariance estimatorV . The large-sample test compares 1 − g z to α, the desired Type I error rate, and rejects if 1 − g z ≤ α ⇔ g z ≥ 1 − α. The Gaussian prepivoted randomization test instead rejects if g z ≥P −1 G (1 − α). The following Theorem, in concert with Lemma 11.2.1 of Lehmann and Romano (2005), show under our assumptionŝ such that the prepivoted randomization test is asymptotically equivalent to large sample test under the weak null. By usingP −1 G (1 − α) instead of 1 − α, exactness under the sharp null is preserved.
Theorem 1. Suppose we have either a completely randomized design or a rerandomized design with balance criterion φ satisfying Condition 1. Suppose T (y(Z), Z) is of the form (4) for some f η andξ satisfying Conditions 2 and 3. Suppose further that we employ a covariance estimatorV satisfying Condition 4 when forming the prepivoted test statistic G(y(Z), Z). Then, under H N :τ = 0 and under Assumptions 1-3, G(y(Z), Z) converges in distribution to a random variableŨ taking values in [0, 1] satisfying for all t ∈ [0, 1]. Furthermore, the distributionP G (t) satisfiesP G (t) p → t for all t ∈ [0, 1]. Corollary 1. Under the conditions of Theorem 1; the prepivoted test statistic G(y(Z), Z) is asymptotically sharp dominant regardless of whether the base statistic T (y(Z), Z) was. Consequently, p-values derived underP G via Algorithm 1 are guaranteed to be exact under H F and asymptotically conservative under just H N .
Theorem 1 states that under the weak null, G(y(Z), Z) converges in distribution to a random variable which is stochastically dominated by the standard uniform. Meanwhile, the reference distribution for G(y(Z), Z) constructed assuming (incorrectly) that the sharp null holds converges pointwise to the distribution function of a standard uniform. As a result, the randomization distribution for G(y(Z), Z) is asymptotically sharp-dominant: the reference distribution generated in this manner yields asymptotically conservative inference for the weak null hypothesis, while maintaining exactness should the sharp null also hold. By exploiting the duality between hypothesis testing and confidence sets Theorem 1 provides the basis for generating exact and asymptotically conservative confidence sets for treatment effect; this is explored in the appendix. Remark 1. Consider the function the estimated distribution function for fξ( √ Nτ ) | φ( √ Nδ) = 1 based upon a finite population central limit theorem.
In special cases, the functionF (t) may have a known closed form. This is true of the test statistics which are sharpdominated by a χ 2 d distribution considered in Wu and Ding (2018), for example. Should this not be the case, one can approximateF (·) by way of Monte-Carlo approximation, replacing the measures γ 0,V and γ 0,V δδ with estimates based upon a B draws from a multivariate normal with mean 0 and covarianceV when enumerating the reference distribution. Importantly, such Monte-Carlo approximation does not corrupt finite-sample exactness under Fisher's sharp null.

Examples of Gaussian prepivoting
Through a series of examples, we now provide illustrations of the transformations achieved by (5). As will be demonstrated, the form recovers several randomization tests previously known to be valid for weak null hypotheses in the literature while providing a basis for new randomization tests for weak nulls using other test statistics. These examples serve four objectives: (i) unify previous ad hoc solutions under the framework of Gaussian prepivoting; (ii) provide an alternative approach to already valid procedures; (iii) highlight that prepivoting can succeed even where studentization fails; and (iv) extend randomization inference for H F and H N to new experimental designs.
Example 1 (Absolute difference in means). Let √ Nτ be univariate, consider a completely randomized design with no rerandomization, and let T DiM (y(Z), Z) = √ N |τ |, with f η (t) = |t| andξ = 1. The randomization distribution for T DiM (y(Z), Z) is not asymptotically sharp-dominant, such that employing the reference distribution assuming that the sharp null holds may lead to anti-conservative inference. The conventional fix is to studentize √ N |τ | using a variance estimator estimator satisfying Condition 4, forming instead T Stu (y(Z), Z) = √ N |τ |/ V τ τ (Loh et al., 2017).
As φ(·) = 1 deterministically in a completely randomized design, Gaussian prepivoting via (5) yields the test statistic where Φ(·) is the standard normal distribution function. For any Z, the pairs {G DiM (y(Z), w), T DiM (y(Z), w)} have rank correlation equal to 1 when computed for all w ∈ Ω. As a result, the reference distribution using the studentized difference in means assuming the sharp null will furnish identical p-values to those attained using Gaussian prepivoting. That is, in the univariate case Gaussian prepivoting is equivalent to studentization for completely randomized designs. This highlights objectives (i) and (ii).
Example 2 (Multivariate studentization). Let √ Nτ now be multivariate and suppose we have a completely randomized design. Wu and Ding (2018) suggest the test statistic withV τ τ = N n1Σ y(1) + N n0Σ y(0) . For this test statistic, f η (t) = t T η −1 t andξ =V τ τ . Wu and Ding (2018) show that under our assumptions, under the weak null this test statistic converges in distribution to d i=1 w i ζ 2 i where w i ∈ [0, 1] are weights and ζ 1 , . . . , ζ d iid ∼ N (0, 1) while the reference distribution of T χ 2 (y(Z), Z) attained assuming that the sharp null holds converges weakly in probability to the χ 2 d -distribution. As a result, T χ 2 (y(Z), Z) is asymptotically sharp-dominant, and its reference distribution assuming the sharp null may be used for inference for the weak null hypothesis. Here, Gaussian prepivoting produces is the distribution function of a χ 2 d random variable. For any Z, the pairs {G χ 2 (y(Z), w), T χ 2 (y(Z), w)} have rank correlation equal to 1 when computed for all w ∈ Ω, such that Gaussian prepivoting yields equivalent inference to that attained using the distribution of T χ 2 (y(Z), Z) under the sharp null. This demonstrates objective (ii).
Suppose instead that, erroneously, a practitioner proceeded with the more typical form of Hotelling's T -squared statistic employing a pooled covariance estimator, For this test statistic, f η (t) = t T η −1 t as before, butξ =V P ool . In this case, T P ool (y(Z), Z) is not asymptotically sharp-dominant, such that the reference distribution using this statistic and assuming the sharp null may yield invalid inference. Gaussian prepivoting returns the test statistic , Z) continues to use the Gaussian measure computed with the covariance matrixV τ τ in forming the suitable transformation, despite the fact that the pooled covariance matrix is used in forming T P ool (y(Z), Z). For fixed Z, G P ool (y(Z), w) generally will not have perfect rank correlation with T P ool (y(Z), w) when computed over w ∈ Ω, such that the two randomization tests assuming the sharp null no longer furnish identical p-values. This divergence is necessary: while T P ool (y(Z), Z) is not asymptotically sharp-dominant, Theorem 1 asserts that G P ool (y(Z), Z) is, such that the reference distribution for G P ool (y(Z), Z) assuming the sharp null yields asymptotically conservative inference for the weak null. Gaussian prepivoting can thus restore asymptotic validity to a test statistic employing improper studentization, illustrating objective (iii). Example 3 (Max absolute t-statistic). Consider again multivariate √ Nτ and a completely randomized design, and consider the test statistic is not asymptotically sharp-dominant under the weak null: the reference distribution generated under the sharp null depends upon the correlation matrix corresponding toṼ , while the true randomization distribution is governed by the correlations encoded within V . The Gaussian prepivoted correction takes the form which composes T |max| (y(Z), Z) with the distribution function for max |A j |/ V τ τ,jj , j = 1, .., d, when A is multivariate Gaussian with mean zero and covarianceV τ τ . Gaussian prepivoting rectifies the insufficiency of the studentization in T |max| , thereby providing an example of objective (iii).
Example 4 (Rerandomization). Let √ Nτ be univariate and suppose we now consider a rerandomized design with balance criterion φ satisfying Condition 1. Consider the absolute difference in means, fξ( Gaussian prepivoting yields the test statistic For completely randomized designs with φ(·) = 1 deterministically, Gaussian prepivoting is equivalent to studentizing as described in Example 1. In general rerandomized designs however, observe that the transformation depends upon the particular form of the balance criterion φ, and that the reference distribution will depend upon the relationship between the potential outcomes and the covariates used in the balance criterion. As a result, it will generally not be the case that the reference distribution of G Re (y(Z), Z) under the sharp null yields equivalent inference to that attained using √ N |τ |/ V τ τ . This suggests that in rerandomized designs, studentization alone is insufficient for attaining an asymptotically sharp-dominant test statistic. In §8.1, we show this through an example in the case of Mahalanobis rerandomization. Lemmas A15 and A16 of Li et al. (2018) show that under our conditions, probability limits for estimatorsV derived under complete randomization are generally preserved under rerandomized designs. Once again, Theorem 1 ensures that G Re (y(Z), Z) will be asymptotically sharp-dominant, such that the randomization distribution assuming the sharp null may be employed for inference for the weak null. The development of a finite sample exact method for testing H F which is asymptotically valid for testing H N in rerandomized designs is novel, but its construction is extremely simple within the framework of Gaussian prepivoting; this highlights Gaussian prepivoting's portability to designs outside of just completely randomized experiments. In the appendix we provide two more examples of this portability: one for matched-pair designs and one for experiments with any finite number of treatment arms. This highlights objective (iv).
For the interested reader, in the appendix we include this same collection of examples written directly in the form of Gaussian integrals, and we include verification of Conditions 2-4.
6 Gaussian comparison, stochastic dominance, and the probability integral transform

Gaussian comparison and Anderson's Theorem
We now highlight the essential technical ingredients underpinning the success of Gaussian prepivoting. Consider two mean-zero multivariate Gaussian vectors (A 1 , B 1 ) T and (A 2 , B 2 ) T , with covariances (1) aa 0 and Λ bb 0; the inequalities are stated with respect to the Loewner partial order on positive semidefinite matrices. Let the dimensions of A j and B j be d and k respectively for j = 1, 2. Compare the tail probabilities for where φ and f satisfy Conditions 1 and Condition 2 respectively. The following result is a straightforward corollary of Anderson's (1955) theorem for multivariate Gaussians; see also Theorem 4.2.5 of Tong (1990). Lemma 1. Under the stated conditions, for any scalar v, The result follows immediately from Anderson's theorem after noting that the set is convex and mirror-symmetric for any v. This can be seen through our assumption that f (·) is quasiconvex and mirror-symmetric, such that its sublevel sets are convex and mirror symmetric. We further have that P(φ(B 1 ) = 1) = P(φ(B 2 ) = 1) > 0 given the structure of the covariance matrices M 1 and M 2 and Condition 1, completing the proof.

Stochastic dominance and the probability integral transform
For two real valued random variables S and T , S (first order) stochastically dominates T if F S (a) ≤ F T (a) for all a ∈ R, where F S and F T are the distribution functions of S and T respectively.
Suppose now that S and T are continuous and that S stochastically dominates T . By the probability integral transform, the distribution of F T (T ) would be standard uniform. The following proposition considers transforming the random variable T not by its own distribution function, but rather by the distribution function of S, its stochastically dominating random variable. Lemma 2. Suppose that S, T are continuous random variables and that S stochastically dominates T . Then, F S (T ) is stochastically dominated by a standard uniform random variable.
In the setup of §6.1, under Conditions 1 and 2 we have by Proposition 1 that would yield a random variable that is stochastically dominated by a standard uniform.

A proof sketch for Theorem 1
While a formal proof of Theorem 1 is deferred to the appendix, here we provide an informal sketch in light of Lemmas 1 and 2. Under Assumptions 1 -3 and Condition 1, satisfying Condition 2 for all η ∈ Ξ, someξ satisfying Condition 3, and with a balance criterion φ satisfying Condition 1. By Condition 3 and the assumption of the weak null, we have thatξ(y(Z), Z) converges in probability to ξ. Therefore, under the weak null, by Lemma 1 the limiting distribution of T (y(Z), Z) would be stochastically dominated by that of transforms T (y(Z), Z) by the distribution function of a random variable which stochastically dominates its limiting distribution. By Lemma 2 and the continuous mapping theorem, asymptoticallyḠ(y(Z), Z) is stochastically dominated by a standard uniform. By Condition 4, the covariance estimatorV used in forming G(y(Z), Z) has a probability limit of the required form for stochastic dominance. Therefore, another application of the continuous mapping theorem yields that G(y(Z), Z) −Ḡ(y(Z), Z) = o p (1), such that by Slutsky's Theorem G(y(Z), Z) is itself stochastically dominated by a standard uniform.
Meanwhile, Proposition 1 and Condition 1 yield that under the weak null the distribution of converges in probability toṼ itself. Further, by Condition 3ξ(y(Z), W) converges in probability toξ. Applying the continuous mapping theorem and Slutsky's Theorem for randomization distributions (Chung and Romano, 2016, Lemmas A5-A6), one sees that Gaussian prepivoting furnishes a transformation that amounts to, asymptotically, an application of the probability integral transform. As a result,P G (t) converges in probability to t, the distribution function of the standard uniform, for all t ∈ [0, 1].

Extensions to asymptotically linear estimators
Theorem 1 may be extended to estimators other than the difference in means. Consider an estimatorτ (y(Z), Z) such that 0)) = 0 along with Assumptions 2 and 3. Suppose further thatτ (ỹ(Z), W), W independent from Z and drawn uniformly from Ω, satisfies (0)) = 0 along with Assumptions 2 and 3. Observe that the difference in means estimator satisfies these conditions with x , Σ τrx be the analogues of Σ y(z) , Σ τ , Σ y(z)x and Σ τ x for z ∈ {0, 1}, and let the same hold with r replaced byr. Define V (r) andṼ (r) as the analogues of V andṼ , computed now based upon r(z) andr(z) instead of y(z) andỹ(z) for z ∈ {0, 1}.
Consider a test statistic for the weak null of the formT (y(Z), Z) = fξ( √ Nτ ) for some f η satisfying Condition 2 and ξ satisfying Condition 3, and suppose that there exists a covariance estimatorV satisfying Condition 4 with V andṼ replaced by V (r) andṼ (r) . The Gaussian prepivoted test statistic is Theorem 2. Suppose that Neyman's null, H N :τ = 0, holds. Then, under the described restrictions onT (y(Z), Z) andV and under Assumption 1 and with Assumptions 2 and 3 applied to In the appendix, we illustrate that the regression-adjusted average treatment effect estimator and its corresponding estimated variance presented in Lin (2013a) can be viewed in this form. As a result, Theorem 2 provides justification for the use of the prepivoted randomization distribution of a regression-adjusted estimator.
8 Simulation studies

Studentization and prepivoting in rerandomized designs
In the bth of B iterations, we draw, for i = 1, ..., N , covariates iid as Given these covariates, we draw r i (0) and r i (1) as (1), and E(λ) representing an exponential distribution with rate λ.
We form the potential outcomes under treatment and control in two distinct ways, one in which the sharp null holds and one in which only the weak null holds: Of the N individuals, n 1 = 0.2N receive the treatment and n 0 = 0.8N receive the control. We use a Mahalanobisbased rerandomized design, with criterion φ( This balance criterion reduces the cardinality of Ω by roughly 80% relative to a completely randomized design. For each b, we draw a single Z ∈ Ω, and proceed with inference using the reference distribution of the following test statistics under the incorrect assumption that the sharp null holds: 1. Absolute difference in means, unstudentized 2. Absolute difference in means, studentized 3. Gaussian prepivoting the absolute difference in means, studentized Table 1: Inference after rerandomization. The rows describe the simulation settings, which vary between the sharp and weak nulls holding and between small and large sample sizes. The first three columns represent the performance of randomization tests assuming the sharp null hypothesis and using the unstudentized absolute difference in means, absolute studentized difference in means, and Gaussian prepivoted absolute difference in means respectively to perform inference. The last column is a large-sample test which is asymptotically valid for the weak null, based upon Li et al. (2018). The desired Type I error rate in all settings is α = 0.05.

Randomization Test
Large The true reference distributions assuming the sharp null are replaced by Monte-Carlo estimates with 1000 draws from Ω for each b, and the desired Type I error rate is α = 0.05. We also perform inference using the large-sample reference distribution for the absolute studentized difference in means in a rerandomized design; see Li et al. (2018) for more details. As a covariance estimatorV , we use the conventional unpooled covariance estimator for ( √ Nτ , √ Nδ) T in a two-sample design. For the generative models reflecting the sharp and weak nulls, we proceed with both N = 50 and N = 1000 to compare performance in small and large sample regimes. For each N , we conduct B = 5000 simulations.  Table 1 contains the results of the simulation study. Under the sharp null with N = 50, we see the benefits of using a randomization test: the randomization tests based upon the unstudentized, studentized, and prepivoted absolute difference in means all resulted in a Type I error rate of 0.05 (up to noise from the Monte-Carlo simulation) as desired. Contrast this with the large-sample test, which had an estimated Type I error rate of 0.110 under the sharp null hypothesis. Figure 1 explains the deficiency of the large-sample test by comparing the true distribution for the large-sample p-values to the standard uniform distribution. As is seen, at N = 50 small p-values are more likely to occur than what the standard uniform would predict at any point t ∈ [0, 1], resulting in inflated Type I error rates. By N = 1000, the asymptotic approximation performs much better, as the true distribution of p-values lies on top of the standard uniform. Gaussian prepivoting uses 1 minus these large-sample p-values as the test statistic whose randomization distribution is enumerated, such that the solid line in Figure 1 reflects 1 minus the randomization distribution of the Gaussian To yield valid randomization tests under the weak null, the solid line needs to lie above the dotted line, such that the solid line attributes less mass in the right tail than the dotted line does prepivoted test statistic. As Gaussian prepivoting uses a randomization test under the sharp null, the solid line also reflects the reference distribution employed for performing inference. That these coincide is a consequence of the sharp null holding, such that the randomization tests are exact tests for any sample size.
Under the weak null, we see in Table 1 that even at N = 1000, the unstudentized and studentized randomization tests erroneously assuming the sharp null have inflated Type I error rates. This pattern will persist even asymptotically, as in this simulation setup these test statistics are not asymptotically sharp-dominant. This may come as a surprise, as in completely randomized designs studentizing does furnish asymptotic sharp dominance. As evidenced here, the impact of covariates on the limiting distribution in rerandomized experiments invalidates studentization as a mechanism for attaining asymptotic sharp dominance. Figure 2 illustrates this in the case of the studentized test statistic. We see in the top panel that the true distribution function for the studentized test statistic lies below that of the reference distribution assuming the sharp null, such that the right-tail probabilities are larger for the true randomization distribution than they are for the reference distribution. This yields anti-conservative inference. We see in the bottom panel of Figure  2 that through use of Gaussian prepivoting, asymptotic conservativeness has been restored: the true randomization distribution of the prepivoted test statistic is stochastically dominated by the reference distribution assuming the sharp null, as predicted by Theorem 1. We further see that the cumulative distribution assuming the sharp null is converging to the distribution function of the standard uniform (a straight line between 0 and 1), again reflecting Theorem 1. Table 1 further shows that the Gaussian prepivoted test and the large-sample test have very similar rejection rates at N = 1000, reflecting the asymptotic equivalence of the two methods under the weak null.

A comparison of multivariate tests
independent from one another and iid from mean zero equicorrelated multivariate normals of dimension k = 25 with marginal variances one. The correlation coefficients governing r i (1) and r i (0) are 0 and 0.95 respectively. We will have two simulation settings, one each for the sharp and weak null: Weak Null: y i (1) = r i (1); y i (0) = r i (0) +r(1) −r(0). Table 2: Inference in completely randomized designs with multiple outcomes. The rows describe the simulation settings, which vary between the sharp and weak nulls holding and between small and large sample sizes. There are three sets of columns, one corresponding to each of the three test statistics under consideration. For each set of columns, the column labeled "FRT" represents the Fisher randomization test using that test statistic. The column labeled "Pre." instead reflects the Fisher randomization test after applying Gaussian prepivoting to the original test statistic. The last column, labeled "LS," is a large-sample test which is asymptotically valid for the weak null. The desired Type I error rate in all settings is α = 0.05.

Hotelling, Unpooled
Hotelling, Pooled Max t-stat In both settings, n 1 = 0.2N individuals receive the treatment and n 0 = 0.8N receive the control. We consider a completely randomized design, and proceed with inference using the reference distribution of the following test statistics under the (erroneous) assumption that the sharp null holds: 1. Hotelling's T -squared, unpooled covariance 2. Hotelling's T -squared, pooled covariance 3. Max absolute t-statistic, unpooled standard error For each candidate test, we proceed with the randomization distribution both of the untransformed test statistic and the Gaussian prepivoted test statistic. These tests are conducted using Monte-carlo simulation to generate the reference distributions, with 1000 draws from Ω for each iteration b. In addition to the two types of randomization tests, we also compute a large-sample p-value for each test which is asymptotically valid under the weak null hypothesis. As a covariance estimatorV , we use the conventional unpooled covariance estimator for

√
Nτ . For each test, we seek to maintain the Type I error rate at or below α = 0.05. For the generative models reflecting the sharp and weak nulls we proceed with both N = 300 and N = 5000 to compare performance as N increases. For each N , we conduct B = 5000 simulations. Table 2 gives the estimated Type I error rates for the candidate tests. We first note the poor performance of the largesample tests under both the sharp and weak null with N = 300. For instance, the large-sample p-values constructed using the unpooled, Hotelling procedure are attained using a χ 2 25 distribution and have estimated Type I error rates of 0.321 under the sharp null for N = 300, and of 0.270 under the weak null for N = 300 despite the desired control at α = 0.05. By N = 5000, the large-sample tests all have estimated Type I error rates approaching the nominal level under the sharp null, and below the nominal level under the weak null.
Naturally, all randomization tests attain (up to Monte-Carlo error) the desired Type I error rate under the sharp null at both N = 300 and N = 5000, highlighting the appeal of the randomization tests. Under the weak null, we see that the randomization test based upon the Hotelling T -statistic with a pooled covariance fails to control the Type I error rate even at N = 5000, reflecting that the test statistic is not asymptotically sharp-dominant. While the randomization test based on the max t-statistic controls the Type I error rate in these simulations, this is not guaranteed in general: in the appendix we conduct this simulation at α = 0.25, where anti-conservativeness of the max t-statistic arises. For both of these test statistics, applying Gaussian prepivoting restores guaranteed asymptotic conservativeness and results in test statistics whose performance closely aligns with the large-sample tests, a reflection of Theorem 1. For the test based upon Hotelling's T statistic with an unpooled covariance estimator, observe that the Type I error rates for the randomization tests with and without Gaussian prepivoting are identical in all four scenarios tested. As discussed in Example 2 of §5.2, this is because Gaussian prepivoting is unnecessary for this particular test statistic: Hotelling's T statistic with an unpooled covariance estimator is already asymptotically sharp-dominant as proven in Wu and Ding (2018). Applying Gaussian prepivoting recovers an equivalent randomization test, furnishing identical p-values for any observed outcomes y(Z) for completely randomized designs.
In the appendix we provide a theoretical analysis of the statistical power of Gaussian prepivoting and include simulations to demonstrate the power in practice. We also provide analysis of real-world data from the Student Achievement and Retention experiment of Angrist et al. (2009). The restrictions on the function f η outlined in Condition 2 require a quasi-convex, continuous function that is mirrorsymmetric about the origin. This restriction results in convex, mirror-symmetric sublevel sets for f η and facilitates the application of Anderson's theorem, such that dominance in the Loewner order on covariance matrices translates to the stochastic dominance under the weak null. While the restrictions on f η are sensible with two-sided alternatives, they preclude testing directional alternatives because of the mirror symmetry condition. For instance, suppose one wanted to test the null hypothesisτ i ≤ 0 for all i = 1, .., d versus the alternative that for at least one i (i = 1, .., d),τ i > 0. In the univariate case, choosing T (y(Z), Z) =τ /V 1/2 τ τ does not provide a valid one-sided test for all α. That said, it does provide a valid test for α ≤ 0.5, such that for any reasonable value for α to be deployed in practice a one-sided test is possible.
Suppose we have multivariate potential outcomes and consider the test statistic T max (y(Z), Z) = max 1≤i≤dτi /V 1/2 τ τ,ii , withV τ τ satisfying Condition 4. Consider the Gaussian prepivoted test statistic G max (y(Z), Z). The following is, to the best of our knowledge, an open question: is it the case that, for any α ≤ 0.5, G max is asymptotically sharp-dominant, in that lim sup E{ϕ Gmax (α)} ≤ α? Under the assumptions imposed in this work, the answer would be true should the following conjecture on Gaussian comparisons hold: Conjecture 1. Let X = (X 1 , ..., X d ), and Y = (Y 1 , ..., Y d ) be d-dimensional multivariate Gaussian vectors, with a common mean µ = (µ 1 , ..., µ d ) but distinct covariances Σ X and Σ Y , with ij entries σ X ij and σ Y ij , respectively. Let The conjecture is true in the univariate case. Under the assumptions of this conjecture, the Sudakov-Fernique inequality (Adler and Taylor, 2009, Theorem 2.2.5) asserts that E{max 1≤i≤d X i } ≤ E{max 1≤i≤d Y i }. Should we further assume σ X ii = σ Y ii , the result holds for all points c through Slepian's lemma (Slepian, 1962;Tong, 1990, Theorem 5.1.7). Unfortunately, a refined result about tail probabilities above the median does not appear to be available in the literature under the conditions outlined in the conjecture. A potential path forward may be a modification of the soft-max proof of the Sudakov-Fernique inequality found in Chatterjee (2005).

Summary
In this work, we present a general framework for designing randomization tests that are both exact for Fisher's sharp null and are asymptotically conservative for Neyman's weak null in completely randomized experiments and rerandomized designs. Loosely stated, the approach may be summarized as follows: if one has access to a large-sample test that is asymptotically conservative under Neyman's weak null, then a Fisher randomization test using the p-value produced by that large-sample test will maintain asymptotic correctness under the weak null while additionally restoring exactness should the sharp null be true. As the Fisher randomization distribution of these p-values converges weakly in probability to a uniform, the resulting randomization test assuming the sharp will have the same large-sample performance under the weak null as large-sample test itself, and will further have the same asymptotic power under local alternatives as the large-sample test. We show that Gaussian prepivoting exactly recovers several randomization tests known to be valid under the weak null, while providing a general approach to restore asymptotic correctness to randomization tests for a large class of test statistics. Importantly, our framework immediately provides valid randomization tests of the weak null hypothesis in rerandomized designs, absent from the literature until now.

A Useful lemmas
Lemma A. For any Borel measurable set B ⊆ R , the centered Gaussian measure of B is a continuous function in terms of the covariance parameter. In other words, γ 0,Σ (B) is a continuous function of Σ over the positive definite cone of × real matrices with metric induced by the Frobenius norm.
Proof. Denote the space of positive definite real × matrices by S ++ ; this is a metric space under the metric induced by the Frobenius norm. Consider a sequence of matrices Σ N ∈ S ++ for which Σ N → Σ. By definition for any Borel The function f (M ) = det(M ) −1/2 is continuous over the positive definite cone of × matrices. Thus, since Σ N → Σ it follows that All that remains to be examined is the limiting behavior of Since all convergent sequences are bounded there exits a positive semidefinite matrix Σ * that is greater than or equal to (in the Loewner partial order) all Σ N . Thus, Σ −1 N Σ −1 * for all N ∈ N. Consequently g(Σ N , x) is dominated by g(Σ * , x) for all N and all x ∈ R . Thus, Lebesgue's dominated convergence theorem implies that Combining (7) and (8) implies that for all sequences Σ N ∈ S ++ such that Σ N → Σ (9) establishes that γ 0,Σ (B) is a sequentially continuous function of the parameter Σ for all Σ ∈ S ++ . Sequential continuity in a metric space is equivalent to continuity (Giaquinta and Modica, 2007, Theorem 5.31); so γ 0,Σ (B) is a continuous function of the parameter Σ for all Σ ∈ S ++ .
Lemma B. Let a function f η (·) satisfy Condition 2 and let φ(·) satisfy Condition 1. Let the matrix V ∈ S (d+k) ++ be defined blockwise as The function h(V, η, x) is a continuous function of V , η, and x jointly.
Proof. Because V is positive definite, V δδ must be as well. Thus, the centered Gaussian measure γ (k) 0,V δδ (·) is non-singular. Furthermore, because φ(·) satisfies Condition 1, the set {b : φ(b) = 1} is Borel measurable with positive Lebesgue measure. Thus, γ The range of κ is the set of Borel measurable sets in R (d+k) . This space can be imbued with the metric 1 where B B is the symmetric difference of B and B and µ(·) is Lebesgue measure on R (d+k) ; this is sometimes called the Fréchet-Nikodým-Aronszajn distance (Conci and Kubrusly, 2017, Section 4). Consider sequences of η N which converge to η and x N which converge to x. Let B N denote κ(η N , x N ); the set-theoretic limit of B N converges to B under d(B, B ). This relies upon the continuity of f η (a) in η. Thus, κ is sequentially continuous in η and x jointly. Sequential continuity in a metric space is equivalent to continuity (Giaquinta and Modica, 2007, Theorem 5.31); so κ is jointly continuous in η and x.
The numerator of h(V, η, x) is the composition of γ (d+k) 0,V (B) with κ(η, x); the former is continuous in V by Lemma A and in B by the absolute continuity of Gaussian measure, and the later is jointly continuous in η and x. Thus, the numerator of h(V, η, x) is jointly continuous in V , η, and x. Since the denominator of h(V, η, x) is a continuous function of V that is always positive, the function h(V, η, x) itself is a jointly continuous function of V , η, and x.

B.1 A reminder: assumptions and conditions
As in the main text, we rely on some regularity conditions which we restate below for convenience.
Assumption 3 of the main text is known to be stronger than necessary for certain results. Here we split Assumption 3 into two parts. We do this to show exactly which results can rely upon a weaker assumption and which results seem to rely upon the stronger assumption.
Assumption 3(a). The worst-case squared distance from the average potential outcome is o(N ); i.e., Further, the above holds for the covariates with x ij replacing y ij (z) above for j = 1, .., k.
Assumption 3(b). There exists some C < ∞ for which, for all z ∈ {0, 1}, all j = 1, .., d and all N , Further, the above holds for the covariates with x ij replacing y ij (z) above for j = 1, .., k.

Assumption 3(b) implies Assumption 3(a) (Wu and Ding, 2018, Proposition 1). Assumption 3(b) is made at times for mathematical convenience to simplify the analysis of certain random distributions; though it remains an open question whether such results hold under weaker assumptions.
Recall theỹ(Z i ) is defined asỹ Further recall the following conditions from the main text. Condition 1. φ : R k → {0, 1} is an indicator function such that the set M = {b : φ(b) = 1} is closed, convex, and mirror-symmetric about the origin (i.e., b ∈ M ⇔ −b ∈ M ) with non-empty interior. Condition 2. For any η ∈ Ξ, f η (·) : R d → R + is continuous, quasi-convex, and nonnegative with f η (t) = f η (−t) for all t ∈ R d . Furthermore, f η (t) is jointly continuous in η and t. Condition 3. With W, Z independent and each uniformly distributed over Ω, for some ξ,ξ ∈ Ξ. Condition 4. With W, Z independent, both uniformly distributed over Ω, and for some ∆ 0, ∆ ∈ R d×d , Oftentimes in the proofs it will implicitly be assumed that the weak null holds. For that reason,ξ andV may be written with y(Z) as inputs rather thanỹ(Z). Let Ω CRE denote the set of allowable treatment allocation vectors z for a completely randomized experiment. Formally

B.2 A remark on limiting distributions for rerandomized designs
A completely randomized experiment can be considered a rerandomized experiment for which φ(·) is identically one. This trivial balance criterion satisfies Condition 1. 2 When φ(·) is not vacuous, the interesting case for rerandomized designs, limiting distributions in completely randomized designs continue to provide corresponding limiting distributions after rerandomization under Condition 1.
By the finite population central limit theorem of Li and Ding (2017), Conditioning according to appropriate balance holding requires that V δδ 0. In this case, the conditional probability of √ Nτ (ỹ(Z), Z) ∈ B subject to φ( √ Nδ(x, Z)) = 1 limits to for any Borel measurable set B.
Likewise, by Proposition 1 of the main text and Lemma 4.1 of Dümbgen and Del Conte-Zerial (2013), the conditional probability of The finite population central limit theorem of Li and Ding (2017) and Proposition 1 are statements about joint convergence in distribution for the scaled differences in means for the observed outcomes and for the covariates. Passing to convergence in distribution conditional upon φ( √ Nδ(x, Z)) = 1 or φ( √ Nδ(x, W)) = 1 described in (10) and (11) rests upon the continuity-set argument used in the proof of Proposition A1 in Li et al. (2018). Condition 1 guarantees that such arguments remain valid: in particular the set M defined within Condition 1 is of positive Lebesgue measure. This allows results for completely randomized designs to provide asymptotics when Ω CRE is replaced with Ω from a general rerandomized design.
for all t ∈ [0, 1]. Furthermore, strengthening Assumption 3(a) to Assumption 3(b), the distributionP G (t) satisfieŝ Proof of Theorem 1. A completely randomized experiment can be viewed as a rerandomized experiment for which φ(b) = 1 for all b ∈ R k ; this φ satisfies Condition 1. As such, the proof below proceeds with general φ satisfying Condition 1 -making no distinction between rerandomized designs and completely randomized design.
First, we focus on the randomization distribution of the prepivoted test statistic; in other words, we examine the limiting distribution of G(y(Z), Z) under H N . By the finite population central limit theorem of Li and Ding (2017) in a completely randomized design or a rerandomized design with φ satisfying Condition 1 the √ N -scaled difference in means, Furthermore, by Condition 1 and Corollary A1 of Li et al. (2018), we have that for Z instead uniform over Ω (accounting for the rerandomized design), where C distributed as before. Unwinding the notation of h(·, ·, ·) gives that G(y(Z), Z) converges in distribution to If we had known to plug in V forV , (12) would exactly amount to applying the f ξ -pushforward of the Gaussian measure γ (d+k) 0,V conditional on φ(b) = 1, which would result in a uniform random variable since this is just the asymptotic probability integral transform for T (y(Z), Z) given that φ( √ Nδ) = 1. However, we do not know V and instead estimate it conservatively using aV that satisfies Condition 4; this results in the discrepancy between the covariance of C versus the covariance used in the Gaussian measure γ (d+k) 0,V in (12). Consequently, (12) amounts to f ξ -pushforward of the Gaussian measure γ (d+k) 0,V in the numerator (the denominator stays the same in both cases since the bottom right block of bothV and V is V δδ ). SinceV V , it follows by Lemma 1 of the main text (and Anderson's theorem more generally) that the numerator of (12) is no larger than the numerator of (12) withV replaced by V . Then, since applying the f ξ -pushforward of the Gaussian measure γ where Z, W, and W are independent and uniformly distributed over Ω CRE and (Ã,B) T and (Ã ,B ) T are independent and identically distributed multivariate Gaussians with mean zero and covarianceṼ . By the conditions on φ outlined in Condition 1, we further have that for Z, W, and W independently drawn from Ω (now accounting for the restrictions imposed by rerandomization), where (D, D ) are independent and identically distributed from the conditional distribution ofÃ | φ(B) = 1.

By Conditions 3 and 4 
Moreover, (14) and (15) hold jointly. Thus, the continuous mapping theorem implies that  where D and D are distributed as before.

C Gaussian prepivoting after regression adjustment C.1 Regression adjustment in completely randomized experiments
In completely randomized experiments with covariate information, a common practice is to use regression-based estimators for treatment effects to improve efficiency. Assume that k is fixed and smaller than N , and let the potential outcomes be univariate. Defineτ reg (y(Z), Z) to be the estimated coefficient on Z i in an ordinary least squares regression of y i (Z i ) on Z i , (x i −x), and Z i (x i −x). Lin (2013a) shows that under suitable regularity conditions,τ reg is √ N -consistent forτ and has an asymptotic variance that is no larger than that ofτ . Importantly, this result holds true without assuming that the linear model inspiringτ reg is actually true. Let be the limit of the OLS slopes for potential outcome under treatment regressed upon covariates, and define Q 0 analogously for the potential outcomes under control. The population level treatment residuals based upon the limiting slopes are then defined as Proposition 2. Suppose Assumption 1 holds, and suppose further that Assumptions 2 and 3(b) hold for the potential outcomes and covariates. Then, Letε i (ỹ(Z), W) be the ith sample residual from a regression ofỹ(Z) on W i , (x i −x), and W i (x i −x). Using the sample residualsε i (ỹ(Z), W) form the variance estimatorŝ Observe thatσ 2 j (y(Z), Z) =σ 2 j (ỹ(Z), Z) for j = 0, 1 regardless of whether or not the weak null holds, but that σ 2 j (ỹ(Z), W) =σ 2 j (y(Z), W) unless the weak null holds. Proposition 3.V reg (ỹ(Z), W) satisfies Condition 4 with V τ τ replaced by V (ε) τ τ andṼ τ τ replaced byṼ (ε) τ τ . The particular form of ∆, the degree to whichV reg (ỹ(Z), Z) is asymptotically conservative, is By Theorem 2, one may apply Gaussian prepivoting to √ Nτ reg usingV reg and any function fξ satisfying Condition 2 and 3; for instance, take fξ( Note that other asymptotically equivalent forms forV reg to the one given here exist. For example, Section 5 of Lin (2013a) suggests using the sandwich variance estimator corresponding toτ reg .

C.2 Proof of Proposition 2
We begin with the following Lemma: Lemma C. If Assumptions 2 and 3(a) hold for the potential outcomes and covariates, then Assumptions 2 and 3(a) hold for the collection of ε i (z). Likewise, if Assumptions 2 and 3(b) hold for the potential outcomes and covariates, then Assumptions 2 and 3(b) hold for the collection ofε i (z).
Proof. For each N , expanding by the definition of ε i (1) yields By inspection, Assumption 2 holds for the collection of ε i (1) so long as the potential outcomes and covariates satisfy Assumption 2. Similar proofs establish Assumption 2 for ε i (0),ε i (0), andε i (1). PREPRINT -JUNE 15, 2021

Suppose that Assumption 3(a) holds for the potential outcomes and covariates. Then
As a consequence of the second statement, and so, by the Cauchy-Schwarz inequality, (19) and (20) that Assumption 3(a) holds for the collection of ε i (z). Now suppose that Assumption 3(b) holds for the potential outcomes and covariates: there exists some C < ∞ for which, for all z ∈ {0, 1} and all N , Modifying the argument from above to accommodateQ instead of Q 1 and applying Hölder's inequality gives the desired result. Specifically, Hölder's inequality implies that where C Q is a constant that does not change with N and depends only uponQ. Combining this inequality with Assumption 3(b) on the potential outcomes then gives that Assumption 3(b) holds for the collection ofε i (z).
We split the proof of Proposition 2 into two: Proposition 2(a) and Proposition 2(b). Proposition 2(a).
Proof. By Lemma A.3 of Lin (2013a), where the sample residualsε i (y(Z), Z) are derived from the regression of y i (Z i ) on Z i , (x i −x), and Z i (x i −x). LetQ 1 (y(Z), Z) be the sample slope coefficient in the OLS regression of y i (Z i ) on x i in the group of individuals for which Z i = 1; similarly, letQ 0 (y(Z), Z) be the sample slope coefficient in the population OLS regression of y i (Z i ) on x i in the group of individuals for which Z i = 0 (Lin, 2013a).
, Z); these are random and depend upon Z. The sample residualε i (y(Z), Z) isε i (Z i ).
By standard OLS theory the slope coefficient matrixQ 1 (y(Z), Z) is defined bŷ By weak laws of large numbers for covariance matrices in finite populations,Q 0 (y(Z), Z) andQ 1 (y(Z), Z) converge in probability to Q 0 and Q 1 , respectively (Lin, 2013a, Lemma A.5). Thus, From this, it follows that This proof closely parallels the logic used in the proof for Theorem 1 of Lin (2013a).
Before proving the remaining component of Proposition 2 we provide a convenient lemma.
Consider a function g : Ω × Ω → R. Let Z and W independently distributed uniformly over Ω. Define two properties: Property A. The random variable g(Z, W) | Z = z converges in probability to c for all conditioning sets {z} N ∈N except for a set of measure zero. Property B. The random variable g(Z, W) converges in probability to c with respect to randomness in both Z and W. Lemma D. Consider a function g : Ω × Ω → R. For Z and W independently distributed uniformly over Ω Property A implies Property B.
Proof. Assume that Property A holds. Fix ε > 0; then Consider P W,Z (|g(Z, W) − c| ≥ ε); by the law of total probability Thus, g(Z, W) converges in probability to c with respect to randomness in both Z and W.
Proof. By definitionτ reg (y(Z) − Zτ , W) is the estimated coefficient on W i in an ordinary least squares regression of By the same logic that gave rise to Lemma A.3 of Lin (2013a), where the sample residualsε i (ỹ(Z), W) are derived from the regression of . LetQ 1 (ỹ(Z), W) be the sample slope coefficient in the OLS regression of y i (Z i ) − Z iτ on x i in the group of individuals for which W i = 1; similarly, letQ 0 (ỹ(Z), W) be the sample slope coefficient in the OLS regression of y i (Z i ) − Z iτ on x i in the group of individuals for which W i = 0. For convenience of notation, denote these are random and depend upon both Z and W.
The centering of the potential outcomes under treatment that occurred when translating y i (z) toỹ i (z) does not impact Assumptions 1, 2, 3(a), and 3(b). Thus, the finite population strong law for second moments (Wu and Ding, 2018, Lemma A.3, Part ii) applies to the sample covarianceŝ Since the centering of the potential outcomes under treatment that occurred when translating y i (z) toỹ i (z) does not impact the above covariance structure, it follows from Lemma A.3 of Wu and Ding (2018) (0)x,∞ (This statement relies upon Assumptions 1, 2, and 3(b).). Condition on a sequence of treatment allocations {Z} N ∈N for the growing sequence of experiments such thatΣỹ (1)x | Z → Σ y(1)x,∞ and Σỹ (0)x | Z → Σ y(0)x,∞ ; this requirement is met for all Z except for a set of measure zero.
Fix the treatment allocations {Z} N ∈N ; after this conditioning we are left with fully determined "imputed potential outcomes": for the "imputed treatment potential outcomes" for the "imputed control potential outcomes" The imputed population can be envisioned as the population that an experiment would imagine to exist if she observed outcomesỹ(Z) and believed that Fisher's sharp null held. Consider W as a treatment allocation for this imputed population. Under this interpretation M 1 (ỹ(Z), W) is the sample covariance between covariates and the imputed outcomes observed under "treatment" W i = 1. Instead of working with M 1 (ỹ(Z), W), we first focus attention to the underlying quantity that M 1 (ỹ(Z), W) seeks to estimate: the covariance between covariates and the imputed potential we proceed with analysis based upon a fixed sequence of treatment allocations Z. This quantity is By the strong laws for the sample means, this shares the same limit as In turn, these two terms can be rewritten as which limits to pΣ y(1)x,∞ + (1 − p)Σ y(0)x,∞ for all Z except for a set of measure zero. Since the centering of the potential outcomes under treatment that occurred when translating y i (z) toỹ i (z) does not impact Assumptions 1, 2, 3(a), and 3(b) it follows from Lemma 1 of Lin (2013a) that x,∞ almost surely in Z; combining this with Lemma D implies that The remainder of the proof proceeds in direct analogy with the proof used for Proposition 2(a).
Remark 2. The utility of Lemma D in the proof of Proposition 2(b) arose from our choice to analyze M 1 (ỹ(Z), W) through conditioning upon treatment allocation Z. With this conditioning argument, Assumption 3(b) is leveraged to attain strong laws with respect to randomness in Z; these guarantee that arguments based upon conditioning on Z = z hold for all but a set of measure zero. An alternative approach to arrive at the statement M 1 (ỹ(Z), W) p − → pΣ y(1)x,∞ + (1 − p)Σ y(0)x,∞ may be to work unconditionally: appealing to a suitable weak law while allowing for randomness in both Z and W. With an approach of this nature, Assumption 3(b) may be stronger than necessary.

C.3 Proof of Proposition 3
First we show thatV with ∆ defined in the statement of Proposition 3. From the proof of Proposition 2(a) Since N/n 1 → p −1 and N/n 0 → (1 − p) −1 this has the same limit as N → ∞ as Thus, (22) holds by the weak law of large numbers for second moments (Lin, 2013a, Lemma A.1) and second part of Theorem 2 from Lin (2013a).
Next we show thatV By the proof of Proposition 2(b) By conditioning upon Z, an argument similar to that used to analyze M 1 (ỹ(Z), W) in the proof of Proposition 2(b) can then be applied to compute the probability limits of the first two terms in (24) almost surely with respect to the conditioning variable Z. Then leveraging Lemma D yields that the probability limit is the same when considering randomness in both Z and W. Finally, using N/n 1 → p −1 and N/n 0 → (1 − p) −1 yields that

D An example for paired designs
The main text focuses upon rerandomized experimental designs. Since a completely randomized experiment is simply a rerandomized experiment with trivial balance criterion, the results of the main text automatically apply to completely randomized experiments as well. However, Gaussian prepivoting is not limited to just these contexts. Here we illustrate the utility of Gaussian prepivoting for a matched-pair experimental design. Before describing the exact details of Gaussian prepivoting for paired designs, we prove a generalization of Theorem 1 and Theorem 2.
Suppose that A(·, ·) is a function such that under the weak null H N and for some positive definite matrices V andṼ , Remark 3. With the introduction of the covariance matrices in (25) and (26) we must modify Condition 4 slightly. It now becomes: with W, Z independent, both uniformly distributed over Ω, and for some ∆ 0, ∆ ∈ R d×d , Theorem 3. Suppose that T (y(Z), Z) = fξ(A(y(Z), Z)) for some f η andξ satisfying Conditions 2 and 3 . If we employ a covariance estimatorV satisfying the revised Condition 4 when forming the prepivoted test statistic G(y(Z), Z) then, under H N :τ = 0 and the assumption that (25) holds, G(y(Z), Z) converges in distribution to a random variableŨ taking values in [0, 1] satisfying for all t ∈ [0, 1]. Furthermore, if (26) holds, then the distributionP G (t) satisfieŝ Proof. The proof of this theorem proceeds exactly as that of Theorem 1, but with √ Nτ (·, ·) replaced by A(·, ·).
The structure assumed in Theorem 3 is commonly encountered in finite population causal inference across a host of experimental designs. Armed with Theorem 3 we turn to the problem of Gaussian prepivoting in paired designs. Consider a population with I matched pairs of individuals, so that the total population size is N = 2I. Attributes of the j th unit in the i th pair are subscripted with ij; e.g. potential outcomes are y ij (0) and y ij (1). For simplicity take d = 1, though these results are not bound to the univariate case. For a paired design Ω = Ω pair with In words, allowable treatment allocations assign one unit of each pair to treatment and the remaining unit of the pair to control. The average observed treated-minus-control difference in outcomes iŝ , with T i (y(Z), Z) representing the treated-minus-control difference in pair i. Subject to Conditions 1 and 2 of Fogarty (2018) -which are the paired-design analogues of our Assumptions 2 and 3(b) -the random variableτ pair (y(Z), Z) obeys a finite population central limit theorem. This finite population central limit theorem for √ I (τ pair (y(Z), Z) − τ ) can be derived from Theorem 1 of Fogarty (2018) by dropping the regression-assisting terms.
Theorem 3 applies to G pair (y(Z), Z) and so prepivoting naturally extends to paired experimental designs. In fact, for paired designs there are numerous candidates for the variance estimatorV that extend beyond the Neyman-style estimatorV pair ; for examples, see the regression-assisted variance estimators of Fogarty (2018) or the pairs of pairs estimator discussed in Abadie and Imbens (2008) and Fogarty et al. (2019).

E Experiments with many treatments
Theorems 1 and 2 are not limited to experiments with only two treatment arms (e.g. treatment versus control). In this section, we show that these results extend naturally to experiments with an arbitrary finite number of treatment arms. For simplicity we present notation and results for completely randomized designs, but extensions are available to rerandomized designs as in the two-armed case. Special cases of balance criteria for general multi-armed rerandomized designs are discussed in Morgan and Rubin (2012, Section 5.2); for rerandomization in factorial experiments Li et al. (2020) and Branson et al. (2016) Modify Assumption 1 to be that N −1 n a → p a with p a ∈ (0, 1) for all a. In words, no treatment arm is asymptotically degenerate. The remaining assumptions are modified to hold for z ∈ [A − 1] instead of just z ∈ {0, 1}. In the multi-arm setting we redefine Fisher's sharp null to be

The corresponding generalization of Neyman's weak null is
See Ding and Dasgupta (2018) for discussion of this generalization of the sharp and weak nulls. Further generalizations of these nulls can be found in Wu and Ding (2018). Consider a matrix C y of dimensions A × d for some d . We stipulate that this matrices is comprised of column-wise contrasts; i.e., each column contains some non-zero element but sums to zero. In place ofτ (y(Z), Z) we now turn to the weighted treatment effect estimatorτ where the vec(·) operator reshapes d-by-d matrices to (dd )-length vectors by vertically concatenating the columns. In the classical two-armed experiment C y = [−1, 1] T returns the standard difference in means asτ C (y(Z), Z).
We extend the notationΣ y(a) to denote the sample variance estimator in the population which received treatment arm a.
Mimicking the argument of Wu and Ding (2018, Section 2.3) with the natural extension to multivariate outcomes gives that an asymptotically conservative covariance estimator for where ⊕ denotes the direct sum of matrices, resulting in a block-diagonal matrix of dimension Ad × Ad with (N/n a−1 )Σ y(a−1) in the a th of A blocks. Henderson and Searle (1979) present numerous algebraic properties of the Kronecker product and the vectorization operator; exploiting their equation (6)  N . Using Hoeffding's Lemma -see for instance Dümbgen and Del Conte-Zerial (2013, Lemma 4.1) -in an approach mirroring that used in our proof of Theorem 1 the analysis of Wu and Ding (2018) provides an unconditional understanding of √ N (τ C (y(Z), W) under the weak null. Combining their results gives that These results lay the basis for applying Gaussian prepivoting to test statistics T (y(Z), Z) of the form fξ (τ C (y(Z), Z)) with fξ satisfying Conditions 2 and 3. The prepivoted test statistic takes its usual form a : fξ(a) ≤ fξ (τ C (y(Z), Z)) .
The central limit behavior of √ Nτ C (y(Z), Z) and asymptotic conservativeness of the Neyman-style variance estimator V C (y(Z), Z) apply to Theorem 3 to show that the true distribution R G C is asymptotically stochastically dominated by the standard uniform distribution. The central limit behavior of √ Nτ C (y(Z), W) in conjunction with (27) implies that the reference distribution P G C limits weakly in probability to the standard uniform distribution; thereby furnishing inferences which are exact for H (A) F and asymptotically conservative for H (A) N . The only major difference between the results for A > 2 and A = 2 is the use of more general finite population central limit theorems forτ C (y(Z), Z) in place of those forτ (y(Z), Z); such a central limit theorem is given by Li and Ding (2017, Theorem 5). Through particular choices for C y , f η , andξ, we can recover the test statistic proposed in Ding and Dasgupta (2018) for testing the weak null in multi-armed trials while additionally providing an alternative fix to the usual F -statistic to restore asymptotic sharp-dominance. Similarly, the same reasoning can be applied to asymptotically linear estimators (cf. Section 7 of the main text) and the proof of Theorem 2 does not change substantially.

F Exact and asymptotically valid confidence sets
Our results readily extend to constructing confidence intervals which are both asymptotically conservative for the sample average treatment effect and exact if a constant treatment effect model holds. To this end, we first describe how to test the hypotheses H F,c : . . . , N and H N,c Then by replacingτ (y(Z), Z) andτ (ỹ(Z), W) withτ (ỹ c (Z), Z) andτ (ỹ c (Z), W), respectively, in Equation (4) of the main paper the test for H F and H N developed in Section 5 of the main paper yields a single procedure that is exact for for H F,c and asymptotically conservative for H N,c . Now, first suppose that one is willing to assume a constant effect model and desires a confidence set for the value c such that τ i = c for all i = 1, . . . , N . Second, suppose that one wants a confidence set for the average treatment effect without assuming a constant effect; i.e., a confidence set for the value c for which τ = c. Fix a confidence level 1 − α ∈ (0, 1) and let C(T, y(Z), Z) be the acceptance region for the test of H F,c conducted at level α based upon the test statistic T (·, ·) evaluated over c ∈ R d .
Theorem 1 in the main text implies that randomization inference using the prepivoted test statistic G(·, ·) yields exact tests for H F,c and asymptotically valid tests for H N,c . Leveraging the duality between hypothesis testing and confidence sets (Casella and Berger, 1990, Theorem 9.2.2) implies the following corollary of Theorem 1.
Corollary 2. Assume that the regularity conditions of Theorem 1 hold and consider the set C(G, y(Z), Z) formed by inverting the randomization test of H F,c conducted using the prepivoted test statistic G(ỹ c (Z), ·). C(G, y(Z), Z) is both an asymptotically conservative confidence set for the sample average treatment effect and an exact confidence set under the additional assumption of constant treatment effects.
Corollary 2 implies that inverting hypothesis tests based on Gaussian prepivoting yields a single confidence set with both 1 − α coverage for under a constant effect model at all finite N and also at least 1 − α asymptotic coverage for the sample average treatment effect. The extension to generating confidence sets based upon asymptotically linear estimators of the formT follows similarly but rests upon Theorem 2 instead of Theorem 1. Consequently, regression adjustment can be incorporated into the confidence set generating procedure.
G Additional simulations G.1 The generative model Theorem 1 and its generalizations concern the finite sample and asymptotic Type I error rates of testing H F and H N . Here we provide additional simulations to highlight the potential for anti-conservative inference in the absence of prepivoting and to investigate the statistical power of the Fisher Randomization Test based upon prepivoted test statistics.
Our simulations proceed similarly to those of Section 9.2 in the main text. For completeness, we detail the simulation set-up here. In each iteration b = 1, ..., B, we draw {r i (1)} N i=1 and {r i (0)} N i=1 independent from one another and iid from mean zero equicorrelated multivariate normals of dimension k = 25 with marginal variances one. The correlation coefficients governing r i (1) and r i (0) are 0 and 0.95 respectively. For both our type I error and power simulations we will have two simulation settings, one with constant treatment effects and one with heterogeneous treatment effects: Constant Effects: y i (1) − τ = y i (0) = r i (1).
The experimental design is that of a completely randomized experiment in which n 1 = 0.2N . We test for treatment effect using randomization inference based upon the following three statistics: 1. Hotelling's T -squared, unpooled covariance, 2. Hotelling's T -squared, pooled covariance, 3. Max absolute t-statistic, unpooled standard error.

G.2 Type I error rates
When τ = 0 in the constant effects simulation setting in §G.1, Fisher's sharp null holds; likewise taking τ = 0 under heterogeneous effects enforces Neyman's weak null. In these contexts, we reexamine the Type I error rate simulations of the main paper, but at α = 0.25 instead of 0.05. While this value of α is larger than typical in scientific practice, the larger value of α allows frequent rejections of the null hypothesis despite the inherent conservativeness of inference under the finite population model. We stress that the conservativeness of these tests rests upon the finite population inference framework and not upon the mechanics of Gaussian prepivoting: the non-identifiability of Σ τ,∞ forces conservativeness of any procedure which asymptotically guarantees Type I error rate control under H N . We conduct simulations with N = 300 and N = 5000; for each N , we conduct B = 5000 simulations. These tests are conducted using Monte-carlo simulation to generate the reference distributions, with 1000 draws from Ω CRE for each iteration b. Table 3 presents Type I error rates under simulation with τ = τ = 0.
As observed both in Table 3 above and in Table 2 of the main text, the Fisher Randomization Test using the pooled Hotelling T -statistic demonstrates significant anti-conservativeness under H N . Moreover, we see from Table 3 that at α = 0.25 the Fisher Randomization Test using the max t-statistic is also anti-conservative under H N , a problem that persists even when N = 5000. As a demonstration of Theorem 1, the Fisher Randomization Tests using the prepivoted versions of T pool and T |max| control the Type I error rate under Neyman's null for large N . Table 3: Type I error rates in completely randomized designs with multiple outcomes. The rows describe the simulation settings, which vary between the sharp and weak nulls holding and between small and large sample sizes. There are three sets of columns, one corresponding to each of the three test statistics under consideration. For each set of columns, the column labeled "FRT" represents the Fisher Randomization Test using that test statistic. The column labeled "Pre." instead reflects the Fisher Randomization Test after applying Gaussian prepivoting to the original test statistic. The last column, labeled "LS," is a large-sample test which is asymptotically valid for the weak null. The desired Type I error rate in all settings is α = 0.25. For all columns τ = τ = 0.

G.3 Power after prepivoting
Below we provide a theoretical discussion of the power of the Fisher Randomization Test based on G(y(Z), Z) and use simulations to highlight key aspects of its statistical power in practice.
When prepivoting is not necessary because the test statistic being deployed is already sharp dominant and pivotal, its use does not affect the power of the test. Suppose that the Fisher Randomization Test using the pivotal test statistic T (y(Z), Z) provides exact inferences under H F and asymptotically valid inferences under H N . Examples of such test statistics include the studentized absolute difference in means for d = 1 and its multivariate analogue T χ 2 (y(Z), Z); see Wu and Ding (2018) for further examples. Let N fξ,V denote the fξ-pushforward of the Gaussian measure γ (d+k) 0,V . Since fξ takes values in R the pushforward measure N fξ,V is a distribution on the real line, and so -in a slight abuse of notation -we write its corresponding cumulative distribution function evaluated at t ∈ R as N fξ,V (t). For a completely randomized experiment G(y(Z), Z) = N fξ,V (T (y(Z), Z)). If N fξ,V is pivotal in the sense that its distribution does not depend upon unknown parameters requiring estimation, then G(y(Z), W) is a fixed continuous non-decreasing transformation of T (y(Z), W). Consequently, for any fixed Z the pair (G(y(Z), w), T (y(Z), w)) has rank correlation 1 when enumerated over w ∈ Ω and so p-values derived underP T exactly match those underP G . In this case, prepivoting has no impact upon the power of the test: a test statistic T (y(Z), Z) with high power under the alternative will yield G(y(Z), Z) with high power as well. An example of such a case is T χ 2 .
However, as demonstrated in Section 5 of the main paper, there are cases for which T (y(Z), Z) cannot be used for randomization inference under H N because it is not asymptotically sharp dominant. Examples of this include T pool (y(Z), Z) and T |max| (y(Z), Z). Even in these cases, the asymptotic power of the Fisher Randomization Test using G(y(Z), Z) can be computed. Regardless of pivotality, the test statistic G(y(Z), Z) itself is the complement of a p-value for an asymptotically valid test of H N . In other words, 1 − G(y(Z), Z) can be used directly as a p-value for testing H N with asymptotic control of the Type I error rate (simply reject H N if 1 − G(y(Z), Z) ≤ α.) The power of this large-sample test must be computed on a case-by-case basis since it is reliant on the structure of the underlying test statistic T (y(Z), Z). Because the reference distribution employed by Gaussian prepivoting converges to a standard uniform even under the alternative, the asymptotic power of the randomization test using G(y(Z), Z) converges to the power of this large-sample test of H N , with the added benefit that the randomization test is exact for finite N under H F . Randomization inference after Gaussian prepivoting leverages an asymptotically valid test for H N and furnishes exact inference for H F with no sacrifice in asymptotic power against H N . In other words, for test statistics T (y(Z), Z) satisfying Conditions 2 and 3 exactness under H F can be achieved for free, with limiting power remaining equal to that of the large-sample test upon which G(y(Z), Z) is based. Table 4 presents power simulations under the set-up detailed above for τ = τ = 0.05e, where e denotes the vector of all ones. Under constant effects, the power of all of the tests is high and the Type I error rate is controlled for H F because we are using Fisher Randomization Tests. Although the power of the Fisher Randomization Tests using T pool and T |max| is very high for heterogeneous effects, as observed in Table 3 the randomization tests of T pool and T |max| do not control the Type I error rate for testing H N even asymptotically. However, for the tests which do asymptotically control the Type I error rate under H N the randomization test of G(y(Z), Z) has power observed to be close to that of the large-sample test. Furthermore, the gap in power between the two diminishes as N increases. As stated above, this is because the critical value deployed by the randomization test of G(y(Z), Z) is converging to 0.75 as N increases, Table 4: Power simulations in completely randomized designs with multiple outcomes. The rows describe the simulation settings, which vary between constant and heterogeneous effects and between small and large sample sizes. There are three sets of columns, one corresponding to each of the three test statistics under consideration. For each set of columns, the column labeled "FRT" represents the Fisher Randomization Test using that test statistic. The column labeled "Pre." instead reflects the Fisher Randomization Test after applying Gaussian prepivoting to the original test statistic. The last column, labeled "LS," is a large-sample test which is asymptotically valid for the weak null. The desired Type I error rate in all settings is α = 0.25. For all columns τ = τ = 0.05e where e is the vector of all ones.

Hotelling, Unpooled
Hotelling, Pooled Max t-stat while the large-sample test rejects for G(y(Z), Z) ≥ 0.75. This further highlights the asymptotic equivalence between the two approaches.
The results for the prepivoted test based upon the usual (pooled) Hotelling test yield two interesting observations. First, the pooled test has markedly worse power than the unpooled or max-t statistics, as the use of the pooled covariance matrix in forming the test statistics amounts to a choice of a suboptimal norm for constructing the test. Second, the power actually decreases for both the prepivoted randomization test and the large-sample test when going from N = 300 to N = 5000. This can be attributed to the large-sample approximation being quite poor for the pooled test at N = 300 in the setting under consideration. As shown in the simulations in the manuscript and in Table 3, the Type I error rate exceeds the nominal level when the weak null is true at N = 300, but falls below it at N = 5000. As N increases the large-sample approximation becomes better, hence restoring the conservativeness of the test under the null. This behavior also drives the apparent reduction in power in the above table. The power still tends to 1 for the pooled Hotelling test as N → ∞ in the generative model yielding this simulation study.

H Gaussian integral formulation
In the main text we used the notation γ ( ) µ,Σ (B) to denote the measure of a Borel-measurable set B ⊆ R under Gaussian measure with mean µ and covariance Σ. Here we provide equivalent formulations of the example Gaussian prepivoted test statistics examined in Section 5 of the main text, but instead of using γ ( ) µ,Σ we directly write the corresponding Gaussian integrals.
Example 6 (Multivariate studentization). Let √ Nτ now be multivariate and suppose we have a completely randomized design; consider the test statistic where F d (·) is the distribution function of a χ 2 d random variable. Example 7 (Max absolute t-statistic). Consider again multivariate √ Nτ in a completely randomized design and the test statistic whereV τ τ,jj is the jj element ofV τ τ . For this statistic, f η (t) = max 1≤j≤d |t j |/η j , andξ = (V 1/2 τ τ,11 , ...,V 1/2 τ τ,dd ) T . After Gaussian prepivoting we are left with Importantly, (29) and (30) differ only in the support of the Gaussian integral. The same Gaussian measure is used; the difference is that the support of (29) is an ellipsoid while the support of (30) is a hyperrectangle.
Example 8 (Rerandomization). Let √ Nτ be univariate and suppose we now consider a rerandomized design with balance criterion φ satisfying Condition 1. Consider the absolute difference in means, fξ( √ Nτ ) = √ N |τ |, such that ξ = 1. Gaussian prepivoting yields the test statistic Since φ(·) is a boolean-valued function it directly constrains the support of the Gaussian integrals in (31).

I Discussing Condition 2
Recall Condition 2 from the main text: Condition 2. For any η ∈ Ξ, f η (·) : R d → R + is continuous, quasi-convex, and nonnegative with f η (t) = f η (−t) for all t ∈ R d . Furthermore, f η (t) is jointly continuous in η and t.
Each condition on f η plays an important role in the underlying mechanics of Gaussian prepivoting, and each deserves some degree of attention. First, the joint continuity of f η (t) in η and t plays a critical role in the asymptotic behavior of the test statistic T . When computing the asymptotic distributional behavior of T (y(Z), Z) we leverage the central limit theorem governing √ Nτ (y(Z), Z) and the continuous mapping theorem to obtain the distributional limit of fξ (y(Z),Z) √ Nτ (y(Z), Z) .
Without the joint continuity of f η (t), such a generic asymptotic result would not be feasible. The same reasoning shows the utility of Condition 2's joint continuity requirement when analyzing fξ (y(Z),W) √ Nτ (y(Z), W) .
In this sense, the joint continuity assumption is of technical importance for deriving asymptotic distributional behavior. The quasi-convexity and mirror symmetry assumptions are of a more fundamental nature to our results; they are inextricably linked to Anderson's (1955) theorem for multivariate Gaussians and so they play a crucial role in guaranteeing the asymptotic sharp dominance of G(y(Z), Z). A quasi-convex function is a function with convex sublevel sets; for those unfamiliar with quasi-convex functions, we suggest the excellent review of Avriel et al. (2010, Chapter 3). A simple example function from R d → R that is both quasi-convex and mirror symmetric about the origin is the Euclidean norm t → ||t||. Generalizing slightly more, if f η (t) is any seminorm on R d which is jointly continuous in η and t, then f η (·) satisfies Condition 2. In fact, this is nearly a complete characterization; we will show that the criteria of Condition 2 stipulate that f η (·) is tightly related to a seminorm; though f η (·) need not be a seminorm itself. Our discussion centers around the case of a completely randomized experiment, but this restriction is only for the sake of explication; similar reasoning applies in the rerandomized case as well.
Consider a convex set U ⊂ R d ; suppose that U is balanced in the sense that cU ⊆ U for all scalars u ∈ [−1, 1]. Such a set U is necessarily mirror-symmetric about the origin, and so Anderson's theorem states that: Moreover, such a set U defines a seminorm on R d via its Minkowski functional ρ U (t) = inf k>0 {t ∈ kU}. In light of this, Anderson's theorem can be rewritten as: Denote the preimage of a set S under f η by f −1 η (S). By quasi-convexity and symmetry of f η , the set f −1 ξ ([−∞, T ]) is convex and symmetric about the origin for any T ∈ R. Specifically, taking U = f −1 ξ ([−∞, T (y(Z), Z)]) yields a random seminorm ρ U . Finally, taking S Y =V and S X = V gives exactly that the randomization distribution of the Gaussian prepivoted test statistic G(y(Z), Z) is asymptotically dominated by the uniform distribution.
In other words, Anderson's theorem can be rephrased to say that when S Y − S X 0 the random variable X is more concentrated in any semi-norm than Y; Condition 2 is designed exactly so that the random set U = f −1 ξ ([−∞, T (y(Z), Z)]) generates a seminorm via its Minkowski functional.

J Details of examples
In the main text, we provide several examples of test statistics which are amenable to Gaussian prepivoting. Here we provide details to verify the conditions of Theorem 1 for these examples.
Verifying the second part of Condition 4 requires examining the limiting behavior ofV (y(Z), W). Such an analysis can be found in Wu and Ding (2018, Appendix A2); while their work focuses on the scalar-outcome many-treatment case, the techniques convert straightforwardly to the vector-outcome treated-versus-control case.
Lemma E establishes that the covariance estimators used for prepivoting in Section 5 of the main text are indeed in accordance with Condition 4. Next, we examine each example provided in the main text to establish that Conditions 2 and 3 are met.
Example 9 (Absolute Difference in Means). In a completely randomized experiment, we consider T DiM (y(Z), Z) = √ N |τ |, with f η (t) = |t| andξ = 1. Condition 3 is trivially satisfied sinceξ is not stochastic. Condition 2 follows from the continuity, convexity, non-negativity, and symmetry of the absolute value function.
Example 10 (Multivariate studentization). Let √ Nτ now be multivariate and suppose we have a completely randomized design. We examine the statistic withV τ τ = N n1Σ y(1) + N n0Σ y(0) . For this test statistic, f η (t) = t T η −1 t andξ =V τ τ . Sinceξ matches the top-left block of the Neyman covariance estimator Lemma E shows that Condition 3 is met. Condition 2 holds because the quadratic form f η (t) = t T η −1 t is certainly mirror symmetric, jointly continuous and convex by standard results for quadratic forms, and is non-negative since η is positive definite, and so its inverse must be as well.
Example 11 (Max absolute t-statistic). Consider again multivariate √ Nτ and a completely randomized design. The max-absolute t-statistic is whereV τ τ,jj is the jj element ofV τ τ . For this statistic, f η (t) = max 1≤j≤d |t j |/η j , andξ = (V 1/2 τ τ,11 , ...,V 1/2 τ τ,dd ) T . Sinceξ is the square-root of the diagonal elements ofV τ τ , Lemma E again establishes Condition 3. Certainly each coordinate projection |t j |/η j is jointly continuous in η j > 0 and t j . Taking the maximum over these functions preserves continuity. The maximum of linear functions is convex so f η (t) is quasi-convex. Non-negativity and mirror symmetry are trivial algebraic properties inherited from the coordinate-wise absolute value function. Thus, Condition 2 holds.
Example 12 (Rerandomization). Consider a rerandomized design with balance criterion φ satisfying Condition 1 and let √ Nτ be univariate. Consider the absolute difference in means, fξ( √ Nτ ) = √ N |τ |, such thatξ = 1. As before, sinceξ is non-stochastic Condition 3 is immediate. The continuity, quasi-convexity, non-negativity, and symmetry of f η are immediate consequences of the properties of the absolute value function. Thus, Condition 2 holds. We demonstrate inference using Gaussian prepivoting in a completely randomized experiment. Angrist et al. (2009) implemented a moderate-scale completely randomized experiment to test the effectiveness of several strategies intended to boost academic performance. Their experiment, the so-called Student Achievement and Retention (STAR) project, enrolled incoming first-year undergraduate students -except those with high-school grade point average (GPA) in the top 25% -in one of three treatment arms: a student support program, a financial incentive program, or both. Allocation to the programs was performed completely at random. Numerous demographic features of program participants were collected; we focus specifically on the participants' reported genders and high-school GPAs. The primary outcomes of the study were first-year GPA and second-year GPA. Further details on the nature of the interventions and the specific demographic features collected can be found in Angrist et al. (2009). The data collected in the STAR project is publicly available in the online supplement to Angrist et al. (2009). Angrist et al. (2009) found no evidence to suggest that the program was effective at improving educational outcomes among participants who identified as men. Lin used regression-adjusted estimators to examine inference for the marginal effect of offering financial incentives given that support services were offered; his analysis focuses on only the male participants of the study. Lin performs several simulations under the assumption that Fisher's sharp null holds, but he remarks that Chung andRomano (2011a, 2011b) discuss and extend a literature on permutation tests that do remain valid asymptotically when the null hypothesis is weakened. One such test is based on the permutation distribution of a heteroskedasticity-robust t-statistic. Exploration of this approach under the Neyman model (with and without covariate adjustment) would be valuable.
Gaussian prepivoting allows us to meet and exceed this objective: a permutation-testing framework can be applied with asymptotic validity under H N for a wide class of statistics -including multivariate statistics for which studentization alone is insufficient to restore asymptotic conservativeness. Theorem 1 guarantees finite sample exactness under H F and asymptotic conservativeness under H N of the prepivoted test statistic G(y(Z), Z) subject to mild conditions; moreover, Section C of the supplement extends this result to the context of regression-adjusted estimators.
We re-analyze the data studied by Angrist et al. (2009) through the lens of Gaussian prepivoting. Instead of restricting to the univariate outcome of first-year GPA, we examine the effect of treatment on both first-year and second-year GPA. We implement prepivoting using the test statistics of Section 5 of the main text: • the Euclidean 2-norm of the difference in means, denoted T ||·||2 , • the multivariate studentized statistics T χ 2 and T P ool , • the maximum absolute t-statistic T |max| .
Furthermore, we implement prepivoting in the cases above using the regression adjusted estimator of the difference in means -regressing on high-school GPA -instead of the naïve difference in means. In total N = 141 male-identifying participants have complete covariate and outcome data (high-school GPA, first and second year GPAs, respectively) and were offered at least support services. Of these individuals, n 1 = 55 were offered both support services and financial incentives while n 0 = 86 received only the offer for support services. Table 5 contains p-values of the Fisher Randomization Test before and after prepivoting. The p-values obtained after prepivoting provide exact inference for H F ; moreover, the asymptotic results of Theorems 1 and 2 of the main text suggest that these p-values are likely to provide conservative inference for H N . We stress that without Gaussian prepivoting only for T χ 2 would the Fisher Randomization Test be appropriate for Neymanian inference. The other three base statistics of Table 5 can exhibit asymptotically anti-conservative inference with the Fisher Randomization Test under H N . For both T ||·||2 and T P ool -with and without regression adjustment -the p-value obtained after prepivoting is no less than than the p-value derived without first prepivoting. These increased p-values suggest that the non-prepivoted procedures for T ||·||2 and T P ool may have been anti-conservative. In fact, with α = 0.1 an experimenter erroneously using the Fisher Randomization Test based upon regression adjustment with T ||·||2 to test H N would have rejected the null of no average effect. Once prepivoting is applied the practitioner is asymptotically entitled to test H N with the Fisher Randomization Test and we observe that the procedure no longer rejects H N , thereby rectifying the potentially anti-conservative nature of the preceding result.
The code to implement our analysis is provided online to facilitate reproducibility.

L Software
Code written in R that builds the figures of the paper is available online at https://github.com/PeterLCohen/ PrepivotingCode. Furthermore, at the same location, we provide concrete examples -also written in R code -illustrating how one might choose to implement Gaussian prepivoting from scratch. For simplicity, we present prepivoting the absolute √ N -scaled difference in means for a univariate completely randomized design. In other words, we exactly demonstrate the implementation of Algorithm 1 for Gaussian prepivoting used in Example 1 of Section 5 in the main paper. At the same location, we provide R code to reproduce the results of the data analysis in Section K.