{'fraction_non_alphanumeric': 0.033990337338064905, 'fraction_numerical': 0.013532002223267349, 'mean_word_length': 4.574901680371827, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 10, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
On the Performance of the Neyman Allocation with Small Pilots

The Neyman Allocation and its conditional counterpart are used in many papers on experiment design, which typically assume that researchers have access to large pilot studies. This may not be realistic. To understand the properties of the Neyman Allocation with small pilots, we study its behavior in a novel asymptotic framework for two-wave experiments in which the pilot size is assumed to be fixed while the main wave sample size grows. Our analysis shows that the Neyman Allocation can lead to estimates of the ATE with higher asymptotic variance than with (non-adaptive) balanced randomization. In particular, this happens when the outcome variable is relatively homoskedastic with respect to treatment status or when it exhibits high kurtosis. We also provide a series of empirical examples showing that these situations arise frequently in practice. Our results suggest that researchers should not use the Neyman Allocation with small pilots, especially in such instances.


Introduction
A growing literature on experiment design provides researchers with tools for reducing the asymptotic variance of their average treatment effect (ATE) estimates. Many do so in the context of two-wave experiments, where the researcher has access to a pilot study that can be used to improve the main study. Pilots are typically assumed to be large, allowing population parameters to be well-estimated. However, large pilots may not be realistic in practice. In this paper, we study the implications of small pilots for experiment design through the lens of the Neyman Allocation.
The Neyman Allocation (Neyman 1934) is a simple method for minimizing the variance of the difference-in-means estimator of ATE. In a setting without covariates, suppose that the standard deviations of the treated and control outcomes are known. The Neyman Allocation assigns more units to either treatment or control in proportion to the ratio of their standard deviations. Intuitively, the optimal experiment entails more measurements of the noisier quantity. Since the variances are not known in practice, the feasible Neyman Allocation (FNA) estimates the variances using the pilot study and then plugs the estimates into the assignment rule.
The FNA is an important part of many experiment design procedures. In the econometrics literature alone there are several notable works. Hahn, Hirano, and Karlan (2011) propose to estimate the variance of outcome and control groups conditional on covariate value, implementing the FNA conditional on covariates. In a similar vein, Tabord-Meehan (2021) employs tree-based techniques to stratify units based on their covariates. Units are then assigned to treatment and control based on the FNA conditional on strata. Meanwhile, Cytrynbaum (2021) proposes local randomization to select representative units for participation and treatment in experiments. In what Cytrynbaum (2021) terms the "fully efficient" case, treatment proportion conditional on the randomization group is chosen by the FNA. Despite their differences, the above papers study their proposals in asymptotic frameworks that take the pilot size to infinity. Their analyses, appropriate for large pilots, essentially assume that population parameters are arbitrarily well-estimated from the pilots alone. In practice, pilots are often conducted for logistical reasons and may be small. In such settings, accurately estimating the relevant variances may be difficult.
To understand the implications of small pilots, we study the properties of the Neyman Allocation in a novel asymptotic framework for two-wave experiments. Our framework takes the main wave sample size to infinity while the pilot size remains fixed. We show that when uncertainty in parameter estimation is non-negligible in the limit, the FNA converges in distribution to a mixture of normal distributions. Furthermore, we find that the FNA can do worse than the naive, balanced allocation that assigns half the units to treatment and half to control. This occurs when outcomes have similar variances across treatment and control, i.e. when outcomes are relatively homoskedastic with respect to treatment status. To assess how much homoskedasticity exists in practice, we examine the first 10 completed experiments in the AER RCT Registry. We ask the hypothetical question: if researchers conducted these studies as a two-wave experiment with a random sample from the same population, would they do better with the FNA or with balanced randomization?
We find that the treatment and control groups are often highly homoskedastic across a range of outcomes and across experiments. This suggests that if faced with a small pilot, the authors of those studies would likely not have benefited from implementing the FNA. Finally, we show that as pilot sizes increase, the amount of heteroskedasticity needed for the FNA to be preferable to the balanced allocation decreases, but at a rate that depends on the kurtosis of the outcome variables.
Hence, even when researchers believe they are in a setting with high heteroskedasticity, they may want to avoid the FNA if they also believe that the outcomes are fat-tailed.
Our findings suggest that researchers should be cautious when designing experiments using the FNA with small pilots. However, even when pilots are large, methods which condition on many covariates may end up estimating the FNA using a small conditional sample. Furthermore, if researchers believe that units exhibit cluster-dependence -a common assumption in empirical work -the number of "effective" observations may be smaller still, impeding the estimation of the FNA. We note that our paper specifically addresses the use of the FNA in reducing the asymptotic variance of the difference-in-means estimator in two-wave experiments. It does not speak to papers which take the treatment assignment probability as given, such as Bai (2022).
Our paper is most similar to papers pointing out a similar issue in the design of sequential experiments. Melfi and Page (1998) argue by simulation that treatment assignment rules based on estimated outcomes can do worse than non-adaptive rules due to estimation noise. Theoretical analysis is provided in Hu and Rosenberger (2003), in an asymptotic framework that does not nest ours.
Our paper is also related to those studying small sample problems in experiments. Chaisemartin and Ramirez-Cuellar (2019) is concerned with the problems small strata pose for inference. Bruhn and McKenzie (2009) considers the effectiveness of various randomization strategies in achieving balance when a single-wave experiment is small. Finally, we note that there is a large literature discussing the large-pilot properties of the Neyman Allocation for alternative criteria such as power (e.g. Brittain andSchlesselman 1982, Azriel, Mandel, andRinott 2012), minimax optimality (e.g. Bai 2021) or ethical considerations (e.g. Chapter 8 of Hu and Rosenberger 2006). These criteria fall outside the scope of this present paper.
The remainder of this paper is organized as follows. Section 2 presents the theoretical framework. Section 3 contains analytical and simulation results using a stylized toy example. Our main theoretical results can be found in section 4. We assess the level of homoskedasticity in selected empirical applications in section 5. Section 6 concludes the paper.

Framework
We use a standard binary treatment potential outcomes framework assuming an infinite superpopulation. The potential outcomes are (Y (0), Y (1)), where Y (0) denotes the potential outcome under control or status quo and Y (1) denotes the potential outcome under treatment or the innovation. Assumption 2.1. Potential outcomes (Y (0), Y (1)) have finite second moments. The vector of means is µ and the covariance matrix is Σ where Additionally, assume potential outcome variances are positive so that σ 2 (a) > 0 for each a ∈ {0, 1}.

The estimand of interest is the Average Treatment
To estimate the ATE, the experimenter conducts a two-wave experiment. The smaller first wave, also known as the pilot, and used to inform the experimenter about aspects of the design of the larger main wave (i.e. the second wave). The following assumptions about the two experimental waves will be maintained throughout the paper. For notational clarity, the tilde symbol (e.g. X) refers to quantities associated with the pilot.
draws from the distribution of the random vector (Y (0), Y (1)) . Treatment is randomly assigned in the pilot so that denoting assignments by , are n i.i.d. draws from the distribution of the random vector (Y (0), Y (1)) and are independent to potential outcomes and treatment assignments in the pilot. That is, The experimenter has access to an exogenous randomization device for assigning treatments in the main wave. That is, there is an i.i.d. sample of Uniform[0, 1] random variables, {U i } n i=1 , available to the researcher and satisfying We describe a number of allocation schemes that the experimenter could implement in the main wave using the randomization devices {U i } n i=1 .
• Simple random assignment: For a given p ∈ (0, 1), let The associated observed outcomes in this case are denoted and the estimator for the average treatment effect is the difference-in-means estimator: .
Balanced randomization corresponds to choosing p = 1 2 . We will refer to the associated treatment assignment rule as the balanced allocation.
• The Infeasible Neyman Allocation: For any given p ∈ (0, 1), elementary arguments show The optimal choice of p to minimize the variance of the limiting distribution is the Neyman Allocation: The optimal treatment scheme, associated observed outcomes and difference-in-means estimator are denoted by . (2.1) Implementing the Neyman Allocation requires knowledge of the quantities σ(0), σ(1) and as such is infeasible.
• The Feasible Neyman Allocation (FNA): One feasible implementation is to use the pilot data to form a plug-in estimator for p * . We start by using pilot data to estimate potential outcome variances: The latter case in (2.2) (where at least one σ m (a) = 0) avoids division by zero and additionally avoids the case where the entire main wave sample gets assigned to a single treatment arm.
If the potential outcomes are continuously distributed, this latter case happens with zero probability. The distribution of p depends on m, but we omit the sample size subscript for notational convenience. The associated treatment allocation scheme, observed outcomes and difference-in-means estimator are denoted by

Toy Example
In this section, we illustrate the main problems with the FNA in small samples with a toy model.
In the context of this simple example, we ask the question: when does the FNA do worse than the balanced allocation? It turns out that this happens for a range of plausible values of population parameters. Section 4 describes the extension of our findings into more general settings.
We assume in this section that the potential outcomes are bivariate normal: Suppose we have a pilot of size m where m is even. Suppose treatment is assigned deterministically as follows: As such, treatment assignment is strongly balanced in the pilot. Using standard arguments, it follows that the variance estimates are distributed as independent χ 2 random variables: Conditional on the pilot sample, the infeasible Neyman Allocation assigns proportion of the units in the main to treatment. Suppose for simplicity again that we assign the first n p units to treatment and the remainder to control (rounding n p if necessary). Then, Terms in the above expression have conditional distributions: Furthermore, they are independent. Hence, Taking expectation over p, we have that: Letting p be constant at p, we obtain the variance of the difference-in-means estimator under simple random assignment as a special case: Our goal is then to compare the two expressions above when we set p = 1 2 . First, note that we can rewrite (3.1) as The Neyman Allocation does worse than balanced randomization whenever the following obtains: The final implication uses the fact that Z m is reciprocally symmetric under bivariate normality and balanced randomization. By the quadratic formula, the above inequality satisfied if and only if Note that reciprocal symmetry together with Jensen's inequality guarantees that the discriminant is strictly positive: Furthermore, we have that In other words, the interval has the form where c m > 1 is the upper bound in (3.2). Hence, there is a range of parameter values under which the FNA does strictly worse than balanced randomization. We first note that 1 ∈ C m for all m.
This is intuitive since p = 1 2 is the infeasible Neyman Allocation when σ(1)/σ(0) = 1. Secondly, x ∈ C m ⇔ 1/x ∈ C m . That is, the relative performance of the FNA to the balanced allocation does not change when we relabel treatment and control.

Simulation Evidence
Given an underlying distribution, it is simple to compute C m by Monte Carlo integration. In this subsection, we present the values of C m for some simple models and argue that for plausible values of σ(1)/σ(0), the FNA performs worse than the balanced allocation.  (1)) and Y (0) ∼ N (µ(0), σ 2 (0)). Monte Carlo integration using 10,000 draws.
We start with the toy model, where Y (1) ∼ N (µ(1), σ 2 (1)) and Y (0) ∼ N (µ(0), σ 2 (0)). Results are shown in Figure 1 While the intervals may appear rather narrow at first glance, we provide numerous examples in Section 5 in which the amount of heteroskedasticity falls within this range.
Monte Carlo integration using 10,000 draws. As our toy model shows, it is the estimation of σ(1) and σ(0) that causes problems when m is small. It is therefore intuitive that C m will be wide when the distributions are fat tailed, that is, when kurtosis is high. As such, we consider the following parametrization: Y (1) ∼ µ(1) + σ(1) · Pareto(l, s) and Y (0) ∼ µ(0)+σ(0)·Pareto(l, s). Here, l and s are the location and scale parameters respectively. Figure 3 plots C m for l = 1 and s ∈ {2, 3, 4}. Indeed, we see that the bands are much wider than in the previous models. For s = 2, even when m = 100, C m = [0.40, 2.51], which, given the examples in Section 5 appear to be fairly extreme amounts of heteroskedasticity. Finally, we note that, C m decreases in width as we move from s = 2 to s = 4.
In sum, we see that the FNA does worse than the balanced allocation when the treatment and control groups are relatively homoskedastic. Furthermore, C m can be quite large when m is small. The small pilot problem is exacerbated when observations exhibit cluster dependence, so that the "effective observations" are fewer in number. Small pilot issues can also arise when researchers perform stratified randomization with many strata, so that each stratum ends up with few observations. In Section 5, we argue that many empirical applications in fact have fairly homoskedastic outcomes. As such, unless a researcher has reason to believe that their outcomes are highly heteroskedastic, they should exercise caution in using the FNA with small pilots.

Theoretical Results
In this section, we study the theoretical properties of the FNA. We first review the case where n, m → ∞. Next, we study the FNA in our asymptotic framework where n → ∞ but m is fixed, highlighting implications of our results for empirical researchers. (1, s) and s ∈ {2, 3, 4}. Monte Carlo integration using 10,000 draws.

Large-m asymptotics
Consider an asymptotic regime where both m, n → ∞, corresponding to situations where both the pilot and the main wave are large. It is straightforward to show that p p → p * . Furthermore, we have that: In other words, θ p and θ p * have the same limiting distribution after suitable centering and scaling. This occurs because in the limit, noise coming from estimating p * with p is negligible in comparison to the sampling error of the difference-in-means estimator. Researchers employing the large m framework essentially assume that p is an arbitrarily good estimator for p * so that its sampling error can be ignored. In practice, error in p can be large, particularly when m is small.
Asymptotic approximations that do not take this into account will likely perform poorly in finite sample.

Fixed-m Asymptotics
To better understand the behavior of the FNA under small pilots, we study its properties in a novel asymptotic framework that takes m to be fixed, even as n → ∞.
A growing literature in econometrics uses fixed-sample asymptotics to study settings in which the "effective sample size" is small. For example, when data exhibit cluster-dependence, settings with few clusters pose unique challenges for estimation and inference. To tailor their analyses to these problems, papers such as Ibragimov and Müller (2010), Canay, Romano, and Shaikh (2017) and Canay, Santos, and Shaikh (2021)  Our approach is similar in spirit to these papers. In keeping m fixed, we have that p is a noisy estimate of p * even in the limit. Preserving this important feature of the statistical problem makes our framework more appropriate for analyzing experiments with small pilots. In this setting, θ p converges in distribution to a mixture of Gaussians instead of N (0, Σ * ). Furthermore, the form of the limiting mixture distribution depends on the distribution of p. This is the content of the following proposition: where L m is a random variable whose distribution takes the form To see the intuition for our result, recall that for each p ∈ (0, 1), When m is held fixed, in the limit as n → ∞, p remains a non-degenerate random variable. In particular, the limiting distribution of p is its finite-sample distribution. Thus, the distribution of √ n θ p − θ becomes a mixture of the marginal distributions of the process where the mixing distribution is the distribution of p, denoted here by G m .

Implication for Experiments
Our results have implications for the use of the FNA when pilot sizes are small. In this sub-section we compare the asymptotic variances of θ p , θ p * as well as θ p with p = 1 2 . Since these estimators all have limit distributions that have mean zero (after centering around θ and scaling by √ n), comparing asymptotic mean squared errors is equivalent to comparing the variances of the limit distributions. We first show that θ p has larger asymptotic variance in the fixed-m regime than in the large-m regime. We next show that under reasonable ranges of parameter values, the asymptotic variance of θ p can exceed that of θ p with p = 1 2 .
Remark 4.2. By Proposition 4.2 and Remark 4.1, θ p has a limit distribution which has mean zero.
Given that θ p * and θ p also have mean zero normal limit distributions, comparing asymptotic mean squared errors is equivalent to comparing the variances of the limit distributions.
We begin with the corollary: In words, the asymptotic variance of θ p is larger under the fixed-m regime than under the largem regime. When pilots are small, uncertainty in p may be large and could affect the asymptotic variance of θ p . In particular, θ p will not be able to attain the optimal asymptotic variance of the infeasible allocation p * . Conventional large-m asymptotics may be too optimistic about the effectiveness of the Neyman Allocation with small pilots. We expect our analysis to better capture the behavior of θ p when pilots are small.
In addition to not attaining Σ * , the θ p can do worse than θ p for certain values of σ 2 (1) and σ 2 (0), as our next two results asserts. For convenience, define the following: Definition 4.1. Let C m be the set such that for a pilot of size m, θ p has higher asymptotic variance For our first result, we characterize the region C m in terms of B m . Furthermore, The properties of C m are intuitive. Firstly, x ∈ C m implies that 1/x ∈ C m . That is, the relative performance of the FNA to the balanced allocation does not change when we relabel treatment and control. Secondly, 1 ∈ C m . This is because when σ(1)/σ(0) = 1, the balanced allocation is optimal.
Finally, note that |C m | depends on the bias of σ m (1)/ σ m (0) and σ m (0)/ σ m (1). In particular, if both terms are unbiased, B m = 1 and |C m | = 0. However, |C m | is strictly positive as long as p is not degenerate.
The exact properties of C m depends on the underlying distributions of potential outcomes. To understand its behavior in a more general setting, we study its first-order approximation. This yields the following: Provided that the potential outcomes are sub-Gaussian, the relative efficiency of θ p and θ p under p = 1/2 is, to a first order, determined by the kurtosis of Y i (1) and Y i (0). Intuitively, if the potential outcomes have fatter tails, p will be poorly estimated, leading to larger variance in θ p .
Furthermore, |C m | is shrinking to 0 at the rate 1/ √ m. Letting m → ∞, θ p has weakly lower asymptotic variance across all parameter values. Hence, we recover the classic result concerning the optimality of the Neyman Allocation. When m is small, however, C m can be wide, as Proposition 4.4 suggests. As we will argue in Section 5, many empirical applications have σ(1)/σ(0) close to 1, so that for small m, they fall within the range in which balanced randomization is preferred.
While the sub-Gaussian assumption limits the applicability of Proposition 4.5, it covers binary and bounded outcomes, which are relevant in empirical work. Furthermore, we consider it to be a negative result: even when potential outcomes are well-behaved, the FNA is sensitive to the kurtosis of the potential outcomes. It can still perform poorly relative to the balanced allocation as a result.

Remark 4.3.
Using an argument based on Taylor expansions, we can weaken the sub-Gaussian assumption to finiteness of the first 14 th moments. The proof is available on request.

Empirical Evidence of Approximate Homoskedasticity
To assess the amount of heteroskedasticity that empirical researchers face, we revisit the first 10 completed experiments in the AER RCT Registry. In each experiment, we ask the following question: Suppose the authors had access to a small pilot prior to the main study, would they have done better using the FNA instead of balanced randomization? In each experiment presented, we use the full experimental sample to estimate the standard deviations of each treatment arm and compute the corresponding ratios to see if these are close to one. In practice, researchers cannot do this given a small pilot since they do not have access to consistent variance estimators. Our

Avvisati et al. (2014)
There is a significant body of research examining the impact of school-level factors such as class size or teacher quality on educational performance of students. These are typically seen as the primary instruments for educational policy intervention. A large body of work also examines the impact of parental inputs on educational outcomes. Avvisati et al. (2014) Table 1 reports student-level standard deviations in treatment and control groups, as well as their ratios, for the main outcomes of interest. These are the outcomes considered in Tables   2, 3 and 5 in Avvisati et al. (2014). The ratios are all close to one, so that by and large the treatment and control groups are relatively homoskedastic at the student level. This indicates that if the experimenters were to run a randomized control trial in the same population with treatment assigned at the individual-level, the FNA would likely yield no improvement relative to the balanced allocation. Table 2 reports standard deviations and their ratios in class-level means of outcomes. This corresponds to a scenario in which classes are the units of interest, with class-level means as the relevant outcomes. We first calculate class-level means and then compute their standard deviations across classes for the treatment and control groups respectively. The standard deviation ratios for class-level means are by and large also close to one. Hence, if the hypothetical experiment was to be conducted at the class-level, the treatment and control groups would still be relatively homoskedastic. In this case, the FNA would again not improve upon the balanced allocation.  Partnering with a rural bank in Mindanao, they authors surveyed 1,777 of their existing or former clients, of which 842 were placed into the treatment group, while 469 were placed in the control group. As treatment involved receiving a briefing on the importance of savings, the remaining 466 individuals were placed in the marketing group, receiving the briefing but not access to SEED. We focus on the Table VI of Ashraf, Karlan, and Yin (2006), containing results on saving behavior.
The parameter of interest is the Intent-to-Treat effect, with approximately 25% of the treated taking up treatment. Here, the authors find that relative to the control group, the treatment group had a higher change in savings 6 months (6m) and 12 months (12m) after treatment. Comparing treatment to marketing group led to weaker but still positive results.
We present standard deviations of the outcomes as well as their ratios in Table 3. We first note that Change in Balance > 0% (12m) and Change in Balance > 20% (12m) are binary variables which are relatively homoskedastic. Change in Total Balance (6m) and Change in Total Balance (12m), measured in Philippine pesos, exhibit more heteroskedasticity. In particular, comparing the treatment group to the marketing group at the 12 month period, we observe a standard deviation ratio of 3.13. At first glance, this suggests that the FNA might outperform balanced randomization, at least with respect to this specific outcome. This turns out to be false once we investigate the heteroskedasticity in Change in Total Balance (6m) and Change in Total Balance (12m). Quantiles of these variables are displayed in Table 4. Clearly, they have extremely fat right tails, which we confirm by computing the kurtosis, contained in Table 5. Fat tails worsen the performance of a variety of statistical techniques, including the FNA, as our analysis in Section 4 shows.
-2000 -1200 -800 -100 100 900 6500 8100 34300 Researchers cannot estimate C m if the only have access to a small pilot. However, using the full experiment data, we are able to estimate C m for a range of m. Our results are displayed in Figure 4. Given the high kurtosis, there is a relatively large range of ratios of standard deviations for which the balanced allocation is preferred to the FNA. We can further compute the m necessary before the FNA outperforms the balanced allocation. Our results are collected in Table 6. The first two rows, labelled "Exact", refer to intervals which are computed using the full experiment.
Here, we see that the necessary pilot sizes are between 25-50% of the full experiment. For the comparison of Treatment and Marketing group at 6 months, the necessary pilot size exceeds 2, 000, falling outside the set of grid points we explored. To complete the analysis, we use our asymptotic result to obtain approximations of the necessary m. Comparing the asymptotic intervals to the exact ones, we see that the former is far too optimistic for the fat-tailed DGP in Ashraf, Karlan, and Yin (2006). This is likely because the sub-Gaussian assumption is inappropriate for this data.
Nonetheless, applying the asymptotic bounds to the case of Treatment vs Marketing group at 6 months, we find that a pilot size of 7,000 would be necessary for the FNA to outperform balanced randomization. This is nearly 4 times the size of the actual experiment. In sum, our analysis suggests that the FNA would have performed poorly in the context of Ashraf, Karlan, and Yin (2006). Even though the outcomes of interest exhibit stronger het- eroskedasticity, the fat tails of the outcome distributions also impedes the estimation of p, so that ultimately, very large pilot sizes are needed for the FNA to outperform the balanced allocation in this example.

Conclusion
We study the properties of the Feasible Neyman Allocation (FNA) in a novel asymptotic framework for two-wave experiments that takes pilot size to be fixed as the size of the main wave tends to infinity. In this setting, the estimated allocation has error that is not negligible even in the limit. Our asymptotic model therefore corresponds more closely to the finite sample statistical problem when pilots are small and the optimal allocation may be poorly estimated. We establish the limiting distribution of the difference-in-means estimator under the FNA and characterize conditions under which it has larger asymptotic variance compared to balanced randomization, where half of the main wave is assigned to treatment and the remainder to control. This happens when the potential outcomes are relatively homoskedastic with respect to treatment status or exhibit high kurtosis -situations that may arise frequently in practice. This issue is likely to be exacerbated when observations exhibit cluster dependence or when researchers perform stratified randomization with many strata, so that the "effective observations" used for variance computations are fewer in number. Our results suggest that researchers should not use the FNA with small pilots, particularly when they believe any of the above occurs.

References
Consider the partial-sums process where B(·) is a two-dimensional standard Brownian motion and Σ where the covariance matrix is derived using the fact that a standard Brownian motion has independent increments. By (A.6), A p a.s.
→ p * and Slutsky's Theorem, as m, n → ∞, We should note that measurability of U is derived in Lemma A.6. Using (A.5) in combination with The distribution of L m can be derived as a corollary of Lemma A.6, but we include the direct derivation here for completeness. To derive the distribution of L m in closed form, notice that By the Law of Total Probability, it follows that where G m (·) is the distribution of p.

A.1.4 Auxiliary lemmas
Lemma A.1. Suppose Assumption 2.2 holds and additionally that, there is some p 1 ∈ (0, 1) such Proof. Let a ∈ {0, 1} be given. Note that for any Borel function h : The hypothesis of the question gives us that 1 Setting h equal to the maps y → y and y → y 2 , we get respectively that Combining these with the Continuous Mapping Theorem again, it follows that as m → ∞ By another application of the Continuous Mapping Theorem, → p as n → ∞. Next, we consider the case with both m, n → ∞. Using the triangle inequality, Proof. We can rewrite θ p using the potential outcomes as is standard: Thus, Note that the distribution √ n θ p − θ is invariant to permutation of the sample indices. Hence the distribution of √ n θ p − θ does not change if we reorder the sample indices to have i = 1, . . . , nA p correspond to the observations in the treatment group and i = nA p + 1, . . . , n correspond to the observations in the control group. Notice also, that under Assumption 2.2, Assumption 2.3, and (A.9), the resulting permuted sum has the same distribution as Proof. Recall that we define the following in the preamble to ?THM? ??
By (2.3), we have that for each i ∈ N, By the Continuous mapping theorem, the proof is completed if we show both Similarly to the arguments in the proof of Lemma A.2, conditional on p, Proof. We prove this for k = 1, since the extension to k ∈ N follows in similar fashion with more complicated notation. Let ε > 0 be given. Note that Since f is continuous, and [0, 1] is a compact set, f is uniformly continuous. Hence, there is δ f,ε > 0 it follows that sup z∈[0,1] |h(z) − f (z)| < ε and |x − y| < δ f,ε so that |g(y, h) − g(x, f )| < 2ε. are independent, then the distribution of H is a mixture defined by where P U is the pushforward measure of U against P.
Proof. Notice that H is defined by the evaluation functional g as defined in Lemma A.5, since H(ω) = B(U (ω), ω) = g(U (ω), B(·, ω)). Since g is a measurable function against the product The final claim then follows from the Law of Total Probability.

A.2 Proof of Corollary 4.3
Let L m denote a random variable whose distribution is given by G m (dp) Using the Law of Iterated Expectations, it can be shown that L m has mean zero and variance given by Next, note that for all p ∈ [0, 1], p = p * , Since σ 2 (0) > 0 and σ 2 (1) > 0, P ( p = p * ) > 0. That is, By the strict monotonicity of expectation, we are done.

A.3 Proof of Theorem 4.4
By our earlier derivation, θ p has lower asymptotic variance than θ p if and only if Define the variable Z m : We can then rewrite the above condition as By the quadratic formula, the above inequality is satisfied whenever Next note that on R + , f (x) = 1 x + x is strictly convex and attains its strict minimum at x = 1. Since Z m is non-degenerate, Jensen's inequality yields Hence, |C m | > 0. We can then write:

A.4 Proof of Theorem 4.5
We start by evaluating the asymptotic distributions of Z m . First, By the Delta Method, Similarly, Since the above two displays contain independent random variables, another application of the Delta Method yields that Next, let f (x) = x + 1/x. Note that By the second order Delta Method, Since the left hand side is an analytic function of sub-Gaussian random variables, all moments can be bounded uniformly in m. Conclude that:

B Additional Empirical Examples
We Since the authors are interested in persistence of treatment effects, they measure these outcomes at the following times after intervention: 0-3 months (Endline 1), 6-9 months (Endline 2) and 12-15

months (Endline 3). They find that the Math intervention has positive effects on Non-Symbolic
Mathematical skills across all three Endlines, while Symbolic Mathematical skills only improves in Endline 1. Table 7 displays the standard deviation of each outcome variable by treatment arm, computed at the individual level. In the absence of correlation among students in the same pre-school, and assuming that treatment is assigned at the individual level, the numbers shown are the relevant empirical counterparts to σ(1) and σ(0). We see that the outcomes are relatively homoskedastic across the outcome variables and across time. The ratio σ(1)σ(0) falls between 0.94 and 1.31, suggesting that naive experiment will do well when pilots are small.
Suppose we are concerned about correlation across students in the same pre-school. We can redefine our unit of observation to be the school by taking averages across students in the same school. The Neyman Allocation then tells us how many schools to allocate to treatment. In this case, the standard deviation in the mean across schools is the relevant counterpart to σ(1) and σ(0). They are presented in Table 8. As before, we see that outcomes are relatively homoskedastic across schools, though less so than in the individual level case. Nonetheless, the ratio falls between 0.84 and 1.55. Once we consider schools to be the unit of treatment, however, the effective pilot size also shrinks, such that the drawbacks of the estimated Neyman Allocation may be even more pro-   The authors used the data to study a variety of outcomes. In this section, we focus on their first set of results, which concern healthcare utilization. In particular, we revisit the outcome variables used in Tables V and VI of Finkelstein et al. (2012), which are obtained from survey data (as opposed to administrative data), and are hence publicly available. Inline with the authors' results on the Intent-to-Treat effect, we define the treated group as those selected by the lottery. We note that the authors apply sampling weights to correct for differential response rates to the survey. We follow their weighting scheme in computing our results.
The standard deviation of individual-level outcome are presented in Table 9. Results taking household to be the unit of observation are presented in Table 10. Across both tables, we see that that the standard deviations in outcomes are remarkably similar across treatment and control groups. They are also very similar across individual and household level groups, since households with more than one person represented less than 5% of the survey sample.  determined to be of medium quality, were entered into a lottery, from which 729 were selected for the award. Three rounds of surveys were then conducted at 1,2 and 3 years after the application respectively.
We focus on the first set of results in McKenzie (2017) -presented in Table 2 -concerning the effect of the award on start-up and survival. Here, they find that the grant persistently increased the probability that the entrepreneur was operating a firm, as well as the number of hours they spent in self-employment. We present the standard deviations of these outcome variables in Table 11. Here we see that hours in self employment is roughly homoskedastic across all three periods. However, the outcome on whether the entrepreneur is operating a firm is arguably highly heteroskedastic, with standard deviations that is as small as 0.44 that of the control group. As in our earlier example, we find high kurtosis in these outcome variables, displayed in Table 12.
We do not estimate C m in this example because almost all entrepreneurs in the treated group operate their own firms in the sample. As such, a small random sub-sample (e.g. of size below 200) from this group has variance 0 with high probability, impeding the estimation of C m . These pathological cases are revealing. In a pilot, if the treated group has variance 0 in the outcome, the estimated Neyman Allocation assigns 0 units to treatment in the full experiment. The high probability of such an "extreme" outcome with small pilots is precisely the danger which we are warning against. We conclude that the estimated Neyman from a small pilot will likely lead to adverse results given the DGP in McKenzie (2017).  B. 4 Chong et al. (2015) Partnering with the Peruvian nongovernmental organization PRISMA, Chong et al. (2015) conducted two RCTs to investigate the efficacy of various interventions in encouraging recycling. First, the Participation Study considers the following 9 different messaging strategies and their relative success in enrolling members into recycling programs: Out of a total of 6,718 households, approximately 600 were assigned to each treatment arm, with the exception of Signal: Local, which were assigned 932 participants. 1,157 households were assigned to the control group. Three measures of participation were considered: 1. "Participates any time" is an indicator that takes the value 1 if a household turned in residuals over the course of the study.
2. "Participation Ratio" is the number of times a household turns in residual over the total number of opportunities they had to turn in residuals.
3. "Participates during either of last two visits" is an indicator that takes value 1 if the household turned in residual during one of the last two canvassing weeks.
The results in Table 3 of Chong et al. (2015) shows that messaging had no effect in increasing participation in the program.    Table 4A that bin provision was highly effective in increasing recycling, though SMS reminders had no effect. We compute the standard deviation in each of their outcome variables by treatment type in Table 15. Also displayed is the ratio of these standard deviation with respect to the control group. Again, we see strong evidence of homoskedasticity, suggesting that the naive experiment will perform well in this scenario.  Bloom et al. (2014) study the effect of working from home on employees' productivity via a randomized experiment at Ctrip, a NASDAQ-listed Chinese travel agency with 16000 employees. The main concern is whether or not working from leads to shirking. Ctrip decided to run a nine-month experiment on working from home. They asked the 996 employees in the airfare and hotel departments of the Shanghai call center if they were interested in working from home four days a week and one day in the office. 503 of these employees were interested and of these, 249 were qualified to take part on the basis of tenure, broadband access and access to private work space at home.
Qualified employees were assigned to working from home if they had even-numbered birthdays so that those with odd-numbered birthdays formed the control group. The treatment and control groups were comprised of 131 and 118 employees respectively. The only difference between the two 1. The authors also consider providing bins with and without instructions as well as generic vs personalized SMSes. However, these finer definitions leads to treatment arms with fewer 50 households. Hence we focus on the coarser definition of the treatments, as employed in panel 4A. groups was location of work -both groups used the same equipment, faced the same workload and were compensated under the same pay system. The authors find a 13% increase in productivity of which the main source of improvement was a 9% increase in the number of minutes worked during a shift. The remaining 4% came from an increase in the number of calls per minute worked. Table 16 reports standard deviations for treatment and control groups in the experiment as well as the standard deviation ratios for the main outcomes of interest in Bloom et al. (2014). Strong evidence of relative homoskedasticity with respect to treatment status presents in this study as well -suggesting that the balanced allocation would outperform the FNA.  randomly assigned sector and selectivity of (degree-awarding) institutions. Fictitious resumes are created using a vast online database of actual resumes of job seekers, with applicant characteristics varying randomly (i.e. characteristics are randomly assigned). Outcomes are callback rates. There are three main comparisons in the paper: • for-profit vs. public institutions, • for-profits that are online vs. brick-and-mortar (with a local presence), • more selective vs. less selective public institutions. Deming et al. (2016) find that BA degrees in business from large online for-profit institutions are more 22% less likely to receive a callback than applicants with similar degrees from non-selective public schools when the job vacancy requires a BA. When a business job opening does not list a BA requirement, they find no significant overall advantage to having a post-secondary degree. For health jobs, resumes with certificates from for-profit institutions are 57% less likely to receive a callback than those with similar certificates from public institutions when the job listing does not require a post-secondary certificate. No significant difference in callback rates are found when the health job listing requires a certificate. Table 17 reports standard deviations across treatment arms across the various sub-populations of interest in Deming et al. (2016). Since in most sub-populations, there are more than two treatment arms, we do not report ratios. However, pairwise comparisons between treatment arms within any chosen subpopulation shows strong evidence of relative homoskedasticity across the board.  (2015) Bryan, Karlan, and Zinman (2015) conduct a field experiment to study efficacy of peer intermediation in mitigating adverse selection and moral hazard in credit markets. To identify the effects of peer screening and enforcement, they use a two-stage referral incentive field experiment. The experiment was conducted through Opportunity Finance South Africa (Opportunity), a for-profit lender in the consumer micro-loan market. Over the period of February 2008 through July 2009, Opportunity offered individuals approved for a loan the option to participate in its "Refer-A-Friend" program. Referred individuals earned R40 if they brought in a referral card and were approved for a loan. The referrer could earn R100 for referring someone who was subsequently approved for and/or repaid the loan, depending on the referrer's incentive contract. Referrers were randomly assigned to one of two ex-ante incentive contracts: • Approval incentives: the referrer would be paid only if the referred was approved for a loan.
• Repayment incentive: the referrer would be paid only if the referred successfully repaid the loan.
Among referrers whose referred friends were approved for a loan, Opportunity randomly selected half to be surprised with an ex-post incentive change: • Half among the ex-ante approval group were phoned and told that in addition to the R100 approval bonus, they would receive an additional R100 if the loan was successfully repaid by the referrer.
• Half among the ex-ante repayment group were phoned and told that they would receive the R100 now, and that receipt of the bonus would no longer be conditional on repayment of the loan by the referrer.
The overall incentive structure is as follows • Ex-ante and ex-post approval (EA = A): no enforcement or screening incentive.
• Ex-ante repayment and ex-post repayment (EA = R, EP = R): Enforcement and screening incentive.
The authors find no evidence of screening but do find large enforcement effects.   were further assigned to four treatment arms termed G1, G2, G3, G4. G1 received CCT's in education and health. G2 received CCT's in addition to direct investment in education and health centers. G3 received only direct investments and finally, G4 served as the control group and received no interventions. The 70 municipalities were further divided into 5 strata each consisting of 14 municipalities on the basis of quintiles of mean height-for-age. Random assignment was performed within these strata (stratified randomization). The final sample consisted of 20 municipalities in G1, 20 in G2, 10 in G3, and 20 in G4. The authors match the experimental data with census data and use the latter to construct the outcomes of interest which are three dummy variables.
The first is an indicator for whether a child is enrolled in and attending school during the time of the census. The second indicates whether the child worked during the week prior to the census or conditional on a negative response to the former, whether they reported non-wage employment in a family farm or business. The third indicates whether the child worked exclusively on household chores. The authors find that overall, children eligible for CCTs were 8% more likely to enroll in school and 3% less likely to work. Table 19 reports standard deviations for the outcomes of interest across treatment arms. Again, for a given outcome, pairwise comparisons across arms yields evidence of relative homoskedasticity with respect to treatment status. We conclude that in this case, the balanced allocation would outperform the FNA.