{'fraction_non_alphanumeric': 0.04446737651859843, 'fraction_numerical': 0.01941929734040058, 'mean_word_length': 4.106586826347305, 'pattern_counts': {'":': 0, '<': 4, '<?xml version=': 0, '>': 9, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Adaptive Estimation of Quadratic Functionals in Nonparametric Instrumental Variable Models

This paper considers adaptive estimation of quadratic functionals in the nonparametric instrumental variables (NPIV) models. Minimax estimation of a quadratic functional of a NPIV is an important problem in optimal estimation of a nonlinear functional of an ill-posed inverse regression with an unknown operator using one random sample. We first show that a leave-one-out, sieve NPIV estimator of the quadratic functional proposed by Breunig and Chen [2020] attains a convergence rate that coincides with the lower bound previously derived by Chen and Christensen [2018]. The minimax rate is achieved by the optimal choice of a key tuning parameter (sieve dimension) that depends on unknown NPIV model features. We next propose a data driven choice of the tuning parameter based on Lepski’s method. The adaptive estimator attains the minimax optimal rate in the severely ill-posed case and in the regular, mildly ill-posed case, but up to a multiplicative √ log n in the irregular, mildly ill-posed case.


Introduction
Long before the recent popularity of instrumental variables in modern machine learning causal inference and biostatistics, the instrumental variables technique has been widely used in economics. For instance, instrumental variables regressions are frequently used to account for omitted variables, mis-measured regressors, endogeneity in simultaneous equations and other complex situations in observational data. In economics and other social sciences, as well as in medical research, it is very difficult to estimate causal effects using observational data sets alone. When treatment assignment is not randomized, it is generally impossible to discern between the causal effect of treatments and spurious correlations that are induced by unobserved factors. Instrumental variables are commonly used to provide exogenous variation that is associated with the treatment status, but not with the outcome variable (beyond its direct effect on the treatments).
To avoid mis-specification of parametric functional forms, the nonparametric instrumental variables regressions (NPIV) have gained popularity in econometrics and modern causal inference in statistics and machine learning. The simplest NPIV model assumes that a random sample {(Y i , X i , W i )} n i=1 is drawn from a joint distribution of (Y, X, W ) satisfying where the conditional density f X|W (and hence the conditional expectation operator T ) is unknown. Under mild conditions, the conditional density f X|W is continuous and the operator T is compact. This makes the estimation of h 0 an ill-posed inverse problem with an unknown operator T . See, for example, Newey and Powell [2003], Carrasco et al. [2007], Blundell et al. [2007] and Horowitz [2011].
This paper considers minimax rate-optimal estimation of a quadratic functional of the h 0 in the NPIV model (1.1): for a known non-negative weighting function µ, which is assumed to be uniformly bounded from below on its support. Although Chen and Pouzo [2015] and Chen and Christensen [2018] already studied plug-in sieve NPIV estimation and inference on nonlinear functionals of h 0 , there is no results on the minimax rate optimal estimation of any nonlinear functionals of h 0 yet. Since a quadratic functional is a leading example of a nonlinear functional and is also widely used in goodness-of-fit testing on h 0 , this paper focuses on optimal estimation of the quadratic functional f (h 0 ) in the NPIV model. In this paper, we first analyze a simple leave-one-out, sieve NPIV estimator f J for the quadratic functional f (h 0 ) recently proposed by Breunig and Chen [2020]. We establish an upper bound on the convergence rate for this estimator, and show that it coincides with the lower bound previously derived by Chen and Christensen [2018].
Thus, the leave-one-out, sieve NPIV estimator f J is minimax rate-optimal for f (h 0 ). Depending on the smoothing properties of the conditional expectation operator T , we distinguish between the mildly and severely ill-posed cases. For the severely illposed case, the minimax optimal convergence rate for estimating f (h 0 ) is of (log n) −a order, where a > 0 depends on the smoothness of the NPIV function h 0 and the degree of ill-posedness. For the mildly ill-posed case, the optimal convergence rate for estimating f (h 0 ) exhibits the so-called elbow phenomena: the rate is of order n −1/2 for the regular mildly ill-posed case; the rate is of order n −b , where 0 < b < 1/2 depends on the smoothness of h 0 , the dimension of X and the degree of ill-posedness. As already noticed in Chen and Christensen [2018], the minimax convergence rate for the severely ill-posed case is so slow that even a plug-in sieve NPIV estimator f ( h) of f (h 0 ) achieves it, where h is the sieve NPIV estimator of h 0 . However, the plug-in estimator f ( h) fails to achieve the minimax rate for the mildly ill-posed case.
Nevertheless, we show that the simple leave-one-out sieve NPIV estimator f J of f (h 0 ) is minimax rate-optimal regardless the degree of ill-posedness.
The minimax optimal rate is achieved by the optimal choice of a key tuning parameter (sieve dimension) J that depends on the unknown smoothness of the NPIV function h 0 and the unknown degree of ill-posedness. We next propose a data driven choice of the tuning parameter (sieve dimension) J based on Lepski's principle. The adaptive, leave-one-out sieve NPIV estimator f J of f (h 0 ) is shown to attain the minimax optimal rate in the severely ill-posed case and in the regular, mildly illposed case, but up to a multiplicative √ log n in the irregular, mildly ill-posed case. We note that even for adaptive estimation of a quadratic functional of a unknown density or a regression function from a random sample, Efromovich and Low [1996] shown that the extra √ log n factor is the necessary price to pay for adaptation to the unknown smoothness of the function. Previously for the nonparametric estimation of the NPIV function h 0 in the model (1.1), Horowitz [2014] considers adaptive estimation of h 0 in L 2 norm using a model selection procedure. Breunig and Johannes [2016] consider adaptive estimation of a linear functional of h 0 in root-mean squared error metric using a combined model selection and Lepski method. These papers obtain adaptive rate of convergence up to a factor of log(n) (of the minimax optimal rate) in both severely ill-posed and mildly ill-posed cases. Chen and Christensen [2015] propose adaptive estimation of h 0 in L ∞ norm using Lepski method, and show that their data-driven procedure attains the minimax optimal rate in sup-norm and thus fully adaptive. Our data-driven choice of the sieve dimension is closest to that of Chen and Christensen [2015], which might explain why we also obtain minimax optimal adaptivity for quadratic functionals of a NPIV function h 0 .
Minimax rate optimal estimation of a quadratic functional in density and regression settings has a long history in statistics. The elbow phenomenon was perhaps first discovered by Bickel and Ritov [1988] in estimation of the integrated square of a density. An analogous result was established by Donoho and Nussbaum [1990] in estimation of quadratic functionals in Gaussian sequence models. Efromovich and Low [1996], Laurent [1996] and Laurent and Massart [2000] establish that Pinsker type estimators of quadratic functionals can achieve the minimax optimal rates of convergence. For estimation of a integrated squared density, Giné and Nickl [2008] showed that a leave-one-out estimator attains the rate optimal and provided an adaptive estimator based on Lepski's method. Collier et al. [2017] consider minimax rateoptimal estimation of a quadratic functional under sparsity constraints. Minimax rate optimal estimation of a quadratic functional has also been analyzed in density deconvolutions and inverse regressions in Gaussian sequence models. See, for example, Butucea [2007], Butucea and Meziani [2011] and Kroll [2019]. As far as we know, there is no published work on minimax rate-optimal adaptive estimation of a quadratic functional in NPIV models yet.
The rest of the paper is organized as follows. Section 2 presents the leave-one-out sieve NPIV estimator of the quadratic functional f (h 0 ). It also derives the minimax optimal convergence rates for our estimator of f (h 0 ). Section 3 first presents a simple data-driven procedure of choosing tuning parameters based on Lepski's method. It then establishes the convergence rates of our adaptive estimator of the quadratic functional. Section 4 briefly concludes. All proofs can be found in the Appendices A-C.

Minimax Optimal Quadratic Functional Estimation
This section consists of three parts. Subsection 2.1 first recalls a lower bound result in Chen and Christensen [2018] for estimation of a quadratic functional of the NPIV function h 0 . Subsection 2.2 introduces a simple leave-one-out, sieve NPIV estimator of the quadratic functional f (h 0 ). Subsection 2.3 establishes the convergence rate of the proposed estimator, and shows that the rate coincides with the lower bound and hence is optimal.

The Lower Bound
We first introduce notation. For any random vector V with support V, we let If {a n } and {b n } are sequences of positive numbers, we use the notation a n b n if lim sup n→∞ a n /b n < ∞ and a n ∼ b n if a n b n and b n a n . Denote Let { ψ} j≥1 be an orthonormal basis and ψ j,k,G be a tensor-product CDV wavelet basis in L 2 µ . For simplicity we assume that the structural function h 0 belongs to the Sobolev ellipsoid Finally we let T : L 2 (X) → L 2 (W ) denote the conditional expectation operator given Condition LB (ii) specifies the smoothing properties of the conditional expectation operator relative to ψ j,k,G . This assumption is commonly imposed in the related literature, see also Chen and Reiß [2011] for an overview. As in the related literature, we distinguish between a mildly ill-posed case where ν(t) = t −ζ and a severely illposed case where ν(t) = exp(− 1 2 t ζ ) for some ζ > 0. Condition LB (iii) ensures identification of the nonlinear functional f (h 0 ). The next result was established by Chen and Christensen [2018, Theorem C.1] and hence, its proof is omitted.
for some constant c > 0, where inf f is the infimum over all estimators that depend on the sample size n and 1. Mildly ill-posed case: 2. Severely ill-posed case: According to Lemma 2.1, there is the so-called elbow phenomena in the mildly ill-posed case 2.1: the regular case with a parametric rate of n −1/2 when p > ζ + d/4; and the irregular case with a nonparametric rate when p ≤ ζ + d/4.

A Leave-one-out, Sieve NPIV Estimator
denote a random sample from the NPIV model (1.1). The sieve NPIV (or series 2SLS) estimator h of h 0 can be written in matrix form as where {ψ 1 , ..., ψ J } and {b 1 , ..., b K } are collections of basis functions of dimension J and K used for approximating h 0 and the instrument space, respectively. Based on many simulation results in Blundell et al. [2007] and the minimax sup-norm rate results in Chen and Christensen [2018], the crucial regularization parameter is the dimension J of the sieve space used to approximate unknown function h 0 . In this paper, we keep the relationship of J and K fixed, i.e., the function K(·) does not depend on the sample size and satisfies K(J) ≥ J for all J. As pointed out by Chen and Christensen [2018], although one could estimate f (h 0 ) by the plug-in sieve NPIV estimator f ( h), it fails to achieve the lower bound in Lemma 2.1.
Recently Breunig and Chen [2020] propose a leave-one-out, sieve NPIV estimator for the quadratic functional f (h 0 ) as follows: We will show that this simple leave-one-out estimator f J can achieve the lower bound for estimating f (h 0 ).

Rate of Convergence
We first define the sieve measure of ill-posedness which, roughly speaking, measures how much the conditional expectation operator T : Following Blundell et al. [2007] the sieve L 2 measure of ill-posedness is where Ψ J = clsp{ψ 1 , ..., ψ J } ⊂ L 2 (X) denotes the sieve spaces for the endogenous variables. We call a NPIV model (1.1) (i) mildly ill-posed if τ j ∼ j ζ/d for some ζ > 0; and (ii) severely ill-posed if τ j ∼ exp( 1 2 j ζ/d ) for some ζ > 0.
The next assumption captures conditions that we require to obtain our rate of convergence of the estimator.
In the following, we introduce the vector τ J = (τ 1 , . . . , τ J ) ′ where τ j > 0 for all 1 ≤ j ≤ J and τ j ≤ τ j ′ for all 1 ≤ j < j ′ ≤ J. For a r × c matrix A with r ≤ c and full row rank r we let A − l denote its left pseudoinverse, namely (A ′ A) − A ′ where ′ denotes transpose and − denotes generalized inverse. Below, · denotes the vector ℓ 2 norm when applied to vectors and the operator norm induced by the vector ℓ 2 norm when applied to matrices. Assumption 3 (i) is a mild condition on the approximation properties of the basis used for the instrument space. In fact, (Π K T − T )h L 2 (W ) = 0 for all h ∈ Ψ J when the basis functions for clsp{b 1 , ..., b K } and Ψ J form either a Riesz basis or an eigenfunction basis for the conditional expectation operator. Assumption 3 (ii) is the usual L 2 "stability condition" imposed in the NPIV literature (cf. Assumption 6 in Blundell et al. [2007]). Note that Assumption 3 (ii) is also automatically satisfied by Riesz bases. Assumption 4 is a modification of the sieve measure of ill-posedness and was used by Efromovich and Koltchinskii [2001]. Assumption 4 is also related to the extended link condition in Breunig and Johannes [2016] to establish optimal upper bounds in the context of minimax optimal estimation of linear functionals in NPIV.
The next result provides the rate of convergence of our estimator f J .

Severely ill-posed case: choosing
Theorem 2.1 provides an extension of Breunig and Chen [2020, Theorem F.1] under Assumption 4. Theorem 2.1 also provides concrete convergence rates of f J once J is chosen optimally in either the mildly ill-posed case or the severely ill-posed case. Within the mildly ill-posed case, depending on the smoothness of h 0 relatively to the dimension of X and the degree of mildly illposedness ζ, either the first or the second variance term is dominating which then leads to the so-called elbow phenomenon. In particular, the simple estimator f J of Breunig and Chen [2020] achieves the lower bound rate in Lemma 2.1 and is thus, minimax rate optimal.

Rate Adaptive Estimation
The minimax rate of convergence depends on the optimal choice of sieve dimension J, which depends on the unknown smoothness p of the true NPIV function h 0 and the unknown degree of ill-posedness. In this section we propose a data-driven choice of the sieve dimension J based on Lepski's method; see Lepski [1990], Lepski and approaching one, where J satisfies the rate restrictions imposed in the next assumption. We also make use of the notation ζ = ζ J .
Assumption 5 is a uniform version of the rate conditions imposed in Assumption 2 (i). The next result establishes an upper bound for the adaptive estimator f J .
Theorem 3.1. Let Assumptions 1-5 be satisfied. Then, we have with probability approaching one.
The proof of Theorem 3.1 is based on a Bernstein-type inequality for canonical U-statistics. Also note that Theorem 2.1 provides an upper bound for | f J 0 − f (h 0 )|. The following result illustrates the general upper bound for the mildly and severely ill-posed case.
2. Severely ill-posed case: Corollary 3.1 shows that our data-driven choice of tuning parameter can lead to fully adaptive rate-optimal estimation of f (h 0 ) for both the severely ill-posed case and the regular mildly ill-posed case, while it has to pay a price of extra √ log n factor for the irregular mildly ill-posed case. We note that when ζ = 0 in the mildly ill-posed case, the NPIV model (1.1) becomes the regression model with X = W . Thus our result is in agreement with the findings of Efromovich and Low [1996] who showed that one must pay a factor of √ log n penalty in adaptive estimation of a quadratic functional of unknown density and regression functions when p ≤ d/4.

Conclusion
In this paper we first show that the simple leave-one-out, sieve NPIV estimator of the quadratic functional proposed in Breunig and Chen [2020] is minimax rate optimal. We then propose an adaptive leave-one-out sieve NPIV estimator of the quadratic functional based on Lepski's method. We show that the adaptive estimator achieves the minimax optimal rate for the severely ill-posed case as well as for the regular mildly ill-posed case, while a multiplicative √ log n term is the price to pay for the irregular, mildly ill-posed NPIV problem.
In adaptive estimation of a nonparametric regression function it is known that Lepski method has the tendency of choosing small sieve dimension, and hence may not perform well in empirical work. We wish to point out that due to the ill-posedness of the NPIV model (1.1), the optimal sieve dimension for estimating f (h 0 ) is smaller than the optimal sieve dimension for estimating f (E[Y |X = ·]). Therefore, we suspect that our simple adaptive estimator of quadratic functional of a NPIV function will perform well in finite samples. Implementation of our data driven method still relies on choice of a calibration constant. To improve finite sample performance over the original Lepski method, Spokoiny and Vial [2009] offered a propagation approach, and Spokoiny and Willrich [2019] proposed a bootstrap calibration of Lepski's method. We believe that the bootstrap calibration will be helpful in the NPIV setting as well.
A leading nonlinear ill-posed inverse problems with unknown operator is the nonparametric quantile instrumental variables (NPQIV) model: Recently Dunker [2020] considers adaptive estimation of the NPQIV function h 0 in L 2 norm under an a stronger assumption that U and W are fully independent, without establishing minimax rate optimality. Previously Chen and Pouzo [2015] presented the estimation and inference on possibly nonlinear functionals of a NPQIV function h 0 . It will be interesting to study the minimax rate adaptive estimation of a quadratic functional in a NPQIV model.

A. Proofs of Results in Section 2
We define s J = s min (G by Assumption 3. In the following we let . For a r × c matrix A with r ≤ c and full row rank r we let A − l denote its left pseudoinverse, namely (A ′ A) − A ′ where ′ denotes transpose and − denotes generalized inverse. Thus, we can write Proof of Theorem 2.1. Proof of (2.3). Under Assumptions 1-3, we may apply Breunig and Chen [2020, Theorem F.1] which yields In the remainder of this proof we bound the second summand on the right hand side. Let us define ψ J = G −1/2 µ where we used again Assumption 4 in the last inequality. Consider the second term on the right hand side. We obtain by making use of Assumption 3 (ii).
We now distinguish between the two regularity cases of the result. First, consider the case p ≤ ζ + d/4, where the function j → j 2(ζ−p)/d+1/2 is increasing and consequently, we observe Moreover, using h 0 ∈ H 2 (p, L), i.e., j≥1 h 0 , ψ j 2 µ j 2p/d ≤ L, we obtain Finally, it remains to consider the case p > ζ + d/4. In this case, we have that and consequently, the second variance term satisfies O(n −1 ) which is the dominating rate and thus, completes the proof of the result. Proof of (2.5). The choice of which shows the result.
Proof of Theorem 3.1. Due to Theorem 3.2 of Chen and Christensen [2015] we may assume that J ≤ J max ≤ J on E * n . The definition J = min J∈ J J implies J ≤ J 0 on the set E * n and hence, we obtain Moreover, using h 0 ∈ H 2 (p, L), i.e., j≥1 h 0 , ψ j 2 µ j 2p/d ≤ L, we obtain ( log n/n) 8p/(4(p+ζ)+d) .
Finally, it remains to consider the case p > ζ + d/4, where as in the proof of Theorem 2.1 we have which is the dominating rate and thus, completes the proof of the result.
Proof of (3.4). In the severely ill-posed case, the definition of the oracle choice in (3.1) implies and ensures that the variance and squared bias term are of the same order. Specifically, we obtain increasing we obtain and finally which completes the proof.

C. Supplementary Lemmas
We first introduce additional notation. First we consider a U-statistic where Z i = (U i , W i ) and the kernel R 1 is given by where M i = {|U i | ≤ M n } with M n = ζ −1 √ n/(log J). Note that the kernel R 1 is a symmetric function such that E[R 1 (Z i , z)] = 0 for all z.
We also introduce the U-statistic U n,2 = 2 n(n − 1) where the kernel R 2 is given by We make use of the following exponential inequality established by Houdré and Reynaud-Bouret [2003].
We obtain P max Consider T 1 . We make use of U-statistics deviation results. To do so, consider Λ 1 , . . . , Λ 4 as given in Lemma C.1. From Lemma C.2 we we infer with u = 2 log J and M n = ζ −1 √ n/(log J) that for all J ≤ J we have Λ 1 √ u + Λ 2 u+Λ 3 u 3/2 + Λ 4 u 2 by the definition of c n (J). Consequently, Lemma C.1 with u = 2 log J yields ≤ 6 exp (log J) − 2(log J) where the last equation is due to the lower bound J ≥ J min = ⌊log log n⌋.
Consider T 2 . We first observe for all J ≤ J that For all J ≥ 1 such that s J = s min (G