{'fraction_non_alphanumeric': 0.03956641800272019, 'fraction_numerical': 0.013848246300952067, 'mean_word_length': 4.337439507259129, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 10, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Sequential Specification Tests to Choose a Model: A Change-Point Approach

Researchers faced with a sequence of candidate model specifications must often choose the best specification that does not violate a testable identification assumption. One option in this scenario is sequential specification tests: hypothesis tests of the identification assumption over the sequence. Borrowing an idea from the change-point literature, this paper shows how to use the distribution of p-values from sequential specification tests to estimate the point in the sequence where the identification assumption ceases to hold. Unlike current approaches, this method is robust to individual errant p-values and does not require choosing a test level or tuning parameter. This paper demonstrates the method's properties with a simulation study, and illustrates it by application to the problems of choosing a bandwidth in a regression discontinuity design while maintaining covariate balance and of choosing a lag order for a time series model.


Introduction
Null hypothesis tests and p-values play a central role in model checking. In this context, the null hypothesis may be that that the data are drawn from a distribution contained in the the model under study, or it may be derived from an underlying assumption. Typically, researchers use these specification tests to check the fit of a model chosen by other means, but in some cases hypothesis tests form the basis of a model selection procedure. In these cases, researchers construct a sequence of model specifications, ordered by preferability, and test each one. The best model whose assumptions "pass" the hypothesis test is chosen.
For example, take the datasets displayed in Figure 1, which will be discussed in more detail in Section 5. Figure 1A shows the annual total unemployment rate in the United States from 1890 to 2015. One of the simpler models for time series such as these is an order p autoregression, or AR(p), under which the value of the time series at point t may depend on its historical values at t − 1, ..., t − p but, conditional on those, is independent of values at points before t − p. To choose the order p, researchers may test model fit for a sequence of lag orders p, and choose the smallest p that the tests fail to reject. Here a smaller lag orders p are preferable because they lead to more parsimonious models and more precise estimates. Figure 1B plots data that Lindo et al. (2010) used to estimate the effect of academic probation on college students' subsequent grade point averages. University students were put on academic probation if their first-year cumulative grade point averages fell below a cutoff. This is an example of a regression discontinuity design (RDD; Thistlethwaite and Campbell, 1960), in which treatment is assigned if a numeric "running variable" R falls below (or above) a pre-specified cutoff c. Since treatment assignment is entirely a function of R and c, researchers can model the relationship between R and an outcome variable Y in order to estimate the effect of the treatment without confounding. A common tool for ensuring that RDD models are well specified is to limit the data analysis sample to subjects with R ∈ {c − ω, c + ω}, where ω > 0 is a bandwidth selected by the data analyst. One method for choosing ω relies on subjects' baseline covariates: researchers will estimate "effects" of the treatment on baseline covariates using data from subjects with R within ω of c. Since the treatment cannot possibly have an effect on baseline covariates, any estimated effects are due to model misspecification or an overly-large choice of ω. Following this reasoning, some methodologists recommend testing for effects on covariates using an array of candidate bandwidths, and choosing the largest bandwidth within which the null hypothesis of no effect cannot be rejected. The bandwidth tradeoff is similar to the AR(p) case: if ω is too large, the causal model might be misspecified and the effect estimate will be biased. If ω is too small, there will not be enough data to precisely estimate the effect of interest.
These are both examples of the use of sequential specification tests (SSTs) to choose a model. SSTs are also used in covariate selection for regression models (Greene, 2003), selecting the number of components in mixture models, latent class analysis, and factor analysis (Nylund et al., 2007) and in propensity-score matching (Hansen and Sales, 2015).
Do hypothesis tests make any sense in model selection? The results of a null hypothesis test, of course, are never evidence in favor of a null hypothesis; null hypotheses can only be rejected, not accepted. Along similar lines, the logic of controlling type-I error rates seems backwards when it comes to model selection, in which accepting a problematic specificationa type II error-is the major concern. These issues have prompted some methodologists (e.g. Cattaneo et al., 2015) to propose adjusting the size of specification tests to a value higher than the conventional α = 0.05. However, the appropriate value for α, and the criteria for selecting α, remain unclear.
On the other hand, a conceptually-sound model-selection method based on SSTs would be particularly useful; specification tests already exist for most common models, and they are regularly taught in introductory quantitative methods classes.
This paper develops such a method, based on a clever idea in change-point or threshold estimation. Mallik et al. (2011) points out that in a process with a change point, the pvalues from a sequence of tests of a null regression function are uniformly-distributed as long as the regression function is correct, but asymptotically zero when the function is not   Pfaff (2008) and Bureau of Labor Statistics (2016). Plot (B) shows data from Lindo et al. (2010): average subsequent grade point averages (GPAs), as a function of first-year GPAs, centered at the academic probation cutoff (dotted line).The points are sized proportionally to the number of students with each first-year GPA. correct. They use this dichotomous behavior to construct a simple, consistent estimator of the change-point, that is, the point at which the null model stops being correct.
We adapt that idea to the case of SSTs, choosing the change-point in a sequence of models, i.e. the point when models stop being correct. Our change-point estimator is based on the entire sequence of p-values, so that (unlike under current approaches) an individual outlier p-value will not drive its conclusions. What's more, unlike other SST model selectors, the change-point approach does not require the researcher to specify a level α or any other tuning parameter. This approach shifts the model selection rationale away from the logic of hypothesis testing, based on type-I and type-II error rates, and towards the logic of estimation.
Model selection is a very broad field in statistics with a rich history and-since an appropriate model must be chosen before any data analysis can proceed-it is centrally important.
A nice overview can be found in Rao et al. (2001) 1 , who state that "[a]lmost all statistical problems can be considered as model selection problems" (p. 3). That monograph focuses on regression problems, and describes model selection based on hypothesis tests (including SSTs) as well as a range of methods that seek to optimize prediction errors, information criteria, or posterior probability for a range of different scenarios. This paper is both more broad and much more restricted in its focus. It is restricted to model selection based on hypothesis tests (more specifically, p-values), and in the particular case in which the researcher is choosing between a sequence of models, ranked by preference. We are not presenting any new hypothesis tests (unlike, e.g. Vuong 1989) or any other way to decide between two competing models (e.g. Schwarz et al. 1978)-instead, we assume that a hypothesis test already exists. Our goal is to provide guidance on how to use p-values from that test to choose between models in a sequence. On the other hand, the method we are introducing will apply in any scenario where such a hypothesis test is available.

Sequences of Models and Tests
Say, in specifying a model, a researcher must choose from a discrete set of specifications S d ; d = 1, 2, . . . , D. In this setup, we consider the set of D candidate specifications as fixed, and not dependent on sample size. The models are ordered by preference, subject to a testable assumption operationalized as H 0d ; that is, if d > d , and H 0d holds for both S d and S d , then The assumption H 0d can have the same form for each d (e.g. model S d fits the data) or can be relative (e.g. model S d fits the data as well as S d−1 ). Denote the optimal specification choice as S d * , where Then we assume the following: Now assume the researcher has chosen a testing procedure for H 0d producing a p-value for each H 0d , p 1 , . . . , p D . Let n d be the sample size included in the test for H 0d . This can take a number of forms-for instance, in the unemployment example of Figure 1A, as d increases the model becomes more parsimonious, but the data used to fit the model stays the same, so n d = n for all d. In contrast, in the RDD example of Figure 1B, a specification with larger d includes more observations, so d 2 > d 1 implies n 2 ≥ n 1 . When a dataset has a multilevel or hierarchical structure, the concept of a sample size becomes more subtle. For instance, in a clustered survey sample n d may refer to the number of clusters included in a hypothesis test, rather than the number of observations. n d plays no direct role in the computations we will introduce, so for the sake of generality we can afford to be slightly vague about its meaning.
When we wish to emphasize the dependence of p d on n d , we write it as p nd . In any event, we assume that the test of each H 0d is valid and asymptotically powerful. That is, we assume: so that when H 0d is true, the associated p-value is uniformly distributed, and when H 0d is false the resulting p-value should be small in large samples. In some cases, it will suffice to substitute the following weaker form of Assumption 2: The goal here is to use p D to choose a specificationd that is as large as possible without violating the specification assumption encoded in H 0d .
For the methods we describe here and in the following section, it is not necessary for the p-values p D to be mutually independent; indeed, they typically are not.

Two Existing Approaches to Sequential Specification Tests
In the contexts of sequential specification tests, Rao et al. (2001) suggests a common approach to choosing a specification: for a pre-specified α ∈ (0, 1), let is the largest value of d for which H 0d cannot be rejected at level α (and 0 if no such d exists). Although it may seem as though multiplicity corrections may be necessary here, it turns out that this is not the case. The "stepwise intersection-union principle" (Berger et al., 1988;Rosenbaum, 2008;Hansen and Sales, 2015) insures that the family-wise error rate is maintained, and the probability of falsely rejecting any null hypothesis H 0d is bounded by α: Proposition 1 (Hansen and Sales 2015). Under Assumptions 1 and 2, pr(d max The proof can be found in Hansen and Sales (2015).

and (3) that under
Assumption 2, the probability of rejecting a true null at level α is equal to α. If d * = 0, is the specification that would result from testing null hypotheses backwards: for d = D, D − 1, . . . , d, . . . , 1, test H 0d , and stop testing Another common choice ford (e.g. Lütkepohl, 2005)  to be too small.
In the following section, we will suggest an alternative way to select a specification from a sequence of p-values that avoids these pitfalls by relying on the logic of estimation instead of the logic of hypothesis testing.

The Change Point Estimator
The asymptotic behavior described in Assumption 2 -when d ≤ d * , Ep dn = 1/2 and when d > d * , p dn → p 0-suggests a least-squares estimator for d * : In other words, the estimated M is the point at which the p-values cease behaving as p-values testing a true null, with mean 1/2, and instead are drawn from a distribution with a lower mean.
Because p-values are inherently unpredictable when the null hypothesis is true, even in large samplesd M may choose a sub-optimal specification-i.e. pr(d M < d * ) > 0 for all n.
In particular, note that under Assumption 2, pr(p d * < 1/4) = 1/4 regardless of n d , and that Ifd M > d * , then specification Sd M violates the assumption encoded in H 0d ; as sample size increases, the probability of this event decreases to zero. The same property holds for d max α , with α > 0 fixed, for the same reason.
In a way,d M is similar to d max 0.25 , the largest d for which p d > α = 0.25, because both penalize p-values lower than 0.25. However, they are not equivalent, as the following proposition shows: Liked M , model selectord ab M looks for behavior that differs between p-values testing true and false null hypotheses. Unliked M , it does not depend on theoretically established distributions for these p-values, but searches over a grid for their location parameters.d ab M will be more computationally expensive to compute thand M , but will may yield better results, especially in small samples.

A Simulation Study
This section will present a simulation study to compare the behavior of model selectors Specification p-values are generated by comparing all of these simulated test statistics against the null distribution N (0, 1).   is competitive in all three criteria and, arguably, balances them the best. At β = 2, and β = 3, the d max estimators have better RMSE and tend to pick the optimal model slightly more often thand M , but are more likely to pick misspecified models. d min 0.25 is the least likely to pick misspecified models, but the models it does pick tend to be much too small.

Figure 2 and
In general,d M tends to be more conservative than d max or d min with α = 0.05 but much less conservative than d min 0.25 . Its performance is most similar to d max 0.25 , while being slightly more conservative.
The appendix gives a larger version of Table 1  Assume that the time series follows an "AR(d)" model; that is, where µ and {φ i } d i=1 are parameters to be estimated and t is white noise. In this model, the unemployment in one year is a function of unemployment rates in the previous d years, but conditionally independent of even earlier measurements.
Having settled on model (2), the analyst must choose d, the lag order. Sequential specification tests can be useful here (e.g. Ivanov et al., 2005). Consider the null hypothesis  (Akaike, 1969;Schwarz et al., 1978). For instance, choosing the model that minimizes aic, defined as 2(d + 2) − 2log(L d ), whereL d is the maximized likelihood of the AR(d) model, or bic, which is defined as log(n)(d + 2) − 2log(L d ).
A large literature surrounds this important question (See, e.g. McQuarrie and Tsai, 1998;Liew, 2004, and the citations therein). This section is not meant as a complete treatment, or even an overview, of lag order selection, but as an illustration of sequential specification tests in a well-known area. Figure 3 gives the p-values from a sequence likelihood ratio tests, as described in Pfaff (2008, Ch.1), which discussed a similar dataset. For each candidate lag order d, the likelihood ratio test compares twice the log of the ratio of the likelihoods of AR(d+1) and AR(d) models to a χ 2 1 distribution. If the AR(d + 1) model fits much better than the AR(d) model, a lag order of d may not be sufficient. The p-values follow a stark pattern: for d < 5, they are close to zero, while for d ≥ 5, they appear roughly uniformly distributed. Table 2, and vertical lines in Figure 3, show the lag order choices from d max α , d min α ,d M , andd ab M , which are based on the p-values, and the lag orders that minimize aic and bic, based directly on the models' likelihood and numbers of parameters. Here, smaller models are preferable to larger models, so d * is the smallest acceptable value for d.

Sequential specification tests in Regression Discontinuity Bandwidth Selection: Estimating the Effect of Academic Probation on College GPAs
At many universities, students who fail to achieve a minimum GPA c are put on academic probation. Lindo et al. (2010) recognized that academic probation can form a regression discontinuity design, in which treatment is a function of a "running variable" with a predetermined cutoff. Specifically, probation Z is a function of a "running variable" R, students' GPAs: students with R < c are put on probation-Z = 1-and students with R > c are not, Z = 0. That being the case, students with GPAs just below c may be comparable to students with GPAs just above c, so comparing these two sets of students allows researchers to estimate the effect of probation on outcomes Y (perhaps after adjusting for Y 's relationship with R). The challenge becomes defining "just above" and "just below"-that is, selecting a "bandwidth" ω * > 0 such that subjects i with R i ∈ (c − ω * , c) are suitably comparable to subjects with R i ∈ (c, c + ω * ).
A number of authors (Lee and Lemieux, 2010;Cattaneo et al., 2015;Li et al., 2015, e.g.) recommend sequential specification tests, using baseline covariates X, as part of the procedure for choosing ω. At a sequence of candidate bandwidths 0 < ω 1 < · · · < ω d < · · · < ω D , they recommend testing the equality of covariate means (again, perhaps after adjusting for R) between subjects with R i ∈ (c − ω d , c) and those with R i ∈ (c, c + ω d ), and choosing a bandwidth ω * = ω d * . These are essentially placebo tests-since the treatment cannot affect baseline covariates, differences in covariate means between treated and untreated subjects must be an indicator of incomparability between the groups, or model misspecification.
In a secondary analysis of the academic probation dataset, Sales and Hansen (2020) chose an RDD bandwidth using a set of seven baseline covariates: students' high-school GPA (expressed in percentiles), age at college matriculation, number of attempted credits, gender, native language (English or other), birth place (North America or other) and university campus (the university consisted of three campuses). For each covariate X k and for each candidate bandwidth ω d , they let p kd be the p-value corresponding the coefficient on Z from the regression of X k on R and Z, fit to the subset of students with R ∈ (c − ω d , c + ω d ). These regression models were linear for continuous covariates and logistic for binary covariates, with heteroskedasticity-consistent sandwich standard errors (Zeileis et al., 2020;Zeileis, 2004Zeileis, , 2006. Then, the omnibus specification p-value for bandwidth ω d was p d = min{1, 7p 1d , . . . , 7p 7d }, the minimum of the Bonferroni-adjusted p-values p kd . The resulting p-values are plotted in Figure 4, with bandwidth selections corresponding to d max 0.05 , d max 0.25 , d min 0.05 ,d M , andd ab M . Also plotted is the more conventional bandwidth recommended by Imbens and Kalyanaraman (2011), denoted IK, which is based on non-parametric estimates of the curvature of the regression function of Y on R, rather than covariate placebo tests. These bandwidth selections are also listed in Table 3.
For most small bandwidths ω d , p d is fairly large, and in many cases equal to 1. This apparent super-uniform distribution is probably due to the conservative Bonferroni correction applied to the p-values from individual covariates. On the other hand, at the smallest candidate bandwidth ω 1 =0.02, the p-value is p 1 =0.072, and the p-value at the 6th bandwidth, ω 6 = 0.07, another small bandwidth, is p 6 = 0.009. After around ω = 0.75, the p-values   Table 3: Selected regression discontinuity bandwidths ("d" is the point in the sequence selected, and "Bandwidth" is the actual bandwidth) using covariate balance tests, or using the method described in Imbens and Kalyanaraman (2011), along with their associated estimates for the average treatment effect of academic probation on subsequent GPAs (ATE), with 95% confidence intervals in parentheses.
Since the p-value at the smallest candidate bandwidth, p 1 = 0. It is worth noting that super-uniformity of the p-values for small bandwidths inflates the sums t≤d (p t − 1/4) in (1). Therefore, in this cased ab M , which relies less on the uniform model for p-values under H 0 , may be a more appropriate choice thand M . Table 3 also lists estimated treatment effects of academic probation on students' subsequent GPAs, along with 95% confidence intervals. The effects were estimated following the method described in Sales and Hansen (2020), with the exception of the estimate for the IK bandwidth which used local linear regression, as implemented in the R package rdd

Discussion
As long as data analysts use specification tests and p-values to select their models, decision rules translating a sequence of p-values to a model choice will be necessary. Currently, the most common approach compares the p-values to a pre-specified threshold. This approach turns the logic of null hypothesis testing on its head, using p-values to identify well-specified models-i.e. true null hypotheses-rather than to reject misspecified models. Moreover, the d min α approach, by failing to control the familywise type-I error rate, can be extremely conservative, for instance recommending very high lag order in the unemployment example of Section 5.1 and very small bandwidths (or none at all) in the academic probation example of Section 5.2. In contrast, d max α does control familywise type-I error rates. However, both threshold-based approaches, d min and d max , require specifying a threshold, and there is rarely any clear guidance on how to do so.
The alternatives introduced here,d M andd ab M , drawn from the change-point literature, skirt these issues entirely. Rather than using p-values to reject (or accept) null hypotheses, they examine the full distribution of p-values. They require no arbitrary threshold to be specified. As shown in the simulation study and the two data examples, they tend to avoid the conservativism and outlier sensitivity of d min 0.25 and the anti-conservativism of d max 0.05 (indeed, Proposition 2 states thatd M is asymptotically conservative).
Actually, the simulation results and examples show thatd M tends to agree with d max 0.25 , which itself performs rather well. This suggests that d max with a default threshold of 0.25 may be a good option for data analysts who wish to continue using threshold-based approaches.
There are several open questions regardingd M 's behavior and use. First, it is unclear whether or when the more flexible versiond ab M should be preferred tod M ; there is good reason to expect it to perform better when sample sizes are small, but is there a cost associated with usingd ab M in larger samples? Further, there may be ways to construct sequential specification tests in a way that improvesd M 's performance. How to best construct specification tests for different model selectors is a topic for future research.
Ultimately, the goal of model selection is to produce parameter estimates or predictions with desired properties. Ideally, researchers would select a model with this end in mind; however, the effect of model selection on final estimates or predictions depends heavily on specific circumstances. That said, a careful study of the effect ofd M on estimates and predictions in a wide range of cases could be useful.
Model selectorsd M andd ab M may be particularly useful in measurement modeling, where model choice based on sequences of p-values are common (such as to select the number of components in factor or latent class analysis). The encouraging performance ofd M when d * = 2 suggests thatd M may be appropriate even when the true number of components is small.
A Simulation Results with different D and d * Table 4 gives the results of Table 1 for varying values of D (the total number of candidate models) and d * (the optimal model).