Model Order Selection Based on Information Theoretic Criteria: Design of
  the Penalty
  Information theoretic criteria (ITC) have been widely adopted in engineering
and statistics for selecting, among an ordered set of candidate models, the one
that better fits the observed sample data. The selected model minimizes a
penalized likelihood metric, where the penalty is determined by the criterion
adopted. While rules for choosing a penalty that guarantees a consistent
estimate of the model order are known, theoretical tools for its design with
finite samples have never been provided in a general setting. In this paper, we
study model order selection for finite samples under a design perspective,
focusing on the generalized information criterion (GIC), which embraces the
most common ITC. The theory is general, and as case studies we consider: a) the
problem of estimating the number of signals embedded in additive white Gaussian
noise (AWGN) by using multiple sensors; b) model selection for the general
linear model (GLM), which includes e.g. the problem of estimating the number of
sinusoids in AWGN. The analysis reveals a trade-off between the probabilities
of overestimating and underestimating the order of the model. We then propose
to design the GIC penalty to minimize underestimation while keeping the
overestimation probability below a specified level. For the considered
problems, this method leads to analytical derivation of the optimal penalty for
a given sample size. A performance comparison between the penalty optimized GIC
and common AIC and BIC is provided, demonstrating the effectiveness of the
proposed design strategy.
