Weight Vector Tuning and Asymptotic Analysis of Binary Linear
  Classifiers
  Unlike its intercept, a linear classifier's weight vector cannot be tuned by
a simple grid search. Hence, this paper proposes weight vector tuning of a
generic binary linear classifier through the parameterization of a
decomposition of the discriminant by a scalar which controls the trade-off
between conflicting informative and noisy terms. By varying this parameter, the
original weight vector is modified in a meaningful way. Applying this method to
a number of linear classifiers under a variety of data dimensionality and
sample size settings reveals that the classification performance loss due to
non-optimal native hyperparameters can be compensated for by weight vector
tuning. This yields computational savings as the proposed tuning method reduces
to tuning a scalar compared to tuning the native hyperparameter, which may
involve repeated weight vector generation along with its burden of
optimization, dimensionality reduction, etc., depending on the classifier. It
is also found that weight vector tuning significantly improves the performance
of Linear Discriminant Analysis (LDA) under high estimation noise. Proceeding
from this second finding, an asymptotic study of the misclassification
probability of the parameterized LDA classifier in the growth regime where the
data dimensionality and sample size are comparable is conducted. Using random
matrix theory, the misclassification probability is shown to converge to a
quantity that is a function of the true statistics of the data. Additionally,
an estimator of the misclassification probability is derived. Finally,
computationally efficient tuning of the parameter using this estimator is
demonstrated on real data.
