Online Feature Selection for Efficient Learning in Networked Systems
  Current AI/ML methods for data-driven engineering use models that are mostly
trained offline. Such models can be expensive to build in terms of
communication and computing cost, and they rely on data that is collected over
extended periods of time. Further, they become out-of-date when changes in the
system occur. To address these challenges, we investigate online learning
techniques that automatically reduce the number of available data sources for
model training. We present an online algorithm called Online Stable Feature Set
Algorithm (OSFS), which selects a small feature set from a large number of
available data sources after receiving a small number of measurements. The
algorithm is initialized with a feature ranking algorithm, a feature set
stability metric, and a search policy. We perform an extensive experimental
evaluation of this algorithm using traces from an in-house testbed and from a
data center in operation. We find that OSFS achieves a massive reduction in the
size of the feature set by 1-3 orders of magnitude on all investigated
datasets. Most importantly, we find that the accuracy of a predictor trained on
a OSFS-produced feature set is somewhat better than when the predictor is
trained on a feature set obtained through offline feature selection. OSFS is
thus shown to be effective as an online feature selection algorithm and robust
regarding the sample interval used for feature selection. We also find that,
when concept drift in the data underlying the model occurs, its effect can be
mitigated by recomputing the feature set and retraining the prediction model.
