A hybrid ensemble method with negative correlation learning for
  regression
  Hybrid ensemble, an essential branch of ensembles, has flourished in numerous
machine learning problems, especially regression. Several studies have
confirmed the importance of diversity; however, previous ensembles only
consider diversity in the sub-model training stage, with limited improvement
compared to single models. In contrast, this study focuses on the sub-model
combination stage of the ensemble. It solves a non-convex optimization problem
using an interior-point filtering linear-search algorithm to select and weight
sub-models from a heterogeneous model pool automatically. This optimization
problem innovatively incorporates negative correlation learning as a penalty
term. Thus, a diverse model subset can be selected. Experimental results show
that the approach outperforms single model and overcomes the instability of the
models and parameters. Compared to bagging and stacking without model
diversity, our method stands out more and confirms the importance of diversity
in the ensemble. Additionally, the performance of our proposed method is better
than that of simple and weighted averages, and the variance of the weights is
lower and more stable than that of a linear model. Finally, the prediction
accuracy can be further improved by fine-tuning the weights using the error
inverse weights. In conclusion, the value of this study lies in its ease of use
and effectiveness, allowing the hybrid ensemble to embrace both diversity and
accuracy.
