Competitive Machine Learning: Best Theoretical Prediction vs
  Optimization
  Machine learning is often used in competitive scenarios: Participants learn
and fit static models, and those models compete in a shared platform. The
common assumption is that in order to win a competition one has to have the
best predictive model, i.e., the model with the smallest out-sample error. Is
that necessarily true? Does the best theoretical predictive model for a target
always yield the best reward in a competition? If not, can one take the best
model and purposefully change it into a theoretically inferior model which in
practice results in a higher competitive edge? How does that modification look
like? And finally, if all participants modify their prediction models towards
the best practical performance, who benefits the most? players with inferior
models, or those with theoretical superiority? The main theme of this paper is
to raise these important questions and propose a theoretical model to answer
them. We consider a study case where two linear predictive models compete over
a shared target. The model with the closest estimate gets the whole reward,
which is equal to the absolute value of the target. We characterize the reward
function of each model, and using a basic game theoretic approach, demonstrate
that the inferior competitor can significantly improve his performance by
choosing optimal model coefficients that are different from the best
theoretical prediction. This is a preliminary study that emphasizes the fact
that in many applications where predictive machine learning is at the service
of competition, much can be gained from practical (back-testing) optimization
of the model compared to static prediction improvement.
