A Reflection on Learning from Data: Epistemology Issues and Limitations
  Although learning from data is effective and has achieved significant
milestones, it has many challenges and limitations. Learning from data starts
from observations and then proceeds to broader generalizations. This framework
is controversial in science, yet it has achieved remarkable engineering
successes. This paper reflects on some epistemological issues and some of the
limitations of the knowledge discovered in data. The document discusses the
common perception that getting more data is the key to achieving better machine
learning models from theoretical and practical perspectives. The paper sheds
some light on the shortcomings of using generic mathematical theories to
describe the process. It further highlights the need for theories specialized
in learning from data. While more data leverages the performance of machine
learning models in general, the relation in practice is shown to be logarithmic
at its best; After a specific limit, more data stabilize or degrade the machine
learning models. Recent work in reinforcement learning showed that the trend is
shifting away from data-oriented approaches and relying more on algorithms. The
paper concludes that learning from data is hindered by many limitations. Hence
an approach that has an intensional orientation is needed.
