What I mean is the following: Instead of processing all the training data at once and calculating a model, we process one data point at a time and update the model directly afterwards.

I have seen the terms "on-line (or online) learning" and "incremental learning" for this. Is there a subtle difference? Is one term used more frequently? Or does it depend on the research community?

Edit: The Bishop book (Pattern Recognition and Machine Learning) uses the terms on-line learning and sequential learning as synonyms but does not mention incremental learning.

  • 1
    $\begingroup$ Interesting, I haven't read the Bishop book, but I've heard online learning and sequential learning treated as different concepts. I understand online learning to mean a learning problem in which a model is able to use new incoming data as a way to "grow" the training set and update the model dynamically. I've heard sequential learning applied to problems where the goal is to predict a sequence instead of individual instances, as in predicting part-of-speech sequences in POS tagging. $\endgroup$ – kylerthecreator Jun 29 '15 at 19:22
  • 1
    $\begingroup$ To add to the confusion with yet another phrase scikit-learn uses "partial fit". $\endgroup$ – Finn Årup Nielsen Oct 9 '15 at 13:59

So, the following answer is just based on different opinions of collegues and professors from the field. I want to try to summarize it briefly:

Sequential and online learning is mostly associated with Bayesian updating. The sequential learning is used widely for an order in time of the data, meaning that $x_1$ is coming always first, then $x_2$, then $x_3$ and so on. The Dataset each has a certain order in that sense. In contrast to that incremental may be a whole block of data at time x and another block of data at time y. While the block internally may be randomly ordered.

Concerning online learning, the people mostly referred to a data stream, hence a online learning is always incremental learning but incremental learning does not have to be online.

These are quite fuzzy definitions, and in my opinion there is not clear definition though. I still hope that helps.


Although the definitions of the two concepts are fuzzy, there is a slight difference between online learning and incremental learning approaches.

In online learning approach, model is updated to adapt to the new data. It is possible that the model can forget the previously learned inferences which is called as Catastrophic Interference.

Whereas in the incremental approach, even as the model is updated, previous inferences are not forgotten.


Your Answer

