image belongs to. Training may happen by minimizing or decreasing the output’s error, also referred to as the loss. Examples of losses are mean squared error, cross-entropy, etc. In deep learning techniques, training is an iterative process, where at each iteration the algorithm modifies the weights of the neural net to make a gradual improvement of the network’s output, i.e. to gradually decrease the loss. In the present description terms “neuralnetwork”,“neuralnet”and “network” are used interchangeably, and also the “weights” of neural network may be referred to as “learnableparameters”or “parameters”. A neural network has two main modes of operation: training phase and testing phase. The training phase is the development phase, where the network learns to perform the final task. Training may involve iteratively updating the weights between units. Training a neural network is an optimization process, where the goal of optimization or training process is to make the neural network to learn the properties of the data distribution from a limited training dataset. In other words, the goal is to learn to use a limited training dataset in order to learn to generalize the previously unseen data, i.e., data which was not used for training the neural network. This may be referred to as generalization. In practice, data may be split into at least two sets, the training set and the validation set. The training set is used for training the network, i.e., to modify its learnable parameters in order to minimize the loss. The validation set is used for checking the performance of the network on data which was not used to minimize the loss, as an indication of the final performance of the neural network. In particular, the errors on the training set and on the validation set are monitored during the training process to understand the following things: - If the neural network is learning at all - in this case, the training set error should decrease, otherwise the neural network is in the region of underfitting. - If the network is learning to generalize - in this case, also the validation set error needs to decrease and not to be too much higher than the training set error. If the training set error is low, but the validation set error is much higher than the training set error, or it does not decrease, or it even increases, the neural network is in the regime of overfitting (i.e. optimization). This means that the neural network has just memorized the training set’s properties and performs well only on that set, but performs poorly on a set not used for tuning its parameters.