Focus-and-Expand: Training Guidance Through Gradual Manipulation of
  Input Features
  We present a simple and intuitive Focus-and-eXpand (\fax) method to guide the
training process of a neural network towards a specific solution. Optimizing a
neural network is a highly non-convex problem. Typically, the space of
solutions is large, with numerous possible local minima, where reaching a
specific minimum depends on many factors. In many cases, however, a solution
which considers specific aspects, or features, of the input is desired. For
example, in the presence of bias, a solution that disregards the biased feature
is a more robust and accurate one. Drawing inspiration from Parameter
Continuation methods, we propose steering the training process to consider
specific features in the input more than others, through gradual shifts in the
input domain. \fax extracts a subset of features from each input data-point,
and exposes the learner to these features first, Focusing the solution on them.
Then, by using a blending/mixing parameter $\alpha$ it gradually eXpands the
learning process to include all features of the input. This process encourages
the consideration of the desired features more than others. Though not
restricted to this field, we quantitatively evaluate the effectiveness of our
approach on various Computer Vision tasks, and achieve state-of-the-art bias
removal, improvements to an established augmentation method, and two examples
of improvements to image classification tasks. Through these few examples we
demonstrate the impact this approach potentially carries for a wide variety of
problems, which stand to gain from understanding the solution landscape.
