Counting the learnable functions of structured data
  Cover's function counting theorem is a milestone in the theory of artificial
neural networks. It provides an answer to the fundamental question of
determining how many binary assignments (dichotomies) of $p$ points in $n$
dimensions can be linearly realized. Regrettably, it has proved hard to extend
the same approach to more advanced problems than the classification of points.
In particular, an emerging necessity is to find methods to deal with structured
data, and specifically with non-pointlike patterns. A prominent case is that of
invariant recognition, whereby identification of a stimulus is insensitive to
irrelevant transformations on the inputs (such as rotations or changes in
perspective in an image). An object is therefore represented by an extended
perceptual manifold, consisting of inputs that are classified similarly. Here,
we develop a function counting theory for structured data of this kind, by
extending Cover's combinatorial technique, and we derive analytical expressions
for the average number of dichotomies of generically correlated sets of
patterns. As an application, we obtain a closed formula for the capacity of a
binary classifier trained to distinguish general polytopes of any dimension.
These results may help extend our theoretical understanding of generalization,
feature extraction, and invariant object recognition by neural networks.
