Post-Selections in AI and How to Avoid Them
  Neural network based Artificial Intelligence (AI) has reported increasing
scales in experiments. However, this paper raises a rarely reported stage in
such experiments called Post-Selection alter the reader to several possible
protocol flaws that may result in misleading results. All AI methods fall into
two broad schools, connectionist and symbolic. The Post-Selection fall into two
kinds, Post-Selection Using Validation Sets (PSUVS) and Post-Selection Using
Test Sets (PSUTS). Each kind has two types of post-selectors, machines and
humans. The connectionist school received criticisms for its "black box" and
now the Post-Selection; but the seemingly "clean" symbolic school seems more
brittle because of its human PSUTS. This paper first presents a controversial
view: all static "big data" are non-scalable. We then analyze why
error-backprop from randomly initialized weights suffers from severe local
minima, why PSUVS lacks cross-validation, why PSUTS violates well-established
protocols, and why every paper involved should transparently report the
Post-Selection stage. To avoid future pitfalls in AI competitions, this paper
proposes a new AI metrics, called developmental errors for all networks
trained, under Three Learning Conditions: (1) an incremental learning
architecture (due to a "big data" flaw), (2) a training experience and (3) a
limited amount of computational resources. Developmental Networks avoid
Post-Selections because they automatically discover context-rules on the fly by
generating emergent Turing machines (not black boxes) that are optimal in the
sense of maximum-likelihood across lifetime, conditioned on the Three Learning
Conditions.
