Parameter Selection: Why We Should Pay More Attention to It
  The importance of parameter selection in supervised learning is well known.
However, due to the many parameter combinations, an incomplete or an
insufficient procedure is often applied. This situation may cause misleading or
confusing conclusions. In this opinion paper, through an intriguing example we
point out that the seriousness goes beyond what is generally recognized. In the
topic of multi-label classification for medical code prediction, one
influential paper conducted a proper parameter selection on a set, but when
moving to a subset of frequently occurring labels, the authors used the same
parameters without a separate tuning. The set of frequent labels became a
popular benchmark in subsequent studies, which kept pushing the state of the
art. However, we discovered that most of the results in these studies cannot
surpass the approach in the original paper if a parameter tuning had been
conducted at the time. Thus it is unclear how much progress the subsequent
developments have actually brought. The lesson clearly indicates that without
enough attention on parameter selection, the research progress in our field can
be uncertain or even illusive.
