A Simple Way to Deal with Cherry-picking
  Statistical hypothesis testing serves as statistical evidence for scientific
innovation. However, if the reported results are intentionally biased,
hypothesis testing no longer controls the rate of false discovery. In
particular, we study such selection bias in machine learning models where the
reporter is motivated to promote an algorithmic innovation. When the number of
possible configurations (e.g., datasets) is large, we show that the reporter
can falsely report an innovation even if there is no improvement at all. We
propose a `post-reporting' solution to this issue where the bias of the
reported results is verified by another set of results. The theoretical
findings are supported by experimental results with synthetic and real-world
datasets.
