Decision tree heuristics can fail, even in the smoothed setting
  Greedy decision tree learning heuristics are mainstays of machine learning
practice, but theoretical justification for their empirical success remains
elusive. In fact, it has long been known that there are simple target functions
for which they fail badly (Kearns and Mansour, STOC 1996).
  Recent work of Brutzkus, Daniely, and Malach (COLT 2020) considered the
smoothed analysis model as a possible avenue towards resolving this disconnect.
Within the smoothed setting and for targets $f$ that are $k$-juntas, they
showed that these heuristics successfully learn $f$ with depth-$k$ decision
tree hypotheses. They conjectured that the same guarantee holds more generally
for targets that are depth-$k$ decision trees.
  We provide a counterexample to this conjecture: we construct targets that are
depth-$k$ decision trees and show that even in the smoothed setting, these
heuristics build trees of depth $2^{\Omega(k)}$ before achieving high accuracy.
We also show that the guarantees of Brutzkus et al. cannot extend to the
agnostic setting: there are targets that are very close to $k$-juntas, for
which these heuristics build trees of depth $2^{\Omega(k)}$ before achieving
high accuracy.
