Approximation to Object Conditional Validity with Inductive Conformal
  Predictors
  Conformal predictors are machine learning algorithms that output prediction
sets that have a guarantee of marginal validity for finite samples with minimal
distributional assumptions. This is a property that makes conformal predictors
useful for machine learning tasks where we require reliable predictions. It
would also be desirable to achieve conditional validity in the same setting, in
the sense that validity of the prediction intervals remains valid regardless of
conditioning on any particular property of the object of the prediction.
Unfortunately, it has been shown that such conditional validity is impossible
to guarantee for non-trivial prediction problems for finite samples. In this
article, instead of trying to achieve a strong conditional validity result, the
weaker goal of achieving an approximation to conditional validity is
considered. A new algorithm is introduced to do this by iteratively adjusting a
conformity measure to deviations from object conditional validity measured in
the training data. Along with some theoretical results, experimental results
are provided for three data sets that demonstrate (1) in real world machine
learning tasks, lack of conditional validity is a measurable problem and (2)
that the proposed algorithm is effective at alleviating this problem.
