Inferred vs traditional personality assessment: are we predicting the
  same thing?
  Machine learning methods are widely used by researchers to predict
psychological characteristics from digital records. To find out whether
automatic personality estimates retain the properties of the original traits,
we reviewed 220 recent articles. First, we put together the predictive quality
estimates from a subset of the studies which declare separation of training,
validation, and testing phases, which is critical for ensuring the correctness
of quality estimates in machine learning. Only 20% of the reviewed papers met
this criterion. To compare the reported quality estimates, we converted them to
approximate Pearson correlations. The credible upper limits for correlations
between predicted and self-reported personality traits vary in a range between
0.42 and 0.48, depending on the specific trait. The achieved values are
substantially below the correlations between traits measured with distinct
self-report questionnaires. This suggests that we cannot readily interpret
personality predictions as estimates of the original traits or expect predicted
personality traits to reproduce known relationships with life outcomes
regularly. Next, we complement quality estimates evaluation with evidence on
psychometric properties of predicted traits. The few existing results suggest
that predicted traits are less stable with time and have lower effective
dimensionality than self-reported personality. The predictive text-based models
perform substantially worse outside their training domains but stay above a
random baseline. The evidence on the relationships between predicted traits and
external variables is mixed. Predictive features are difficult to use for
validation, due to the lack of prior hypotheses. Thus, predicted personality
traits fail to retain important properties of the original characteristics.
This calls for the cautious use and targeted validation of the predictive
models.
