Predicting the Outcome of Judicial Decisions made by the European Court
  of Human Rights
  In this study, machine learning models were constructed to predict whether
judgments made by the European Court of Human Rights (ECHR) would lead to a
violation of an Article in the Convention on Human Rights. The problem is
framed as a binary classification task where a judgment can lead to a
"violation" or "non-violation" of a particular Article. Using auto-sklearn, an
automated algorithm selection package, models were constructed for 12 Articles
in the Convention. To train these models, textual features were obtained from
the ECHR Judgment documents using N-grams, word embeddings and paragraph
embeddings. Additional documents, from the ECHR, were incorporated into the
models through the creation of a word embedding (echr2vec) and a doc2vec model.
The features obtained using the echr2vec embedding provided the highest
cross-validation accuracy for 5 of the Articles. The overall test accuracy,
across the 12 Articles, was 68.83%. As far as we could tell, this is the first
estimate of the accuracy of such machine learning models using a realistic test
set. This provides an important benchmark for future work. As a baseline, a
simple heuristic of always predicting the most common outcome in the past was
used. The heuristic achieved an overall test accuracy of 86.68% which is 29.7%
higher than the models. Again, this was seemingly the first study that included
such a heuristic with which to compare model results. The higher accuracy
achieved by the heuristic highlights the importance of including such a
baseline.
