Back to the Basics: Bayesian extensions of IRT outperform neural
  networks for proficiency estimation
  Estimating student proficiency is an important task for computer based
learning systems. We compare a family of IRT-based proficiency estimation
methods to Deep Knowledge Tracing (DKT), a recently proposed recurrent neural
network model with promising initial results. We evaluate how well each model
predicts a student's future response given previous responses using two
publicly available and one proprietary data set. We find that IRT-based methods
consistently matched or outperformed DKT across all data sets at the finest
level of content granularity that was tractable for them to be trained on. A
hierarchical extension of IRT that captured item grouping structure performed
best overall. When data sets included non-trivial autocorrelations in student
response patterns, a temporal extension of IRT improved performance over
standard IRT while the RNN-based method did not. We conclude that IRT-based
models provide a simpler, better-performing alternative to existing RNN-based
models of student interaction data while also affording more interpretability
and guarantees due to their formulation as Bayesian probabilistic models.
