Unification of HDP and LDA Models for Optimal Topic Clustering of
  Subject Specific Question Banks
  There has been an increasingly popular trend in Universities for curriculum
transformation to make teaching more interactive and suitable for online
courses. An increase in the popularity of online courses would result in an
increase in the number of course-related queries for academics. This, coupled
with the fact that if lectures were delivered in a video on demand format,
there would be no fixed time where the majority of students could ask
questions. When questions are asked in a lecture there is a negligible chance
of having similar questions repeatedly, but asynchronously this is more likely.
In order to reduce the time spent on answering each individual question,
clustering them is an ideal choice. There are different unsupervised models fit
for text clustering, of which the Latent Dirichlet Allocation model is the most
commonly used. We use the Hierarchical Dirichlet Process to determine an
optimal topic number input for our LDA model runs. Due to the probabilistic
nature of these topic models, the outputs of them vary for different runs. The
general trend we found is that not all the topics were being used for
clustering on the first run of the LDA model, which results in a less effective
clustering. To tackle probabilistic output, we recursively use the LDA model on
the effective topics being used until we obtain an efficiency ratio of 1.
Through our experimental results we also establish a reasoning on how Zeno's
paradox is avoided.
