Identifying Semantically Duplicate Questions Using Data Science
  Approach: A Quora Case Study
  Identifying semantically identical questions on, Question and Answering
social media platforms like Quora is exceptionally significant to ensure that
the quality and the quantity of content are presented to users, based on the
intent of the question and thus enriching overall user experience. Detecting
duplicate questions is a challenging problem because natural language is very
expressive, and a unique intent can be conveyed using different words, phrases,
and sentence structuring. Machine learning and deep learning methods are known
to have accomplished superior results over traditional natural language
processing techniques in identifying similar texts. In this paper, taking Quora
for our case study, we explored and applied different machine learning and deep
learning techniques on the task of identifying duplicate questions on Quora's
dataset. By using feature engineering, feature importance techniques, and
experimenting with seven selected machine learning classifiers, we demonstrated
that our models outperformed previous studies on this task. Xgboost model with
character level term frequency and inverse term frequency is our best machine
learning model that has also outperformed a few of the Deep learning baseline
models. We applied deep learning techniques to model four different deep neural
networks of multiple layers consisting of Glove embeddings, Long Short Term
Memory, Convolution, Max pooling, Dense, Batch Normalization, Activation
functions, and model merge. Our deep learning models achieved better accuracy
than machine learning models. Three out of four proposed architectures
outperformed the accuracy from previous machine learning and deep learning
research work, two out of four models outperformed accuracy from previous deep
learning study on Quora's question pair dataset, and our best model achieved
accuracy of 85.82% which is close to Quora state of the art accuracy.
