{'fraction_non_alphanumeric': 0.01957585644371941, 'fraction_numerical': 0.021750951604132682, 'mean_word_length': 4.822784810126582, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
SMS based FAQ Retrieval for Hindi, English and Malayalam

This paper presents our approach for the SMS-based FAQ Retrieval monolingual task in FIRE 2012 and FIRE 2013. Current approach predicts the matching of an SMS and FAQs more accurately as compared to our previous solution for this task which was submitted in FIRE 2011. We provide solution for SMS and FAQs matching in Malayalam language (an Indian language) in addition to Hindi and English this time. In order to perform a matching between SMS queries and FAQ database, we introduce enhanced similarity score, proximity score, enhanced length score and an answer matching system. We introduce the stemming of terms and consider the effects of joining adjacent terms in SMS query and FAQ to improve the similarity score. We propose a novel method to normalize FAQ and SMS tokens to improve the accuracy for Hindi language. Moreover, we suggest a few character substitutions to handle error in the SMS query. We demonstrate the effectiveness of our approach by considering many real-life FAQ-datasets provided by FIRE from a number of different domains such as Health, Telecom, Insurance and Railway booking. Experimental results confirm that our solution for the SMS-based FAQ Retrieval monolingual task is very encouraging and among the top submissions which performed very well for English, Hindi and Malayalam. The Mean Reciprocal Rank (MRR) scores for our approach are 0.971, 0.973 and 0.761 respectively for English, Hindi and Malayalam SMS-based FAQ Retrieval monolingual task in FIRE 2012. Furthermore, our solution topped the task for Hindi language with MRR score equal to 0.971 in FIRE 2013. Our approach performs very well for English language as well in FIRE 2013 despite transcripts of the speech queries are included in test dataset along with the normal SMS queries.