{'fraction_non_alphanumeric': 0.013363028953229399, 'fraction_numerical': 0.0, 'mean_word_length': 4.762820512820513, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
A Statistical Language Modelling Approach for Question Answering System

This paper concerns about Question Answering system in which a statistical language modeling approach is used. The main objective is to build a simple system for question answering without the need for highly tuned linguistic modules which need more human work and is very difficult to find any bugs if any. A mathematical model for answer retrieval and answer extraction is derived, which does not use any linguistic information or annotated data. It makes use of word tokens and web data. We take a statistical, noisy-channel approach and treat QA as a whole as a classification problem. We present a fully data-driven mathematical model for estimating the probability of a candidate answer given a question. In doing so we largely remove the need for ad-hoc weights and parameters that were a feature of many TREC systems.