{'fraction_non_alphanumeric': 0.03392096248642326, 'fraction_numerical': 0.01269947364023728, 'mean_word_length': 4.94044665012407, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 2, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
DUTH at SemEval-2019 Task 8: Part-Of-Speech Features for Question Classification

This report describes the methods employed by the Democritus University of Thrace (DUTH) team for participating in SemEval-2019 Task 8: Fact Checking in Community Question Answering Forums. Our team dealt only with Subtask A: Question Classification. Our approach was based on shallow natural language processing (NLP) pre-processing techniques to reduce the noise in data, feature selection methods, and supervised machine learning algorithms such as NearestCentroid, Perceptron, and LinearSVC. To determine the essential features, we were aided by exploratory data analysis and visualizations. In order to improve classification accuracy, we developed a customized list of stopwords, retaining some opinion- and fact-denoting common function words which would have been removed by standard stoplisting. Furthermore, we examined the usefulness of part-of-speech (POS) categories for the task; by trying to remove nouns and adjectives, we found some evidence that verbs are a valuable POS category for the opinion question class.


Introduction
The significance of Community Question Answering (CQA) forums has risen in the past years. Such forums represent a modern need for information that comes with the abundance of online sources and the needs of millions of people for answers. Popular forums like StackOverflow, Yahoo! Answers, and Answers.com provide platforms for general or specific questions in a wide range of topics by users' and also a communitybased model for user interaction.
The large numbers of questions and answers located in these forums generate many opportunities for information retrieval and data mining applications, such as query-intent detection, opinion mining, fake news classification, etc. (Tsur et al., 2016;Jo et al., 2018;Sethi, 2017). More advanced applications do not only aim at analyzing opinions but-by categorizing the feelings of the Q&As-they may be able to detect inappropriate content such as hate speech and act accordingly (Karadzhov et al., 2017;Baly et al., 2018).
The SemEval Task 8, Fact Checking in Community Forums, aims to determine whether the answers that are provided for a question in a forum are true or false. While answers to factoriented questions can be deemed true or false, opinion-oriented and socializing questions evoke answers for which a true/false categorization does not make much sense. As a result, determining the question type is a necessary first step. Consequently, the subtask A of SemEval Task 8 has the goal of classifying questions in three categories: opinion, factual, or socializing.
The rest of this report is structured as follows. Section 2 reviews some previous studies for CQA classification. Section 3 describes our system, while Section 4 presents experiments and results. Conclusions are summarized in Section 5.

Related Work
In recent years, plenty of research work examined the problem of classifying texts of CQA forums. Some related work which we found useful or inspiring are mentioned below. Mihaylova et al. (2018) proposed a novel approach based on multi-faceted modeling of facts, which integrates knowledge from several complementary sources, such as the answer content (what is said and how), the author profile (who says it), the remainder of the community forum (where it is said), and external authoritative sources of information (external support).
Another study which provided us with helpful information about the importance of feature se-lection on the development of a question classifier was by Huang et al. (2008). They demonstrated the importance of using the wh-word (what, which, when) in question classification. Such words are commonly disregarded and used in stopwords lists. Our approach is also trying to use features such as imperative verbs that indicate an opinion.
The SemEval-2015 Task 3, Answer Selection in Community Question Answering, targeted to classify comments in a thread as relevant, potentially useful, or bad, concerning the thread question (Nakov et al., 2015). This task encouraged solutions for the question classification problem that involved semantic or complex linguistic information.
Finally, (Mihaylova et al., 2016;Baldwin et al., 2016;Franco-Salvador et al., 2016) participated in subtasks A, B, and C at SemEval-2016 Task 3 that involved tasks for Question-Comment Similarity, Question-Question Similarity, and Question-External Comment Similarity. They proposed classification models and provided results that highlighted the importance of lexical and semantic features.
The aforementioned studies help to identify 'gaps' in this research topic and ways to attempt new and different approaches for question classification.

System Description
In this section, we give the details of our question classification model, applied pre-processing techniques, as well as some statistics and visualizations for the dataset of the task.

Dataset
The organizers provided the dataset in an XML format. The given training set consisted of 1,118 questions for Subtask A that were selected from the Qatar Living forum.
We used Python's Element Tree library to parse and isolate specific content from the XML. The interesting tags to select were RelQBody (the question) and RELQ FACT LABEL (labeled question by organizers).
Before pre-processing, an exploratory data analysis gives us the opportunity to better understand the dataset. Because we will develop a multipurpose model that classifies not only the opinion but fact and socializing questions, it is helpful to understand in depth the character of the questions.
A way to understand the contents of the forum is to examine Table 1 where almost 50% of the questions are opinion oriented. Also, Figure 1 presents the most common words in opinion questions.

Label
Number of Questions Opinion 563 Factual 311 Socializing 244

Pre-processing
To reduce the noise of the text, based on the results of Symeonidis et al. (2018), we applied the following pre-processing: • Remove Numbers • Replace all URL addresses, normalizing them to 'URL' Figure 2 shows the most frequent words on the dataset as a wordcloud. The final steps of pre-processing are tokenization and stemming. A basic process in NLP is to identify tokens or those basic units which need not be decomposed in subsequent processing.
The entity word is one kind of token for NLP (Webster and Kit, 1992). Stemming is a process of reducing words to their stems or roots to reduce the vocabulary size and manage the case of data sparseness (Lin and He, 2009). For example, conjugated verbs such as 'goes', 'going', and 'gone' are stemmed to the term 'go'. We used Python's SpaCy 1 library to tokenize the text and convert it to lemmas. This function also removes symbols (or punctuation) such as ' [','...','-'. Stopwords are frequent words that appear in the text, but they can have an impact on retrieval efficacy. The removal of stopwords also modifies the document length and subsequently affects the weighting process and efficiency during processing of the collection (Kwok, 1998).
For our task, we found out that the most commonly-used stopword lists contain words that can be helpful. For example, the word 'believe' is included in most stopword lists. While it is a ubiquitous word, it may also indicate an opinion; therefore, it can be useful for our purpose. In order to tackle this problem, we made a custom stopword list that only removes pronouns such as 'i','he','she', etc. NLTK's 2 list of English stopwords used as guideline and contained 127 words.
Although there is an abundance of stopwords lists that contain even more words we used a small one on purpose. We wanted to eliminate words from our dataset that would not bear any significance in our task. The next step, based on the vocabulary of the dataset, was to manually find words that could help us identify whether the question is opinion oriented, factual, or socializing. We excluded, from the NLTK's stopword list, words such as 'believe', 'think', 'mean', 'consider', and others. Our final revised stopword list consists of 50 words.

Experiments
This section summarizes our experiments in the context of SemEval 2019 Task 8 Subtask A. Beyond our officially submitted runs, we present 1 https://spacy.io 2 https://www.nltk.org/ some additional experiments that although they did not perform very well, there seems to be a promising room for improvement in the future.

Machine Learning Methods
For the training of our classifiers, we used Python's Scikit-Learn library (Pedregosa et al., 2011). We split the dataset into 749 training questions and 369 testing questions, i.e. a typical 2/3-1/3 split (ratio 2:1). After the split, the questions in the training set were shuffled for training. With the class sklearn.pipeline, we performed a sequence of different transformations and parameters.
Vectorizer: We compared three common vectorizers such as CountVectorizer, HashingVectorizer, and TfidfVectorizer. Finally, our selection was the TfidfVectorizer since it yielded the best results when it comes to accuracy. The TfidfVectorizer converts a collection of raw documents to a matrix of tf-idf weighted features.
Classifiers: We experimented with various classifiers, and decided to use the following three since they yielded the best accuracy results.
• NearestCentroid: Each class is represented by its centroid, with test samples classified to the class with the nearest centroid.
• Perceptron: It is a simple and efficient algorithm to fit linear models, and suitable for very large numbers of features.
• LinearSVC: An SVM algorithm which tries to find a set of hyperplanes that separate space into areas representing the classes. The hyperplanes are chosen in a way to maximize the distance from the nearest data point of each class.

Results
The official run that we submitted for the competition proved to be the most successful. In the following tables, we present the produced test results by using the three different classifiers, the Tfid-fVectorizer, and the custom stopword list. The results are shown in Tables 2, 3, and 4. We can observe that the most accurate classifier overall is the NearestCentroid.
We experimented further based on the hypothesis that opinion classification can be more effective by using only verbs. Until recently, most    patzis et al., 2000), where it was found that a traditional keyword-based indexing set can be reduced to retain only its nouns and adjectives without hurting effectiveness, even slightly improving it. Nevertheless, the aforementioned work was on topic classification; later, Karamibekr and Ghorbani (2012) showed that verbs are vital in classifying opinion terms, particularly in social domains. We conducted two experiments by removing either nouns or adjectives from our dataset to help our classifiers adjust mostly on verbs. We can observe, in Tables 5 and 6, that classifiers achieved a better accuracy score when it comes to opinion as opposed to fact and socializing questions. Nevertheless, by removing either nouns or adjectives, there is an overall drop in effectiveness in all classes. Thus, there is evidence that verbs are a useful part-of-speech category for opinion classification, but they are not sufficient by themselves.
Our official submission to the competition ranked our team to the 16th place from 22 teams.    The results of our model are shown in Table 7.

Conclusions
We presented a supervised learning model for classifying questions from online Q&A forums in three categories: factual, opinion, and socializing. We used standard pre-processing techniques, and made a custom stopword list to tackle the specific task at hand. Using standard classification methods, we achieved satisfactory and promising results. We also tried to use verb-oriented feature sets for classification which although they provided mixed results it seems that they can be improved.