{'fraction_non_alphanumeric': 0.026722176501912186, 'fraction_numerical': 0.00987558696809798, 'mean_word_length': 5.241087613293051, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
A Dataset for Research on Modelling Depression Severity in Online Forum Data

People utilize online forums to either look for information or to contribute it. Because of their growing popularity, certain online forums have been created specifically to provide support, assistance, and opinions for people suffering from mental illness. Depression is one of the most frequent psychological illnesses worldwide. People communicate more with online forums to find answers for their psychological disease. However, there is no mechanism to measure the severity of depression in each post and give higher importance to those who are diagnosed more severely depressed. Despite the fact that numerous researches based on online forum data and the identification of depression have been conducted, the severity of depression is rarely explored. In addition, the absence of datasets will stymie the development of novel diagnostic procedures for practitioners. From this study, we offer a dataset to support research on depression severity evaluation. The computational approach to measure an automatic process, identified severity of depression here is quite novel approach. Nonetheless, this elaborate measuring severity of depression in online forum posts is needed to ensure the measurement scales used in our research meets the expected norms of scientific research.


Introduction
With technological advancement, online support forums (OSF) are popular among society and attempt to seek solutions for their physical, social and psychological problems (Dosani, Harding, and Wilson 2014). As subgroups of OSF, online mental support forums (OMSF) are designed primarily for psychological expertise and experienced peer groups to provide support and guidance to the psychologically impaired community.
Early identifying and analysing the severity of depression is crucial for: (a) providing suitable medications and therapeutic processes immediately after diagnosing (b) preventing fatal accidents such as deliberate self-harms or suicides. Depression is greater than definitely feeling unhappy or bored stiff for a few days. The majority undergo durations of feeling down, but whilst you're depressed you experience persistently unhappy for weeks or months, in preference to only some days. The best information is that with the proper treatment and support, the majority with depression could make a complete recuperation (MacKin et al. 2014).
To prevent from those tragic situations, in clinical psychology several psychological assessments such as DSM -V, BDI were defined to measure the severity of depression disorder. Among those assessments, DSM-V criteria are widely used to diagnose clinical depression disorder and have been classified into different severity levels according to their number of symptoms, duration and cause for depressive episodes. Table 1  Recent work on modelling social media data in Natural language processing (NLP) has focused on distinguishing depressive posts from nondepressive posts (Islam et al. 2018;Jeri-Yabar et al. 2019;Shrestha and Spezzano 2019). Moreover, limited datasets are available due to the difficulties such as complex nature, time consumption, etc, that exist in the process of data analysing. However, there are no previous studies to access the severity of depression based on OMSF data. This paper presents a systematic study on analysing the severity of depression disorder from forum posts with computational methods and the following are our main contributions: • Preparing dataset with four severity levels of depression using automatic labelling method. • Create a new classification task for identifying different severity levels of depression from forum posts. • Evaluate deep learning algorithms combined with linguistic information.
The rest of this paper is structured as follows: Section 2 gives a brief overview of existing and similar approaches used to monitor the severity of depression; Section 3 presents our aim and motivation to conduct the research; Section 4 outlines the proposing data and tasks designed for automatic labelling process and the results obtained; Finally, the paper is concluded with Section 5.

Psychological Assessments
Depression diagnosis requires five or more symptoms according to the Diagnostic and Statistical Manual of Mental Disorders (DSM-V) (Medicaidmentalhealth.org 2018). One of them must be either Depressed mood or Anhedonia, named main criteria. Although the secondary symptoms can be divided into somatic and nonsomatic clusters, the DSM-5 identify depression in all or none fashion. In contrast, depression severity is a continuous variable (American Psychiatric and Association 2013). The Beck Depression Inventory (BDI) is widely used to screen for depression and to measure behavioral manifestations and severity of depression. The BDI can be used for ages 13 to 80. The inventory contains 21 self-report items which individuals complete using multiple choice response formats. The BDI takes approximately 10 minutes to complete. Validity and reliability of the BDI has been tested across populations, worldwide (Beck et al. 1961).
The Center for Epidemiologic Studies Depression Scale (CES-D) was designed for use in the general population and is now used as a screener for depression in primary care settings. It includes 20 self-report items, scored on a 4-point scale, which measure major dimensions such as restless sleep, poor apatite and feeling lonely of depression experienced in the past week. The CES-D can be used for children as young as 6 and through older adulthood. It has been tested across gender and cultural populations and maintains consistent validity and reliability. The scale takes about 20 minutes to administer, including scoring (Radloff 1977). The CES-D Scale: A self-report depression scale for research in the general population (Wright 1985). The Hamilton Rating Scale for Depression (Hamilton 1960), abbreviated HDRS, HRSD or HAM-D, measures depression in individuals before, during and after treatment. The scale is administered by a health care professional and contains 21 items, but is scored based on the first 17 items, which are measured either on 5-point or 3-point scales. It takes 15 to 20 minutes to complete and score.
The Children's Depression Inventory is a modification of the BDI for adults. The CDI is now on its second edition. It assesses depression severity in children and adolescents 7 to 17 years old. Two scales measure emotional problems and functional problems. Three separate rater forms are available: one for parents (17 items), one for teachers (12 items) and a self-report (28 items). Administration time is between five and 15 minutes (Kovacs 1981

Linguistic Features in Depression
One of the functions of language is to communicate our inner monologue to others. Depressed persons, predictably, utilize more negative emotion words and fewer positive emotion words than nondepressed people. Depressed college students, for example, used more negatively valenced phrases when writing about their "deepest thoughts and feelings about being in university". Following is an example for language characteristic of people diagnose with severely depressed.

"In May I was admitted into hospital with Severe
Depression. This was my first ever admission and was discharged after a long hard 9 weeks. Since being home, I seem to have some really good days, then suddenly start getting feelings of sadness, insomnia, and feeling worthless.

Workcover is now looking at a return-to-work project and I am so worried at what will happen when one day I am coping and the next plotting my death. Do others experience these highs and lows, and how do manage day to day"
Language style refers to how we express ourselves rather than the content we express. "Absolutist words"words that convey absolute magnitudes or probabilities, such as "always", "nothing", or "completely"were discovered to be better markers for mental health forums than pronouns or negative emotion words (Holtzman 2017).

Depression and Emotion words
People suffering from depression may not be able to identify it by name; they rarely used the word "depressed" when describing their feelings, instead opting for words like "down," "stressed," or "upset". So, the key is to spot a pattern over time many words from depressed children and adults, such as bad, sad, helpless, hopeless, aching, lost, worthless, useless, stupid, stuck, untethered, adrift, hurting, alone, afraid, unsure, insecure, despair, black and blue, to name a few," These words are frequently used when describing themselves or their personal feelings (Houben, Van Den Noortgate, and Kuppens 2015;Palmier-Claus et al. 2012). It is often supported by utilising various machine learning approaches to different NLP techniques (Tadesse et al. 2019a(Tadesse et al. , 2019bZhang et al. 2015). Moreover, some studies employed classification algorithms like Decision Trees (Islam et al. 2018), Naive Bayes (Biyani et al. 2014), and Random Forest (Cacheda et al. 2019;Shrestha and Spezzano 2019). With recent advances in neural network models in NLP, a novel contribution for identification of depression has arisen with the implementation of advanced deep learning architectures to outperform more traditional machine learning methods.

Forum Data & Depression Identification
When assessing the severity of depression, there were fewer studies compared to depression identification. (Fatima et al. 2018) developed a framework to classify the degree of depression in social communities based on the mood tags (ex: Rejected, surprised, lazy). The framework used the ANEW lexicon (Margaret M. Bradley and Peter Lang 1999) and valence value to calculate the severity of depression. However, in real life, clinical experts use psychological scales and more 147 factors to judge the severity of depression and those specific terms not included in the ANEW lexicon. Also, another research has conducted to measure severity of major depression based on self-reports (R.C. et al. 2016).
We first described the psychological scales and their linguistic patterns used for the severity assessment of depression. Then we mentioned studies that use different psychological assessments to capture depression in the online forum posts. We also discuss the approaches used for depression identification related to taxonomies, symptom associations with linguistic styles, and machine learning techniques. According to our analysis, most of the research works focused only to identify the depressive contents expressed by depressed individuals. However, in this paper, we propose a model for predicting the severity of depression as an important factor in determining the treatment type and urgency.

Motivation
In previous studies, forum data was annotated as depressive or non-depressive posts and but with no information regarding the severity level of depression in each post based on the individual's thought process.
We aim to build a level of severity for each forum post, annotated dataset for depression severity assessment using an automatic annotation approach which enables us to extract evidence related to depression. Moreover, to predict the level of depression severity from posts available in OMSF.

Tasks & Data
We selected four different OMSF to extract posts tagged under the category of depression. Following standards were considered in the selection process of forum posts.
• Post should consist minimum of 2 sentences. We have defined two tasks to carry out in the process. The first task is designed for automatic labelling of OMSF data into different severity levels. The second task is defined as a multi-label classification task.

Depression Severity Categories
In automatic depression severity annotation, we adapt the criteria defined by DSM-V and Beck's Inventory. Table 1 illustrates the scores assigned according to the symptoms of depression. Note that the severity levels of depression were classified according to the symptoms that exist on the forum post.

Text Processing
We used four OMSF to extract the posts and all the posts were posted under the depression category.
• Beyond Blue Online Forum • Understand Forum • 7 Cup Forum • Depression Forums A specific web scraper is developed from scratch with python libraries to extract online forum data. After the extraction of following features, data were stored in a relational database. After the collection of dataset, unwanted texts such as formatting, HTML links and quotations of earlier comments were removed (Petz et al. 2012). Table 2

Depression Severity Annotation
Though we collected posts related to depression from different OMSF, some forums posts were irrelevant to depression disorder. Therefore, the dataset has been annotated manually by two psychological experts and one computational linguistic expert. The severity annotation process followed a similar procedure to that defined by Moreno (Moreno et al. 2011). Three annotators were asked to determine which posts can be considered as a reference to depression following the DSM-V criteria and to remove non-depression posts from the dataset. The three annotators achieved a pairwise Cohen's Kappa score ranging between 0.62687 and 0.7192. But achieving a high inter-rater agreement can be a difficult task due complex nature of the posts. Also, 215 non-depressive forum posts were removed, and 157 posts were selected based on majority voting method between the annotators. Manually annotated dataset were used mainly for validating purpose.
The dataset consists of 2140 data points that remained after data filtration and were subjected to automatic data labelling. According to the defined criteria of Table 1, we used domainspecific keywords and sentences (DSM-V and Beck's inventory) for the process of automatic labelling (AL). Two different computational approaches were designed and implemented.
• Based on the taxonomical similarity and ngram models (AL1). • Based on the sentence similarity and pretrained transformer models (AL2).

Lexical similarity and n-gram models (AL1)
In the first approach, we created a lexicon which based on the symptoms of depression defined in DSM-V criteria. In order to build the lexicon, previous taxonomies associated with depression, gathered from online vocabularies were taken into consideration. Moreover, all the pronouns and the negations were included on this lexicon with the help of POS tags generated by Stanford Parser. Furthermore, all the possible sets of words and synonyms relative to symptoms of depression were extracted from Wordnet and added into the lexicon. The lexicon includes all the words relevant to the symptoms which belong to different severity levels. To be more accurate, we validated our lexicon with the aid of clinical psychologists (co-authors of the paper).
N Gram Analysis: We considered lexicon as a unigram model to calculate the word level similarity. Based on the severity levels in depression, we have divided the lexicon into three categories and assigned labels for each category (1, 2 and 3). We use cosine similarity and extract string matching between each post and each category of depression severity lexicon. The goal was to obtain a ranked list of the posts according to the depression severity with the lexicon. Thus, the higher ranked forum posts were those with the highest severity level of depression.

Sentence Similarity and Pre-trained Transformer models. (AL2)
In the second approach we have selected Beck's Depression Inventory to measure the degree of severity. The Beck Depression Inventory (BDI) is a 21-item test presented in multiple-choice format, which measures presence and degree of depression in adolescents and adults consistent with the DSM-IV. It is not intended as a diagnostic instrument. The BDI evaluates 21 symptoms of depression, 15 of which cover emotions, four cover behavioral changes, and six somatic symptoms.
Forum posts were split into sentences and similarities were checked according to BDI score using Universal sentence encoder model.

Universal Encoder Model: The Universal
Sentence Encoder encodes text into high dimensional vectors to measure semantic similarity. We trained our model using the sentences defined in Becks Inventory and assign a score for each sentence. And calculate an individual score using cosine similarity for the score. Figure 1 and Figure 2 illustrated the comparison of results obtained from the twodomain specific automatic labelling models and the manually annotated validation dataset. Following Figures demonstrate the distribution of posts relevant different severity levels.

Multiclass Classification Model
Since depression severity prediction is a new task, our goal is to classify and identify the forum posts that show some evidence of different severity levels. The effectiveness is evaluated using classic metrics such as Precision, Recall and 1.

Experiments and Results
We have selected different classification algorithms and common textual features to perform multiclass classification. Also, the features are the following: (1) Unigrams: represents the raw frequency count of the words extracted from posts; (2) Unigram-TFIDF: list all the relevant words extract from posts; (3) Word2Vec Embedding: captures all the words appear in similar posts; (4) Global Vectors (GloVe) Embedding: focuses on words cooccurrences over the forum posts; (5) Bidirectional Encoder Representation from Transformer (BERT) Embeddings: defines the contextual language representations derived from the posts.
In general, supervised model, input features are not abstracted in high-level and try to predict output by mathematical transformation on a subset of input features. But in deep learning all the abstraction of the input features are formed automatically and internally. Therefore, we have employed both supervised and deep learning classification algorithms for multiclass prediction based on the annotated dataset. Support Vector Machine (SVM), Multinomial Naïve Bayes (MNB), Logistic Regression (LR), Neural were used for multi classification task. Moreover, the classification models were trained on the automatically derived sets and evaluated using the validated set (manually annotated posts). We observe from the above results, an overall higher recall (Average recall) is achieved in approach AL1. Also, a higher precision (Average recall) was obtained in AL2. According to the nature of the proposed approaches, higher recall is usually preferable. Subsequently, 1 measure concludes that any proposed approaches can effectively use to automatically. A set of forum posts and used such set to train the classification model.

Conclusion
There has been growing research interest towards the identification changes of the mental state through the exploitation of social data. The absence of resources, such as datasets might limit the development of cutting-edge technologies to support health practitioners. We developed a dataset and automatic process for labelling the forum posts with four severity levels of depression.    Also, we modelled depression severity level prediction as a new multi-class classification task and conducted experiments using different deep learning-based algorithms combined with linguistic features. The linguistic features contribute to this study were obtained by considering characteristics like frequency, importance of words in given information.
According to the results obtained deep learning algorithms performs better than general classification algorithms, when analysing unstructured forum data. Since this research is ongoing research, we have considered only quarter of our dataset for this experiment. In future, we are planning to train this model with full dataset which requires huge computation time. Also, we are planning improve the overall accuracies of the models by optimizing the classification algorithms. Nowadays, emoji plays a fundamental role in human computer-mediated communications, allowing the latter to convey body language, objects, symbols, or ideas in text messages. The relationship between language, emoji, and emotions is now being studied by several disciplines such as linguistics, psychology, natural language processing (NLP), and machine learning (ML). Therefore, we hold the opinion that in the future, could also benefit from a more qualitative approach, such as interviews and case studies to learn about emoji use in the context of real-world communication. In the future, we plan to identify the progression of depression severity within the forum thread in OMSF.