{'fraction_non_alphanumeric': 0.035469398120714465, 'fraction_numerical': 0.017692372809616524, 'mean_word_length': 4.73216885007278, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
DeepBlues@LT-EDI-ACL2022: Depression level detection modelling through domain specific BERT and short text Depression classifiers

We discuss a variety of approaches to build a robust Depression level detection model from longer social media posts (i.e., Reddit Depression forum posts) using a mental health text pre-trained BERT model. Further, we report our experimental results based on a strategy to select excerpts from long text and then fine-tune the BERT model to combat the issue of memory constraints while processing such texts. We show that, with domain specific BERT, we can achieve reasonable accuracy with fixed text size (in this case 200 tokens) for this task. In addition we can use short text classifiers to extract relevant text from the long text and achieve slightly better accuracy, albeit, trading off with the processing time for extracting such excerpts.


Introduction
Depression has been found to be a major cause behind at least 800,000 deaths committed through suicide each year worldwide 1 . Moreover, It has been found in earlier research that depressed individuals show help seeking behavior through their social media posts (Guntuku et al., 2017). So analyzing social media posts for depression detection is an important research area (Coppersmith et al., 2014;Mowery et al., 2017). In our work, we analyze Reddit social media posts to identify whether a particular post exhibits either of three levels of 1 https://who.int/mental_health/ prevention/suicide/suicideprevent/en/ depression, including (1) No Depression, (2) Moderate Depression, and (3) Severe Depression, as a part of a shared task challenge (Sampath et al., 2022). We use a state-of-the-art transformer-based model called BERT, which was pre-trained on mental health related social media data. Further, we compare our model with two variations of the same, with other models trained on (1) relevant excerpt extracted Reddit posts and (2) a subset of depressive sentences in Reddit posts calculated with the help of short text classifiers. In the next sections, we elaborate on each of our strategies.

Depression Level Detection through
Fine-tuning Mental BERT (MBERT) BERT (Devlin et al., 2018), which stands for Bidirectional Encoder Representations from Transformers, has been found to be very effective in different downstream NLP tasks such as, text classification (Sun et al., 2019) and Depressive post detection (Ji et al., 2021). Here we use a mental health pre-trained BERT model, called Mental BERT (MBERT), which was pre-trained on several mental health forums under Reddit (Ji et al., 2021). Further, we fine-tune this model on the provided training dataset (Kayalvizhi and Thenmozhi, 2022) for this shared task. Since fine tuning BERT based models on longer text requires significant memory resources, we limit our text data to the first 200 tokens, which covers around 70% of the total samples provided. Before feeding input to our model, we convert all texts to lower case, and use an uncased version of the MBERT model for fine tuning. We also experiment with further enhancement of our classifier by fine tuning it through a selection of 200 "relevant" tokens from constituent Depressive sentences for a post from the training sample, which are longer than 200 tokens and also depression-indicative. In addition, we investigate whether the distribution of constituent depressive sentences in each posts also have some predictive power for this task.

Extracting Relevant Excerpts for Fine-tuning Mental BERT (RE-MBERT)
To extract relevant excerpts, we use a majority voting classifier (MVC) which is built using four depressive short text or Tweet classifiers. Three of these classifiers use different pre-trained word and sentence embeddings and represent each sentence through either averaged embedding of all the constituent words of that sentence, or the sentence embedding of the sentence itself. The left classifier uses Zero-shot modelling for classifying each sentence for signs of depression. Description of these classifiers including the datasets they were trained on, have been previously described (Farruque et al., 2019(Farruque et al., , 2021. Since we cannot extract more than 200 tokens for each of our posts and, within those 200 tokens, we may not have all the relevant tokens which are important for this task, we plan to extract relevant (or depressive) constituent sentences or excerpts from the posts which have more than 200 tokens and which are labeled as either carrying signs of "Moderate" or "Severe" depression in the training set. To do this, we parse each post by exploiting punctuation, i.e. ".", "?" and "!" to find its constituent sentences. We then feed those sentences to MVC, where the above mentioned four short text classifiers vote to indicate whether the constituent sentences of a post are depressionindicative or not; we only take a sentence as a representative text for depression if at-least three of those four classifiers agree. We apply the same short text pre-processing as we did while training our short text classifiers to clean each sentences within our posts. In this cleaning process, with the help of a python library named "Ekphrasis" (Baziotis et al., 2017), we re-contract word contractions, replace elongated words in their original form, convert all to lower case and remove nonwords, so our cleaned sentence is mostly regular words separated by spaces. Finally after fine-tuning our MBERT classifier (see Figure 1), we infer the labels from the provided test set and use extracted excerpts only for the posts having greater than 200 tokens (see Figure 2). In summary, in our training set, we only extract excerpts when a post is depression-indicative and longer than 200 tokens. If no excerpts are extracted or the post is less than 200 tokens long or it is not depression-indicative, then we use the cleaned version of the original post and feed it to our MBERT classifier. After this procedure is completed, the total posts left beyond 200 tokens were 1667 which is more than a 50% reduction than the original number of posts having more than 200 tokens (i.e., 4018). All the posts from the original training set beyond 200 tokens and with depression indication are now pre-processed so that those now contain only depressive excerpts which is important for our classification. Our assumption with the posts with less than or equal 200 posts is that, they have more depressive sentence density than their longer counter-parts.

Depressive Sentence Proportion based Method (SPROP)
We calculate the number of sentences which are depression-indicative out of total number of sentences in a post: we call this the Depressive Sentence Proportion Value (DSPV). In our training set, we calculate DSPV for our depressive posts and we assume this to be 0 for the "No Depression" class, as ideally there would not be any depressionindicative sentences or might be too few such sentences present in this class. Later we use this as a feature extracted from all our training samples and train our model and report result in test samples based on same extracted feature.

Experimental Setup
We mix the training and development set provided in the shared task data, which is a total of 13,387 samples, and then split it into a training set of size: 12,589, and validation set of size: 128, and held-out set of size: 670 samples. For the MBERT classifier we use uncased mental BERT 2 . We use maximum token size = 200, number of epochs = 10, training and test batch size = 16. We employ a NVIDIA-GeForce-RTX 3070 GPU with 8 GB of integrated memory and 32GB of RAM. For SPROP, we use a MLP classifier 3 with default settings and max iteration value of 300, during label inference time we take the argmax of the output label probabilities using predict_proba() function.
Although BERT-based modelling takes around 30 minutes for training and testing, excerpt extraction for creating RE-MBERT takes a number of days to complete.
In the next section we report and analyze the performance of our top models, i.e. MBERT and RE-MBERT.

Result Analysis
We report both label based accuracy for our heldout set over all samples (see Tables 1 and 2) and overall accuracy scores (i.e., Avg. Precision, Recall, Weighted-F1 and Macro-F1 across all labels over all samples, see Table 3) for our held-out set. Also, we report overall accuracy scores (i.e., Avg.

Classes
Precision   Table 4). From the F1-scores for different labels in the heldout set, we see added value to the classification performance for "No Depression" and "Severe Depression" classes (see Tables 1 and 2). For "Moderate Depression" class there is still some improvement but it is not very pronounced (only 0.26%). Increased recall in "No Depression" and "Severe Depression" classes (by almost 4.5% and 8%) indicates that the classifier learns a high false positive rate or is more inclined to erroneously identify a post as either not depressive or severely depressive through our training procedure. However, for our "Severe Depression" class, our classifier also achieves better precision scores, means robustness against false negative results. For "Moderate Depression" class we also see 1.6% precision improvement but with a cost of 1.73% decrease in recall.
We can see the reflection of these results in Table 3, with RE-MBERT having significantly better Macro-F1 score due to pronounced recall for "No Depression" (by 3.14%) and "Severe Depression" (by almost 6%) classes.
In the test set accuracy scores in Table 4, we see our strategy (RE-MBERT) helps in achieving a slightly better Macro-F1 score (by 0.3%) whereas the precision score improvement is more pronounced (by 1.8%) than recall compared to MBERT. Additionally, improvement in the Weighted-F1 score (by 1%) suggests that our strategy helps improve the F1-score for one of our Depression level classes. Unfortunately, since we do not have access to test set labels we cannot do detailed label-wise error analysis. We also test   with a single feature SPROP method, which results in Macro-F1 score of 0.3387 in test set. We found SPROP is more robust for more populated classes such as "Moderate Depression" and "No Depression" and performs poorly for the "Severe Depression" class. This seems reasonable because a single feature has less predictive value. We tried that method just to observe whether depressive sentence proportion as a feature has any significance or not. In future, we would like to use this with other features in future to make our modelling robust. Finally, our MBERT modelling does not perform data cleaning as RE-MBERT and SPROP. We find that data cleaning does not provide any significant performance gain, by comparing MBERT trained with cleaned and not cleaned samples. With data cleaning, we achieve only 0.48% accuracy gain in the held-out set. Therefore we believe that the accuracy increase in the held-out and test set for our RE-MBERT modelling is purely attributed to our excerpt extraction algorithm. The Held-out set sample distribution is similar to our training set, which explains why we have better accuracy scores there.

Conclusion
We have described a few strategies for the Depression level detection shared task from Reddit posts. We use state-of-the-art mental health data pre-trained BERT model (MBERT) and further fine-tune it with the shared task data and achieve 7th position in terms of Macro-F1 score and 3rd position in terms of Weighted F1 score compared to 30 other participating teams. We also present a strategy (RE-MBERT) to consider while training MBERT in a resource constrained environment through a subset of relevant sentence selection for longer posts. Our strategy shows some improve-ments in both training and test set which is stimulating and encouraging.