{'fraction_non_alphanumeric': 0.027721993333471355, 'fraction_numerical': 0.008177884515848534, 'mean_word_length': 4.785363516588813, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 1, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Forecasting User Interests Through Topic Tag Predictions in Online Health Communities

The increasing reliance on online communities for healthcare information by patients and caregivers has led to the increase in the spread of misinformation, or subjective, anecdotal and inaccurate or non-specific recommendations, which, if acted on, could cause serious harm to the patients. Hence, there is an urgent need to connect users with accurate and tailored health information in a timely manner to prevent such harm. This paper proposes an innovative approach to suggesting reliable information to participants in online communities as they move through different stages in their disease or treatment. We hypothesize that patients with similar histories of disease progression or course of treatment would have similar information needs at comparable stages. Specifically, we pose the problem of predicting topic tags or keywords that describe the future information needs of users based on their profiles, traces of their online interactions within the community (past posts, replies) and the profiles and traces of online interactions of other users with similar profiles and similar traces of past interaction with the target users. The result is a variant of the collaborative information filtering or recommendation system tailored to the needs of users of online health communities. We report results of our experiments on an expert curated data set which demonstrate the superiority of the proposed approach over the state of the art baselines with respect to accurate and timely prediction of topic tags (and hence information sources of interest).


Introduction
Personal healthcare management increasingly requires accommodating a higher degree of involvement of patients and informal caregivers. Consequently, patients and their caregivers expect healthcare providers to provide more detailed information about the disease, prognosis, and treatment. However, there is a disconnect between healthcare providers and patients (and their caregivers) in terms of language used, and the information that is shared with the patients. For example, clinicians often communicate using medical terminology to describe the patient's medical condition, treatment plans, medications, medical procedures, etc. Patients have a hard time making sense of the information provided to them by their healthcare providers. The situation is not much better with information available through electronic health records (EHR) [6,14,19]. This situation is exacerbated by the fact that only 12% of U.S. adults have proficient health literacy [25,54]. Health literacy is especially low among older adults, minorities, groups with low income and socio-economic status, leading to increased healthcare usage and cost and worse health outcomes. Given the difficulty of comprehending health information available from healthcare providers or personal EHR, patients and caregivers increasingly turn to the internet and online communities. In such communities, patients and their caregivers interact with and seek information, e.g., regarding treatment options, side-effects of medications, etc., from their peers, in simple, lay terms.
While online communities can be useful sources of information and emotional support for patients, they also increase the risk of misinformation [35], or subjective [55], anecdotal and inaccurate or non-specific recommendations, which, if acted on by patients, could cause them serious harm. Hence, there is an urgent need to connect users with accurate and tailored health information in a timely manner to prevent such harm.
Typically, in an online community, a user seeking information posts a natural language query; Peers, or in some cases, subject matter experts (e.g., clinicians or nurses) respond to the query or point to useful sources of information [49]. Somewhat more savvy users may be able to hone in on appropriate keywords or search tokens (tags) to use in order to seek relevant information on the web (Google, Bing, etc.), medical information boards (WebMD, Mayo, etc.) or specialized platforms (HealthUnlocked, PatientsLikeMe, etc.). The keywords used in such queries are suggestive of the respective user's information needs. However, due to low health literacy, users may not necessarily be able to identify the right keywords to use; and it can be tedious and often frustrating for patients to sift through the results retrieved in response to their queries. Against this background, we explore a solution that aims to lower the barriers for patients to obtain accurate and understandable sources, e.g., articles written for laypersons that meet their health information needs.
In this study, we use data from an online community HealthUnlocked 1 , i.e., a social network designed to offer health-related information and emotional support for patients and their caregivers. An example of the data extracted from HealthUnlocked is shown in Figure 1. Though Figure 1 shows only one post, a typical user interacts a lot more with multiple posts and their corresponding keywords (tags). We observe that in such a setting, information needs of a user may involve medications, side effects, treatment options and other time critical concerns. These needs depend on not only the patient demographics and health condition, but also, where the patient is with regard to progression of the disease (e.g., stage of cancer in the case of cancer patients), or treatment being given (e.g. chemotherapy), or response to treatment (e.g., side effects, cancer remission).
The vast amounts of healthcare information available on the internet has led to the requirement of recommendation systems that help users navigate their care by enabling them with accurate information [1] [24]. Additionally, [8] discusses the role of such systems in reducing risks like misinformation and time delays in access to accurate medical information. [52] reviews the recommendation models including the usage of collaborative filtering like [23], [28] and [40] and finds its relevance in healthcare topic recommendation. Unlike other user-centric recommendation systems, healthcare based models are a lot harder to be subjected to collaborative filtering as there is high degree of individuality in the information sought by the users [7]. [38] however, proposes a CNN based feature aggregation approach to perform user-content based recommendation and presents improvements over personalization metrics. Similarly, [18] creates a similarity matching based on heterogeneous networks to perform collaborative filtering for healthcare topic recommendation.
Extracting features from text documents is at the centre of our proposed topic recommendation system. Text documents are in essence a sequence of vectors placed together with context and grammar. Early approaches to model text included the usage of recurrent neural networks (RNNs) like in [15] and [36]. It became evident that the order (positions) of these vectors in relation to their value was vital in text based tasks. Bidirectional long-short term memory models (LSTMs) including [20], [33] and specifically healthcare applications like [17] emphasized the importance of positional embeddings of the word vectors and improved performance on tasks like translation and sequence-to-sequence text generation. Transformers, a recent advancement by [45] along with their attention modules have revolutionised feature extraction like in [48]. Bidirectional Encoder Representations from Transformers (BERTs) an advancement of transformers have additionally  User Clustering Network interaction based [23] Based on disease progression similarity 5 Vulnerability Vulnerable to skewed tag distribution [44] Robust to tag distribution 6 Evaluation Metrics Precision@K (P@K), Recall@K (R@K) [18] Cosine similarity in feature space along with P@K, R@K aided in text based tasks like classification [13], [30], summarization [29], [34] and text generation [5], [27]. They are ideal for sentence/token generation conditioned upon prior context [12]. Table 1 compares our work in view of current literature. The position of current healthcare recommendation literature along with advances in natural language processing technology pose the ideal setting for a novel medical information recommendation system that predicts the future interests of users through topic tags.

Research Question
This study is designed to answer the following research question: Can the user profiles together with the traces of interactions of the user with the community (e.g., keywords used in past queries) and information from similar users be used to effectively predict contextually relevant topic tags associated with the future information needs of users? We hypothesize that patients with similar histories of disease progression or with similar courses of treatment would have similar information needs at comparable stages. In treating many diseases, e.g., cancer physicians follow a particular course of treatment that is customized based on the patient's profile and how they are responding to treatment. For example, the National Comprehensive Cancer Network (NCCN) has laid out guidelines for the treatment of non-small cell lung cancer [10]. If two users, say A and B, have similar histories of disease progression and treatment, we can expect that their information needs at comparable stages are likely to be similar. In Figure 2, we illustrate an example with two users who have similar histories of disease progression and treatment. In this case, the topic recommendations for user B can be improved by utilizing user A's past keywords. In this case, user B can be recommended articles related to "concurrent chemotherapy and radiation".
Based on the preceding discussion, we arrive at the following problem specification. For any given user U in the community, with a user profile B U , temporally ordered sequence of n prior posts {P U t−n , · · · , P U t }, encoded by the corresponding keyword sets {G U t−n , · · · , G U t }, and indexed by their position in the sequence as well as the past information traces of all the similar users in the network, the task is to predict the topic tags G U t+1 associated with a hypothetical future post P U t+1 that describes the future information needs of the user U . Disambiguation: "Tags" and "Keywords" can be used interchangeably in our work as they both describe the abstract token used to represent a larger body of text. However, we try to differentiate by using "Keywords" for tokens that have occurred prior to prediction and "Tags" as tokens that are yet to be predicted (future).

Contributions
We propose an innovative approach for predicting future information interests of participants in online communities as they move through different stages in their disease or treatment. We report results of our experiments on an expert curated data set from which we demonstrate the superiority of the proposed approach over the state of the art baselines with respect to accurate and timely prediction of topic tags (and hence information sources of interest).
The key technical contributions of the work include: • A novel approach to predicting keywords associated with the future information needs of users (patients, caregivers) in an online health community.
• A novel approach to collaborative-hybrid filtering based recommendation leveraging similarity of user disease progression timelines.
• A semantic evaluation of predicted tags along with the traditional metrics such as precision and recall.

Data
We obtain our data from HealthUnlocked.com (HU), which is one of the largest online health communities. Users of the platform interact with other users through an easy-to-use interface as they seek information or emotional support from their peers while managing their personal healthcare. A longitudinal study of HealthUnlocked has shown that it has positively impacted many patients [2] in terms of health outcomes and engagement in care. The website has several sub-communities organized around specific health conditions or diseases. In our work, we focus on three such communities designated for lung cancer patients and their caregivers: Lung Cancer Support , The Roy Castle Lung Cancer Foundation and British Lung Foundation. Our choice of the communities was motivated by three considerations. First, managing chronic conditions and treatment regimes associated with lung cancer presents information needs that can be met through sustained interactions with the community. Second, the progression of disease and treatment for chronic conditions impacts the information needs of users over time, which poses challenges that are not adequately addressed by existing approaches in recommendation systems, personal health libraries, etc. Lastly, these communities represent the largest fraction of the users of the online community. Each user has an opportunity to create a profile which includes demographic data, e.g., age, gender, etc. as well as the health condition. Once registered, a user then may pick one or more communities to join. Once in a community, users can post a "Main-Post" or post a "Comment" on an existing "Main-Post". Additionally, each "Main-Post" can have associated "keywords" at the time of posting. These user supplied keywords can be considered to be indicative of the information needs of the user at the time and act as ground truth for training and evaluating the proposed tag prediction system.
We acquired de-identified data from the lung cancer communities for the period between 2011 and 2020. From the three lung cancer communities, we have a total of 1711 unique users. Note that ethical (or IRB) approval was not needed for our study because the data were de-identified at source (at HealthUnlocked), there is neither direct interaction nor intervention with human subjects, and there is no means to link the de-identified data to specific subjects. For this study, we focus on the subset of anonymized users with at least one post and only valid posts with a minimum of three words or more are considered. The resulting data now filters down to 1032 unique users. Each user has an anonymized hashed User-ID that links their profile-bio, posts, and keywords to the user. Among the 1032 unique users, the average posts per user is about 2.4 (minimum is 1 and maximum is 68). Each post has on average 3.2 keywords (minimum is 2 and maximum is 10). About 55% (574) of the users have a profile-bio. An example of a user's information is provided in Figure 1.
Due to the unstructured nature of the text in our data set, identifying users whose information traces and temporal pattern of disease and treatment progression may be predictive of the future information needs of a target user presents significant challenges. To address this need, we restructured the user data as described below.

Structuring the information traces of users
We organize the information associated with each user, including the profile bio and the traces of the user's interaction with the community (information seeking posts or comments) as shown in Figure 3. Here we assume that a user U creates a profile-bio B before they post their first post P 0 . At time T −1 , we use the bio to predict tagsĜ 0 where ∧ indicates the prediction. This prediction is made before the first post has been created. As time progresses, the user posts P 0 at time T 0 with keywords G 0 , We then accumulate the profile bio B, the post P 0 and keywords G 0 to predict the tagsĜ 1 of a future unseen post P 1 . We continue to do this with a temporal window of P n posts in the past along with the current post to predict tags of a future post. At time T n , though we have n posts from the past, using all the posts and their keywords could add too much noise into the topic tag prediction system. We empirically question the influence of history (see Section 4.1.2) and determine the ideal P n. We then use only the last P n posts, their associated keywords, and the profile bio which remains unchanged throughout time T 0 to T n , as well as the current post and corresponding keywords to predict a future topic tagĜ n+1 . To create a training example, we need an input sequence (input text document) containing profile-bio, posts and keywords and a target sequence (Tags of future post). This is commonly known as sequence-to-sequence training. In our setting, we increase our training example pairs by extracting all possible pairs of data as shown in Figure 4. In order to maintain chronological patterns in the data, we pick pairs that have information from consecutive time steps. For example, if Post-1 and Keywords-1 are from the latest time step, the target can only be Keywords-2. This input set cannot be paired with Keywords-3 as Post-2 and Keywords-2 are not included the input. Using such a rule, a single user's timeline is converted into six chronological training pairs as shown in Figure 4. Following this rule, we obtain a total of 10,791 training pairs from the 1032 unique users. From this dataset, we use 8415 random pairs (78%) for training and the remaining 2376 (22%) pairs for testing.

Tag Statistics
Tag prediction can be approached through a tag-classification task. However, if the dataset has uneven tag distribution like ours, the model is more likely to fail due to high bias towards the more frequent tags. Additionally, with a large number of tags, classification based models tend to perform poorly. Our training set has 472 unique tags while the test set has 375 tags. Among the 375 tags, 18 of them are not in the training set and are completely new words. However, since these words occur in the posts of the users, they are a part of the vocabulary that the models can observe and thus are not deleted from the test set. Figure 5 shows the distribution of the top 20 tags in the test set. It is evident that 'Chemotherapy', 'Cancer and Tumors' occur several times more frequently than other tags. Tag prediction models designed around classification could incorrectly pick frequently occurring tags leading to poorer performance. Additionally, their metrics tend to be sensitive to the tag frequency in the training data. We address this issue of skewness and the resulting bias in Section 4.4.

Methodology
Recent advancements in the field of natural language processing (NLP) have opened a realm of possibilities due to a better understanding of context from text data. Bidirectional Encoder Representations from Transformers [9] have further allowed text representations to be learnt from vast sources like the Book Corpus and the English Wikipedia with more than 3300 million words. Their utility has been recognized in the bio-informatics community through the proposal of Bio-BERT [26], an extension of the BERT, trained over the PubMed and PMC medical word corpora. The extension of the pre-trained models helps in inferring context, more effectively than the traditional frequency based models, from text containing medically relevant words including those from online health communities. We leverage the Bio-BERT to extract illness-relevant information and predict future topics of interest based upon the users' needs.
Our pipeline is intended to extract context from the unstructured data and recommend accurate and time-relevant tags for each user in the online health community database. We make several modifications from works in the literature to accommodate our data and increase the overall accuracy of tag prediction. Among these, is our usage of Bio-BERT model as a sentence generator instead of a simple bag-of-words classifier. Additionally, we show the need for collaborative filtering but move away from traditional K-means method to a disease timeline based clustering method. In this section, we will describe our proposed tag prediction procedure, the model architecture, the objective function for performing the training and the metrics used to compare the models in our experiments.

Tag Prediction
Our primary objective in this work is to predict future topic tags for users in order to forecast their medical interests. Unlike existing tag prediction algorithms, we do not conform to a classification approach where a model picks the most probable tag from a set of tags. Instead, we perform an auxiliary sentence generation (ASG) task. The ASG task is similar to sentence completion where the last word is masked out using the [M ASK] token. In our set-up, the sentence generated is a set of comma separated future tags with no grammar or punctuation. We fine-tune a pretrained Bio-BERT model to read a given input document and pick a word from its entire vocabulary that fits the context and position of the mask. A similar training approach has been employed by [5] for generating questions based on a paragraph of context and an answer sentence. We, however, predict comma separated words which are actually the future tags. Since we are using a Bio-BERT model Tag generation, we refer to our model as BBERT T g . An example of the training process has been shown in Figure 6. First, we have a [CLS] token to indicate the start of the input document. The text in Orange is the profile-bio followed by a [SEP ] separation token. Similarly, in Blue, we have the past and current posts, the Green text corresponds to tags from previous and current posts, the tags from neighboring users or shared tags (see Section 4.2) are in Red and the predicted future tags are in Purple. We experimented with the positioning of these individual elements in the input document. However, they did not lead to a significant difference in the results. Having accumulated all the parts of the input document, we first clean the text through removal of stop words and lemmatization. Following this, the words are converted to fixed length vectors using a BERT word-embedding and tokenization module which converts the sentences into tokens based on the position of the words in the sentence and the position of the sentence in the input document. This encoded data is then fed to the Bio-BERT model along with the BERT-tokens The pre-trained Bio-BERT model uses the originally prescribed BERT base model with 12 layers, 768 hidden dimensions and 12 attention heads. We followed the training procedure set by [5] with the usage of an Adamax optimizer set with an initial learning rate of 5e-5. The training is performed over 15 epochs with 8415 examples while 2376 examples are used for testing. We use softmax as the final layer of the model where the Bio-BERT computes the probabilities of 200K+ words. Following literature, we adapt the Beam search [11] method to pick the top words and decode the predicted Bio-BERT output into regular words (tags).
Modification for Bio-BERT with long text Documents The traditional transformer model is meant for short sentences as computing attention over large input documents is computationally expensive and requires lengthy training. Our input documents range from about 14 words to 550+ words when multiple data sources like profile-bio, posts, keywords etc. are combined. We modify our fine-tuned model based on the recommendations provided by [3] where a "longformer" is proposed. This reduces the attention computation from quadratic to linear space. Gated Recurrent Units (GRUs) and Long-Short Term Memory (LSTM) models, though not entirely bidirectional in attention, are however capable of handling such large documents without any modifications.
In the next two subsections, we shall detail the training objective function and the metrics we use for evaluating the model.

Objective Function
In our sequence-to-sequence based learning approach the input sequence is the input document containing profile bio and posts and the output sequence is the target tags set. Both the sequences are assumed to come from a single long sentence where the target sequence is masked. We first tokenize the input sequence x into n tokens, {x i }, i = 1, 2, ...n, and predict the sentence fragment x p:q where p and q are positions such that p<q<n using the specially masked input which can be denoted as x /p:q . Unlike traditional tasks like translation, x does not have a paired y but we instead have to rely on the conditional probability of a token appearing next in the sequence given the past tokens (until the r-th position, p ≤ r ≤ q ) of the sequence. Let us denote the domain from which x is obtained as χ, the log likelihood objective to learn the model parameters θ then becomes: L(θ; χ) = 1 |χ| Σ x∈χ log P (x p:q |x /p:q ; θ) Expanding the conditional probability as a product of probabilities at each position between p and q, we obtain Equation 2. Here, r is the position at which the [M ASK] token is placed.
L(θ; χ) = 1 |χ| Σ x∈χ log q r=p P (x p:q r |x p:q <r , x /p:q ; θ) The position, or r, can be controlled to dictate the amount of prior information in the input document. In our experiments, we always have a non-empty set of tokens for prior data coming from either Profile Bios or Posts. However, Equation 2 can be used with null prior data where it will generalize into the OpenAI GPT model for generating text [41].

Metrics
To evaluate the tag generation model, we can view it as a recommendation system which recommends topic tags. This is similar to medication recommendation evaluation by [31]. We can generate any number of tags once the model is trained as this would only involve generating longer sentences. Traditionally, if K is the number of tags recommended, Recall@K and Precision@K are used to evaluate the success of the recommendation. In order to enable fair comparison using both Recall and Precision, we use the popular F1@K metric where K is the number of predictions made and F1 is calculated with equal weights to both recall and precision. We go beyond the traditional evaluation and calculate cosine similarity in the feature space between the recommended tags and the ground truth. This is primarily because the tag generation network is not picking words from a list but instead generating new words with a certain degree of variety. To calculate the cosine similarity, we first extract the word embeddings (features) of the K predicted tags and individually check their similarity to the word embeddings of the ground  truth tags. If the ground truth tag is 'radiotherapy' and the predicted tag is 'radiation therapy' the penalty will not be too large.

Individual User Model Ablation
In the first set of experiments, collaborative filtering is not performed and each user is an independent data point. We determined the ideal settings for the tag generation model based on the following variables- • The number of past posts (P U n ) to use for predicting future tags

Input Document Accumulation
To understand the importance of different elements in the data, we conduct three experiments with increasing amount of information presented in the input document. We start with the profile bio if available and then add posts and their corresponding keywords in the subsequent trials. For this experiment, we use 2 past posts along with the current post (P n =3). In the B U run (profile bio only), the model is expected to predict all future interests based on a fixed profile-bio which the user has provided during the creation of their account. In many cases, the user has interests unrelated to the profile-bio as they are not required to update this field regularly. However, the overarching context related to their condition is sometimes provided in the bios. Posts P U are comparatively more in tandem to the users' current interests. They describe what the user is actively querying and make predictions of the future tags easier. Keywords G U from the past and current posts additionally help the model understand abstraction between the post and their corresponding keywords. The average results with 4 runs each are presented in Table 2.

Influence of History
Our dataset is primarily based on chronic health conditions, therefore, utilizing the past information through past posts is vital for predicting future topics of interest. However, for users with many posts, accumulating all the posts may cause the network to pay attention to noisy topics occurring in the past. Additionally, extremely large documents have larger computational costs in terms of feature extraction. For this, we determine the ideal number of previous posts (P U n ) required. P U n =1 corresponds to using only the current post while P U n =4 accumulates the last 3 posts along with the current post. For this experiment, we use the B U + P U + G U (referred to as full mode) of data accumulation.

Model Improvement Through Collaborative Filtering
Our approach for model improvement comes from the intuition that "similar users seek similar information". Collaborative filtering is an approach to view a user as a parametric function of interests defined by other similar users in the community. Users within a group or a cluster tend to seek similar information or products. Multiple recommendation systems including [21] and [43] utilize collaborative filtering to extract a deeper understanding of user needs and provide recommendations based on other users in the group.
Identifying similar users is a crucial step in performing collaborative filtering. In our setting, with healthcare data, we identify similar users by matching relative disease timelines. Users with similar illness progression tend to search for similar information. Information search history of users who have already traversed the timeline can be utilized to recommend topics to users trailing along similar paths. We hypothesize that the similarity in treatment profiles for chronic conditions like non-small cell lung cancer (NSCLC), can be leveraged to recommend information at different time intervals based on the disease progression.
Neighbor Tags: For a query user U with a post P U and keywords G U posted on date T , we identify nearest neighbors h 1 , h 2 , ..h n using one of the three methods described below. For each neighbor h i , we check for posts that have occurred prior to T . Then the keywords associated with such a post will be referred to as Neighbor Tags and are appended to the input document after the posts and their corresponding keywords (see Figure 6). The entire input document is then used for future topic tag prediction.
In order to perform user similarity matching and collaborative filtering, we experimented with three approaches.
• Grouping of user profile-bios using K-means clustering.
• Using profile-bio text features extracted from Bio-BERT model for feature space clustering (t-SNE).
• Novel disease timeline based similarity matching of users.

K-Means Clustering
K-means is a well established method for performing clustering of users in a collaborative filtering setting for the purpose of recommendation [53]. In this baseline approach, we first performed primary cleaning through stop word removal and lemmatization. We then vectorized the words using scikit-learn [37] TfidfVectorizer package and used a gap statistic based elbow approach [42] to identify the ideal number of clusters. We experimented with 2 to 80 clusters and found the characteristic elbow at 34 clusters. Following this, we fit the profile bios into 34 multi-user clusters. The average number of users in each of these clusters were approximately 11. For a query user Q, we randomly pick 3 users from the same cluster that Q belongs to. When 3 users were not available in a given cluster, we dropped the query user and moved to the next. To further avoid sampling bias, we repeated this experiment 4 times to randomly sample neighbors in a cluster. We only selected those neighbor tags which had occurred prior to Q's current post. We noticed an overall lower (compared to no neighbor tags) tag-recommendation F1 score when neighbor tags provided by K-means were included in the BBERT T g input document. This has been detailed in Table 4 (row 3).

Bio-BERT Feature Clustering
A major drawback of the K-means approach is the number of fixed clusters. In an online health community setting, users may be close to multiple clusters. Additionally, the embeddings extracted from profile bios may not sufficiently represent similarity (or dissimilarity) in the feature space due to the simplicity of the encoder used. To overcome these issues, in our second approach, we replaced the TfidfVectorizer feature extractor with a Bio-BERT model in order to improve the semantic representation of profile-bios in the feature space. We reduced the BERT feature vectors to a two dimensional t-Stochastic Neighborhood Embedding (t-SNE) space and found the 3 nearest neighbors (h) using simple euclidean distances as a measure of similarity. The absolute location of points on a t-SNE plot is not an indication of quantitative similarity. However, we can use the relative euclidean distance between two or more points averaged over 4 independent t-SNE runs to provide a closer estimate of the nearest neighbors. In this approach, for a given query user, Q, the Figure 7: Temporal Named Entity Recognition task Labels based on illness progression number of neighbors is not limited (soft clustering) and any number of neighbors can be extracted based on distance from query user. The keywords of the neighbors occurring prior to current post were then used as neighbor tags in the input document. Similar to the K-means baseline approach, we evaluated this clustering method through changes in the tag-recommendation F1 score. This has been recorded in Table 4 (row 4).

Disease Timeline Matching
In this novel approach, we implemented the following steps in order to extract neighborhood clusters and share tags within them.
• The unstructured profile-bios were converted into a structured 13-column disease timeline using a Named Entity Recognition (NER) BERT model • A Bio-BERT model was used to extract features from the disease timelines and were clustered in the feature space using t-Stochastic Neighborhood Embedding • The h nearest neighbors are identified using euclidean distance in the two dimensional t-SNE feature space.
• Keywords of neighbors which have occurred prior to query post are used as neighbor tags (St).
• Neighbor tags are then appended to the input document before being provided to the BBERT T g model.
Conversion of Profile-Bios: First, we converted the unstructured profile bios to a structured temporal timeline. In order to provide structure to the profile bios, we contextually sorted the text into various classes. The classes created are shown in Figure 7. Where "Other" class corresponds to words that do not belong to any of the other 12 classes. "Current" refers to the time when the bio was written, "Past" and "Future" are with respect to the "current" time. As it can be observed, the data is forced to take a temporal shape in order to extract user-disease timeline. This annotation was done by two subject experts who manually read each profile bio and filled in the corresponding columns. The annotators read the entire profile-bio and then determined if parts of the text can be put into one of the 13 classes mentioned above. To verify the usability of the labeled dataset, we compared the similarity across the two annotators and obtained a cosine similarity of 0.811 and a Cohen's kappa of 0.784. According to [32] our score for inter-rater reliability and agreement was "substantial" and 0.16 short of "perfect agreement". We observed the lowest agreement in the Employment class. An intersection of the labels between the annotators is used as ground truth for the NER text classification task. In order to automate this conversion between unstructured profile bios to structured timeline data, we trained a NER model to perform a 13-class classification of profile-bio text. We perform classification only to obtain the temporal disease timeline and not the topic tags. The classes correspond to the individual columns in the temporal timeline of Figure 7. We trained two models namely (i) Bio-BERT [26] and (ii) LUKE-Deep Contextualized Entity Representations with Entity-aware Self-attention [50]. The Bio-BERT is only fine-tuned on our dataset while LUKE was trained entirely from scratch on our dataset. This was done to evaluate the value of information provided by the Bio-BERT pre-training. For both models, we used the B,I,O labelling procedure (Beginning of a label, Inside a label and Outside a label) to maintain standardized practices followed in the NLP community and to increase reproducibility. In the training set, we used 414 profile bios with 44,320 total words. In the non overlapping test set, there were 19,014 words from 160 bios. The models were trained for 25 epochs on a Nvidia DGX Tesla V-100 GPU for approximately 3.5 hours for fine-tuning Bio-BERT and 18 hours for training the LUKE. The Bio-BERT model outperformed the LUKE classifier in the following metrics-(a) The overall accuracy of the Bio-BERT model was 80.59% while LUKE achieved 73.63%. (b) The average recall percentage for Bio-BERT was 74.98% while only 66.29% for LUKE. The precision for each of the classes is low due to the large number of words in the "Other (OUT)" category. The imbalance in the dataset skews the predictions which lead to a high false positive rate (see Figure 8). The confusion matrix for Bio-BERT along with its class-wise precision and recall is shown as well. For all the downstream tasks that follow, we use the Bio-BERT model fine tuned on our dataset as the primary feature extractor. Feature Extraction and t-SNE: The next step in the clustering process is to perform unsupervised clustering based on similarities in the column values. The 13 columns can be treated as a sentence vector for feature extraction. We employed the previously used Bio-BERT fine tuned on profile-bios to extract the features of the structured text data and map it to a two dimensional tdistributed stochastic neighbor embedding (t-SNE). Using the plot, we infer soft-clusters [22] based on user grouping in the 2-D space. Features extracted from profile bios directly and features from the temporal timeline are shown in the t-SNE plots acquired after 750 iterations in Figure 9. As it can be seen, features extracted from temporal timelines following NER form natural clusters compared to features directly extracted from profile bios. This is primarily due to the noise removal inherently performed by the NER task. For illustration, we pick a random user (#248) and check the nearest 3 users' profile-bios. Inferred from Figure 9b, the clustering is around COPD and Emphysema.
For the purpose of clustering, we find the nearest h users to the primary user using Euclidean distances in the t-SNE feature space similar to the previous approach. We accumulate all keywords of the neighbourhood users, occurring prior to current post and append it to the input document of the primary user. The impact of neighbor tags from 3 neighbors clustered with with disease timeline similarity is shown in Table 4 (row 5).
Having established the ideal clustering model, we experiment with number of neighbors to determine influence of similar users in the community. Table 5 illustrates the effect of collaborative filtering, where, row 1 has the original full mode model and the subsequent rows present the improved model (BBERT T g ) with neighbor tags.
At the end of clustering, a particular user has the following items in their input document: (a) The profile-bio, (b) The previous posts and their corresponding keywords, (c) The current post and its keywords and (d) Neighbor tags from similar users. This input document is used to predict future tags G U T +1 .

Model Comparison
In the second set of experiments, we compare our best model (Full-Mode with shared tags, P n =3, h=3) with the current state of the art topic tag recommendation techniques using the same metrics. We also establish a baseline using Latent Dirichlet allocation (LDA) method of tag prediction with an implementation similar to [4] , [16]. We identified two state of the art models based on architectural and feature extraction differences. The first is HashRec [47] and the second is Attention-based Multimodal Neural Network Model for Hashtag Recommendation (AMNN) [51].
HashRec is a hashtag recommendation model for social media text, specifically, conversational text on Twitter and Weibo. The model uses two encoders to encode the user posts and the conversation individually, followed by a bi-attention module to capture their interactions. The extracted features are further merged and fed into the hashtag decoder consisting of sequential Gated Recurrent Units (GRUs). We provide the first encoder with Profile Bios and the second encoder with a combination of prior posts, their keywords and shared tags from the neighborhood. We train the model to produce future tags sequentially. The default (Twitter) hyper-parameters and settings were used. AMNN is an attention based model for predicting tags from multi-modal data. The model uses two individual encoders, one for text and one for images consisting of bidirectional Long-Short Term Memory(LSTM) modules followed by an attention unit. The features are concatenated and fed to a sequential GRU model to predict tags. We modified the network to have two text encoders by removing the image encoder and the remainder of the network is unchanged. Similar to the previous model, the profile bios are provided to one of the encoders and the remaining input document is given to the next text encoder. We used the default hyperparameters and settings to obtain results. The SOTA models are trained and tested on the same training set and test set used for our Bio-BERT (BBERT T g ) model. Table 6 reports the results of this experiment. The best performance is in bold while the second best is underlined.

Data Bias Experiments
Classification based models even with Bio-BERT embeddings do not perform as accurately as text generation models due to the imbalance in the prediction classes (tags). We show this by comparing a Bio-BERT classification (BBERT Cl ) model against our proposed tag generation model (BBERT T g ). This is included in Table 6. Very frequently occurring tags can skew results as they make it easier for models to guess the most frequent tag multiple times. We checked for the robustness of LDA and the BBERT T g models by deleting the top two tags (Chemotherapy and Cancer and Tumors) from the train and test sets. This experiment is to measure the impact of tag frequency on the models. Table 7 shows the F1 scores when (K=3) tags are predicted with the two modified models.

Interpretation of Results
Modeling information seeking behavior in a healthcare setting requires identifying key variables like disease similarity and history of interests. We designed our ablation studies to identify these factors and leverage the information to develop an accurate temporal tag prediction model. We observe that the amount of information presented to the BBERT T g model largely affects the model's ability to understand topics of interest. From Table 2, we can see that using only the profile bios of the users yielded the lowest F1 score. This is due to low correlation between fixed content in the bio and the dynamic changes in search interest. Thus utilizing multiple data sources like the posts and their corresponding keywords help in improving predictions. As anticipated, the number of previous posts in the input document causes variation in the tag prediction metrics. We notice that the F1 score is the highest when 2 previous posts along with the current post are used for future topic tag prediction. Improvements to the model come through collaborative filtering, an additional source of information regarding similar users in the database. In Table 4, experimenting with random neighbors yielded lower F1 scores at all levels indicating that including Chemotherapy / Infection / Cancer / Tumors / Cough BBERT T g (Ours) Chemotherapy / Radiotherapy / Afatinib / Cancer and Tumors / Drug Trial search tags from random users only caused the model to pay attention to noisy inputs. While on the other hand, Bio-BERT features extracted from cleaned NER disease timelines yielded the cleanest set of neighbors who positively contributed to the tag prediction metrics. The search tags from neighbouring users helped the Bio-BERT model predict better as it is likely that they have queried about this topic in the past. An interesting observation from Table 5 is that the number of nearest users did not affect the cosine similarity significantly. We believe the reason for this is that since cosine similarity is calculated in the feature space, the user clusters are also defined in a similar feature space. Due to this, adding more users does not negatively affect cosine similarity. It did however point out that using more than 3 (h >3) neighbors could induce noise in the model due to uniqueness in the timelines.
In Table 6, we illustrate comparison of our proposed BBERT T g model against the state of the art networks. Tag prediction models tend to perform well when the tags to be predicted depend only on the input document provided. However, understanding context and predicting the a future tag is a non-trivial task that leads to low F1 score with the HashRec and AMNN models. To overcome this, we pose the problem as a sentence generation task instead of the traditional classification task and obtain benchmark performance . We even use the Bio-BERT embeddings in a classification task (BBERT Cl ) to show that sentence generation performs better due to robustness to skewed tag distribution which is usually the case with social media based text data.
Finally, we emphasize that our BBERT T g model is robust to tag occurrence frequency and show that even when the 2 most frequent tags are removed, the performance does not change by much when compared to a traditional LDA model as shown in Table 7.
To show that our model is truly understanding the context, we show an example of generated tags for a future post by providing the previous posts, tags, profile-bio etc to all the models in comparison. The f uture post not provided to the model was: "My partner was diagnosed with stage 4 lung cancer (nsclc) in June 2009. He has had Chemotherapy, Radiotherapy and Tarceva. He started taking the new unlicensed drug Afatinib yesterday. So far no side effects! He has been told that the side effects can be more severe than Tarceva. I really hope this drug works for him as we are fast running out of options." Our BBERT T g model was the only one to predict the tag Afatinib and also understand that the patient is undergoing a drug trial. This is shown in Table  8.

Summary and Discussion
In summary, our topic recommendation system, a hybrid system combining collaborative-filtering and content based ideas is designed for online health community users and utilises patient timeline similarity to group similar users. We use a pretrained Bio-BERT to perform the tasks of classification and sequential sentence generation. Our predictions of future topics yield accurate and contextual tags. We compare our model against models proposed for similar tasks like HashRec and AMNN and observe a superior performance.
We have empirically presented the quantitative advantage of our methodology over existing models. This improvement in precision and recall metrics is attributed to two specific contributions-(1) An auxiliary sentence generation task for tag generation instead of tag classification and (2) Usage of "disease timelines" to match similar users and perform collaborative filtering. Additionally, we also performed ablation studies to understand the importance of several factors like history and influence of similar users in the network. As a part of future work, we will focus on two specific aspects. First, we intend to improve the matching of similar users by leveraging interactions and graphs. Second, we plan to utilize the predicted tags to retrieve tailored medical articles from trusted sources like WebMD and Mayo clinic.