{'fraction_non_alphanumeric': 0.04047223258811955, 'fraction_numerical': 0.009939953183838246, 'mean_word_length': 4.907414829659318, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
A Linguistic Annotation Framework to Study Interactions in Multilingual Healthcare Conversational Forums

In recent years, remote digital healthcare using online chats has gained momentum, especially in the Global South. Though prior work has studied interaction patterns in online (health) forums, such as TalkLife, Reddit and Facebook, there has been limited work in understanding interactions in small, close-knit community of instant messengers. In this paper, we propose a linguistic annotation framework to facilitate analysis of health-focused WhatsApp groups. The primary aim of the framework is to understand interpersonal relationships among peer supporters in order to help develop NLP solutions for remote patient care and reduce burden of overworked healthcare providers. Our framework consists of fine-grained peer support categorization and message-level sentiment tagging. Additionally, due to the prevalence of code-mixing in such groups, we incorporate word-level language annotations. We use the proposed framework to study two WhatsApp groups in Kenya for youth living with HIV, facilitated by a healthcare provider.


Introduction
Good communication between patients and healthcare providers as a part of patient-centered care, where the patient is an equal partner in their healthcare decisions, has been shown to improve medical adherence (Schoenthaler et al., 2009;Ciechanowski et al., 2001) (i.e., the practice of consistently taking the medicines as prescribed) . However in the Global South, i.e. the countries, located primarily in the southern hemisphere, that have historically been identified as third world due to perceptions of their socio-economic status, technological advancements, and global dominance, high patient volumes and a need for cost-effective solutions can make patient-centered care difficult. In recent years, various online chat applications such as WhatsApp and WeChat are being increasingly used to ensure smooth access to both patient-provider communication and peer support beyond formal healthcare system (Karusala et al., 2021;Bhat et al., 2021;Karusala et al., 2020). Even though these peer support groups are effective in providing patient-centered care, they can be taxing for already overworked healthcare providers (Viswanathan et al., 2020). There is therefore a necessity for designing technology to support both the providers and patients using such groups.
Our proposed hierarchical framework consists of fine-grained peer support categorization and message-level sentiment tagging. Additionally, we address the prevalence of code-mixing in such WhatsApp groups by incorporating word-level language annotation. We evaluate our framework on chat logs of two WhatsApp groups of Kenyan youth living with HIV, moderated by a healthcare provider. We also develop a user-friendly annotation interface to facilitate annotation. In the rest of the paper, we describe the annotation process using the framework, the challenges faced and our learnings. Our annotation study shows that the annotators achieve high agreement for all categories, thereby indicating the efficacy of our linguistic annotation framework. We believe that this framework can be used to analyze inter-personal relationships among group members and is the first step towards developing NLP solutions to support healthcare workers providing remote patient care.
While developing the framework, we emphasized on code-mixing. Multilingualism in Kenya contributes to a considerable linguistic diversity in a population that speaks Kiswahili, Kikiyu and English fluently (Dwivedi, 2015;Chumbow, 2010). This leads to a high prevalence of codemixing in their linguistic interactions. This is further enriched by the use of Sheng, a Swahili and English-based cant (a mixed language or creole), by the urban youth in Kenya. It emerged as a result of urbanization and globalization (Githiora, 2002;ABD). Any annotation framework for such conversations, therefore, has to take into account this multilinguality.
We developed our annotation framework to support an ethnomethodologically-informed ethnographic analysis of the chat: a qualitative approach used in Human-Computer Interaction (HCI) research to understand social interactions (Button and Sharrock, 1997;Hughes et al., 1994). Such analysis has proven useful in the design of computer systems as it reveals the practices and methods of those who will use these systems and which need to be designed for (Crabtree, 2003); be-ing used to examine text interactions such as instant messages (O'Neill and Martin, 2003), internet forums (Martin et al., 2014) and chat messages Jain et al., 2018a). The annotation framework was designed to be used to support and inform the ethnographic analysis, with the aim of identifying ways in which we might support both the healthcare provider and participants in these chat groups.
Additionally, the framework would be useful for sociolinguistic studies of multilingual conversations and the schema could be extended to understand other peer support settings such as mental health forums. Another area where the framework might prove useful is for research around improving human-machine conversational agents (Ashktorab et al., 2019;Jain et al., 2018b) by leveraging data annotated using these categories.

Dataset
We studied WhatsApp chat logs from two peersupport groups for Kenyan youth, living with HIV. There were a total of 1,655 messages in Group-1 (28 members, 14 female, 14 male, age=14-17 years) and 4,901 messages in Group-2 (27 members, 21 female, 6 male, age=18-24 years). Both the groups were facilitated by a healthcare provider with a background in public health, HIV testing, and counselling. The facilitator sent weekly messages on a range of topics, such as future goals, strategies for medical adherence, etc., responded to queries (within 12 hours), clarified any misinformation posted on the group, and referred medical-questions to a clinic. The chat records were in Kiswahili, English and Sheng, sometimes a mix of these languages in a single message. All the messages were anonymized and translated into English by a native speaker. Each chat message in the dataset had the following information: anonymized speaker ID, timestamp, original message and English-translated message. The data contains sensitive content dealing with people's deeply personal lives and tragedies, and therefore, even with anonymization there are ethical reasons for not making the dataset public. However, anyone wanting access to the data for research purposes can get in touch with the authors.

Annotation Framework
The primary objective of our proposed linguistic annotation framework is to facilitate studies on inter-personal relationships among the peersupporters. In order to satisfy the above objective, the framework is governed by the following guiding principles. 1) We want to model the conversational intent of the peer supporters behind the act of sending messages. 2) We need to assess the psychology of their behavior by understanding their sentiments and emotions. This is even more critical in a healthcare setup where it is important to maintain an overall positive outlook. 3) We also want to capture the morphosyntactic level of linguistic information which can be used to help us tease apart deeper interaction patterns and stylistic features of conversations. The framework (Figure 1) has a hierarchical schema with five top-level categories that deal with linguistic and pragmatic phenomena observed in the chat. The categories include Peer Support (principle 1, 3.1), Sentiment (principle 2, 3.2), Word-Level Language, Topic Change, and Typo (principle 3, 3.3, 3.4, 3.5). Note that this framework can be further augmented to include more linguistic features and can be extended to perform similar studies on other peer support groups. In this section, we define the broader and fine-grained categories.

Peer Support
Peer support refers to members using their own experiences to help others feel accepted and under-stood. Our framework defines the following subcategories of peer support as:

Informational
The group members either seek or provide factual information, advice, or warning. These are further classified as: • Seek Information: A group member is seeking information from fellow group members. E.g., "explain further about circumcision". • Provide Information: A group member is providing information. E.g., "Yeah, it might be a sign. Also it increases when you are stressed.". The informational sub-category is further divided into the following sub-types: Medical: Seeking or providing factual information, advice, or warning about medicine, treatment, or disease. E.g., [Original] "Most people infected with the bacteria that cause tuberculosis don't have symptoms. When symptoms do occur, they usually include a cough (sometimes blood-tinged), weight loss, night sweats and fever".
Lifestyle: Healthy lifestyle related information such as exercise and food. E.g., [Original] "You can take a lot of soup, njahe, omena. These works well for new moms. Don't drink coffe or majani. Tumia to cocoa".
[Translated] "You can take a lot of soup, turtle beans, fish. These work well for new moms. Don't drink coffee or tea. Use only cocoa".
Personal: Information regarding relationships with partner or other family members. E.g., [Original] "The problem is I have a partner na ajue my status am afraid day nitamwambia he can even kill himself " [Translated] "The problem is that I have a partner who doesn't know my status and am afraid the day I decide to say he might kill himself." Admin: Administrative information such as rules and regulations of the group and formal meetings. E.g., [Original] "We shall respect all members and not use words like HIV, ART, CD4 So that people feel comfortable".
Other: This contains any information not included in the above categories. E.g., [Original] "Hi....This is a big warning...my mother in law has just lost over Kes. 70,000 to M-Shwari and KCB-MPESA con-women...".
[Translated] "Please don't be am also loosing appetite but it come back just go see a doctor, sometimes it's usually the type of drug and your timing for your medication, sorry my dear".
The emotional sub-category of messages are further sub-categorized into the following: Empathy: An explicit mention of the difficulty experienced by the seeker that validates his/her distress or an attempt to see things from the seekers' perspective. E.g., [Original] "pia Mimi na shida ingine Niko na mimba yake".
[Translated] "Same here also and another problem is that he got me pregnant".
Hopefulness: A response encouraging others to stay hopeful and optimistic in difficult situations. E.g., [Original] "Sorry for this dear. Like 5002 has said some things need sacrifice and time. Never give up. Just try and overcome the hurdle, all will be fine." Happiness: A response that indicates a happy mood or excitement. E.g., [Original] "Early this month, we were blessed with a baby girl and we thank God for His mercies on us".
Negativity: A response that indicates a mournful mood or shows any negative emotion (such as anger/sorrow/frustration). E.g., [Original] "Hello guys nfeel so bad today c ski poa nko n headache n pia cna strength pray 4 me".
[Translated] "Hello guys am feeling so bad today i have a headache and I also dont have energy please pray for me".

Chit-Chat
The messages in this category include introductions, greetings and other responses that promote social interactions or demonstrate friendliness. E.g., [Original] "Lunch ilipita sasa ni supper" [Translated] "Lunch is over it's now supper time".

Group Work
A response wherein members encourage each other to work as a group. E.g., [Original] "Let's bring topics even on the contemporary experiences so that we can continue keeping the group more active as it should be guys!"

Other
Any response that does not belong to any of the above categories (like emoji, a system-generated message, etc). E.g., [Original] "Waiting for this message". (a default system-generated message in WhatsApp)

Sentiment
Sentiment is a view or opinion that is held or expressed by an individual. The sentiments are further categorized into the following: Negative: It expresses a negative view or opinion. E.g., [Original] "Uko na shida ya kichwa c siri".
[Translated] "You have a mental problem it's no secret".
Positive: A message is classified as positive if the speaker expresses a strongly positive opinion on something or someone. Note: Greetings are not included in this category. E.g., [Original] "I'm much happy to interact and share with you guys!" Neutral: When neither positive nor negative sentiment is expressed, the message is labelled as neutral. This could include a general comment, acknowledgement, chitchat or a simple greeting. E.g., [Original] "I suggest that we should try and at least do this for our health."

Word-Level Language
To understand the code-mixed interactions in the chat, word-level language annotation is needed. For our dataset, we used five language identification codes: En for English, Sw for Kiswahili, Sh for Sheng, CM for Code-Mixed words or phrases (E.g., 'nfeel', 'tutawash'), and Oth for miscellaneous (e.g., emojis, any other language). E.g., [Original] "Mimi Nilichukua my siz tukalost".
[Language] (Sw) (Sw) (En) (En) (CM) [Translated] "I took my sister and I got lost". If a particular word is tagged as Oth, the annotators were asked to explain the reason behind choosing this particular tag. For instance, Kyole 2 (name of a hospital) is assigned the language tag Oth, and the reason specified is Named Entity. Also, if the language is different from En, Sh and Sw, that needs to be specified in the reason correctly. Thus, our schema can be further extensible to annotate more languages.

Topic Change
Each message is labeled with a binary flag (Yes/No) to indicate a change in topic.

Typo
Shortened words, abbreviations and misspellings are pervasive in informal conversations, especially in instant messaging such as WhatsApp. It is important to identify and normalize such typos. The annotators were asked to correctly normalize the typos found during the annotation process. It is worthy to note that the typos and abbreviations need to be separately tagged and identified. E.g., 1.

Annotation Interface Overview
We developed an annotation interface (Figure 2) to facilitate user-friendly annotation. The annotators were provided with detailed guidelines on how to use the interface for annotating the chat messages based on our proposed annotation framework. Django framework (Python) was used on the server-side for implementing this interface. This interface can be also used to annotate the peer support conversations in the mental health chat forums such as Reddit, TalkLife. The annotators need to login into the interface with their credentials to start the annotation process as outlined in the guidelines provided to them. On logging in, a target message is displayed along with ten (or less) previous messages to provide context (Figure 2a). The right side of the screen shows question related to top-level categories: peer support, sentiment, and change in topic (Figure 2b). The annotators then have to choose appropriate labels. For the peer support category, the annotators have to further choose subcategory labels as well (Figure 2c).
As a message can be associated with multiple subcategories, the annotators are asked to highlight specific portions of the message for each subcategory. E.g., in the message "Hi...he got me pregnant..."-"Hi" is tagged as Chit-Chat → Greeting and "he got me pregnant" is tagged as Informational → Personal.
For Typo, the annotators are asked to highlight spelling mistakes and abbreviations, and correct them wherever possible. For instance, in the message "Morning members from today, hence forth untill further notice am the S.T.O (sitting allowance officer) ask any querry you get the right answer" the annotators needs to highlight the typos ("untill", "querry") and normalize those ("until", "query") using the interface. Moreover, they should also annotate the abbreviation ("S.T.O").
The annotation requires a language tag for each word in the message. Instead of specifying a language label for each word, the annotators are asked to highlight the word spans for each language category, facilitating quicker and less errorprone language identification. For example, in the message "Wow, mamboz mrembo" ([Translated]: "Wow, Hello beautiful")-"Wow" is highlighted as English and the word span "mamboz mrembo" is highlighted as Kiswahili.
The annotators are expected to completely annotate each message before moving on to the next message. The submitted annotations are saved in a cloud database. The annotators have an option to revise and modify their current or previously submitted annotations during the process. If the annotators are unsure of how to annotate a particular message, the interface allows them an option to note and skip that message. This annotation interface can be leveraged easily to annotate any complex hierarchical annotation schema comprising of multiple levels.

Annotation Experiments and Observations
The annotation was conducted in three phases. In Phase-1, a small sample of 395 English-translated messages (105 messages from Group-1, 290 messages from Group-2) were annotated, and the corresponding original multilingual messages of the same set were annotated in Phase-2. In Phase-3, the entire multilingual dataset was annotated. Below we discuss the challenges faced and the learnings from all the annotation phases.

Annotation Phase-1
The Phase-1 study was designed to ensure that the entire annotation process as well the guidelines were well-understood by the annotators. In this phase, the short-listed 395 chat messages were annotated by one of the co-authors, an expert English speaker, who led the development of the proposed annotation framework. The co-author, not being able to understand Swahili or Sheng, found it easier to initially develop the schema using the English-translated messages (this translation was manually carried out by a native speaker). We consider that as the gold standard annotation for Phase-1, and use it to assess the quality of the rest of the pilot annotations in Phase 1. Though the English-translations were not of great quality, but that did not affect our interpretations since we have carried out the final annotation on the multilingual messages (Phase-2 and Phase-3). Three annotators (graduate students proficient in English) were asked to annotate the same 395 messages using the annotation guidelines. For quality estimation, Cohen's Kappa (κ) (Vieira et al., 2010) was used to assess inter-annotator agreement (IAA) between each annotator and the expert annotator. We observed an average of 0.67 κ agreement for the fine-grained peer support subcategory tags and 0.89 κ for sentiment. While there was strong agreement for sentiment labelling, the peer support subcategories showed only moderate agreement.
We analyzed the data to understand the annotation differences. First, we found the highest annotation disagreement among Informational messages-46% of the total disagreements were between Medical and Lifestyle. Hence, we revised the annotation guidelines such that Medical was given priority over Lifestyle, i.e., the annotators were asked to select only Medical, or both Medical and Lifestyle, when unsure. For example, in this message, "Hi? It's GXXX here I think you can try tell a person who has not accepted the status the importance of taking the drugs and how they work in our bodies. Yah remember to eat healthy", the individual is stressing upon both the importance of medical adherence and consumption of healthy food. Hence, it should be tagged as both Medical and Lifestyle under Informational.
Second, the ambiguity between Lifestyle and Personal subcategories comprised 12% of the disagreements. For example: "We should always protect ourselves using a condom during sexual intercourse". Our guidelines were revised to clarify that the Personal tag should be applied only to messages dealing with personal relationships, while the Lifestyle tag should only be used to annotate lifestyle habits. Thus, in the above example, only Lifestyle tag is applicable.
Third, another source of confusion was between Group Work and Admin with 24% of total disagreements. To solve this confusion, we provided more examples in the annotation guidelines to clarify the differences between them. These examples were laid out to elucidate that encouraging more activity within the group members belong to Group Work whereas Admin dealt with the meetings arranged by the moderators or formal group regulations. For example, in "With this group, we would like to support each other in staying healthy and supported. If you are having problems or questions about your health, please share them with the group, or inbox me individually. We are stronger together.", the moderator is talking about the rules of communication within the group, thus it should be tagged as Admin. Whereas in "How are we all doing this week? What are you most looking forward to about this group?", the speaker is encouraging more participation in the group and should be annotated as Group Work.
Finally, other minor disagreements originated due to human error (provide versus seek information, Greetings versus Acknowledgement). This did not require any revision of the guideline.

Annotation Phase-2
The annotation guidelines were revised based on the learnings from Phase-1 annotations. We have calculated the pairwise Cohens kappa score using the Scikit-learn library of Python. In Phase-1, we wanted to understand how much each of the pilot annotators agree with the expert annotator using Kappa Agreement. Certain portions of annotation guidelines as well as schema were updated and revised based on their understanding.
Two professional annotators, based in Kenya and proficient in English, Kiswahili, Kikuyu and Sheng languages, were then asked to carry another pilot annotation in Phase-2 based on the revised guidelines. In Phase-2 IAA results (as illustrated in Figure-3), we calculate the pairwise agreement between two annotators of Africa for each of the peer support categories on the multilingual chat messages. No gold standard annotation was used to assess the quality of their annotation in Phase-2 unlike Phase-1. The key difference between Phase-1 and Phase-2 was the addition of language tagging as well as the annotation being undertaken on the original messages rather than the translations.
While analyzing IAA, we observed that tags of English-translations differed from those of the original chat logs, which was due to inaccurate language translation. For instance, "Poa Sana" ([Translated] "Very good") was tagged as Acknowledgement and Socialization by the annotators, whereas the corresponding Englishtranslation "Hey" was tagged as Greetings by the expert annotator. Thus, with the help of the Kenyan annotators, a more accurate English translation was also obtained, and accordingly the gold standard annotation was corrected.
The annotators also found the language tags of certain named entities such as food items (e.g., 'soup', 'cocoa') confusing, mainly because some of the named entities can exist as borrowings in Kiswahili and Sheng. Hence, we modified the instructions to categorize such entities as English.

Annotation Phase-3
In this phase, the same annotators from Phase-2 annotated the entire dataset based on the revised guidelines. An iterative revision of the guidelines was conducted based on any observed confusion in this phase as well, and the annotators were expected to revise their annotations accordingly. Here we describe a few of them.
Second, annotators also faced difficulties in tagging language category for typos (e.g., 'realtionships', 'apa', 'nni'). This was resolved by asking them to choose the language category of the normalized word (e.g., 'realtionships' → 'relationships' in English, 'apa' → 'hapa' in Kiswahili, 'nni' → 'nini' in Kiswahili). It is worthy to note that in order to tag the abbreviations and contracted forms of words, the annotators were also asked to tag it to the language category of the expanded form. We did not come across any ambiguities or unclarities regarding this during the annotation process.
Third, emojis posed another challenge for annotating the peer support categories of the chat messages. In the annotation guidelines, the annotators were instructed to tag non-textual content as Other. However, for certain messages, decoding the emoji was needed to obtain the correct tag. For example, "Is bad?". The semantics of the message is dependent on decoding the meaning of the emoji ("Is drinking alcohol bad?"). Ideally, this message should be annotated as Lifestyle and not Other in the Informational category. Since the point of human annotation is to indeed identify deeper meaning which a naive algorithm might miss, the annotators were asked to tag the messages after decoding the meaning of the emoji.

Observations
The distribution of peer support category labels for Phase-3 in both the groups are summarized in Ta-

Peer Support
Group-1 Group-2  Table 1: Overall distribution of peer support categories in both the peer support groups after Phase-3. Figure 3: IAA for peer support categories in Phase-2 ble 1. We observed that a majority of the messages in both the groups belong to Chit-Chat (65.9% in Group-1 and 66.2% in Group-2), followed by Informational. Within Informational category, Medical stands out as the most prominent for Group-2 (35.0% informational messages) while Admin is the most prominent in Group-1 (34.6% informational messages). It should be noted that the IAA is relatively lower for Hopefulness, Admin, and Group Work indicating disagreement between the annotators. Overall, Chit-Chat and Acknowledgement are easier to annotate as is reflected in their IAA score. Further, we also observe that most messages are short in length with an average message length of 4.34 words in Group-1 and 5.6 words in Group-2.  Table 2: Overall distribution of the messages containing different language category labels in both the groups after Phase-3. The nature of messages (monolingual or multilingual) is determined by the language tags for each of the words in the messages.
With respect to Sentiment labels, most of the messages were tagged as Neutral (93.4% in Group-1 and 87.1% in Group-2), followed by Negative (5.0% in Group-1 and 4.3% in Group-2) and then Positive. It was also found that the annotators agreed mostly on Negative sentiment (Phase-2 κ=0.92) and disagreed the most on Positive (Phase-2 κ=0.67). Table 2 describes the results for language tagging after Phase-3. It shows that the IAA score for English (En) (κ=0.98) and Kiswahili (Sw) (κ=0.98) words is higher compared to those of Sheng (Sh) (κ=0.87) and Code-Mixed phrases (CM) (κ=0.85). It can also be observed that the messages contain a high amount of code-mixing (23.8% messages are code-mixed in Group-1 and 24.4% in Group-2). While English is the dominant language for the monolingual messages, codemixed English Kiswahili (En Sw) is the dominant language-pair for the multilingual messages.
We found that a change of topic was indicated in 3% and 4% of the chat messages in Group-1 and Group-2, respectively. In Group-1, a topic change was initiated mostly by the admin (89% of topic changes), while in Group-2, other group members have also been found to initiate discussions on new topics (only 39% of total topic changes were triggered by the admin). Further, we observe that a topic change is mostly (95.2%) associated with the Informational or Group Work subcategories.
Due to the informal nature of WhatsApp con-versations, a significant percentage of chat logs contain spelling mistakes (19.4% and 18.7% messages in Group-1 and Group-2, respectively) or abbreviations (15.7% and 10.9% messages in Group-1 and Group-2, respectively).

Conclusion and Future Work
In this paper, we presented a comprehensive annotation schema, combining several linguistic and pragmatic phenomena, to enable studies of What-sApp based conversations of youth living with HIV. Unlike Twitter and Facebook, this requires understanding the content of previous discourse to annotate the sentiment and conversational intent of supporting the peers in specific examples. During the process of annotation, we have learnt that a good percentage of examples require understanding the context. To the best of our knowledge, we are the first to combine several standard annotation schemes to annotate the multilingual conversations in a small, close-knit community. We believe our schema can be used to flag important messages (based on user concerns) to the healthcare providers. This will in turn help to alleviate the need to scroll through all the messages individually to identify the ones that require a response from the facilitators. Linguists can also use this framework to study the nature of multilingualism in such conversations from both sociolinguistic and structural perspectives. In future, we plan to use the framework to further study the nature of engagement and multilingual interactions in such peer-support groups in healthcare domain with the aim of enhancing remote patient care.