{'fraction_non_alphanumeric': 0.028623853211009173, 'fraction_numerical': 0.02328236493374108, 'mean_word_length': 5.073798910351659, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 1, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Automatic Identification of 5C Vaccine Behaviour on Social Media

Monitoring vaccine behaviour through social media can guide health policy. We present a new dataset of 9471 tweets posted in Australia from 2020 to 2022, annotated with sentiment toward vaccines and also 5C, the five types of behaviour toward vaccines, a scheme commonly used in health psychology literature. We benchmark our dataset using BERT and Gradient Boosting Machine and show that jointly training both sentiment and 5C tasks (F1=48) outperforms individual training (F1=39) in this highly imbalanced data. Our sentiment analysis indicates close correlation between the sentiments and prominent events during the pandemic. We hope that our dataset and benchmark models will inform further work in online monitoring of vaccine behaviour. The dataset and benchmark methods are accessible online.


Introduction
The development of effective and safe vaccines has been shown as one of the most successful means to mitigate the spread of COVID-19 disease around the globe. As with any other vaccination program, COVID-19 intervention in any country depends on public acceptance and vaccine uptake. This has to be done by a large proportion of society to attain herd immunity, which is estimated to range from 67% to 95% (Mills et al., 2020;Randolph and Barreiro, 2020;Anderson et al., 2020). However, vaccine hesitancy has been identified as one of the top 10 challenges to global health by the World Health Organization in 2019 (World Health Organization, 2019 January [cited 14 ).
Vaccine hesitancy refers to the delay in acceptance or refusal of vaccination by the public despite the availability of vaccination services (MacDonald et al., 2015). This behaviour has been significantly prominent towards COVID-19 vaccines as they differ from previous vaccines in many respects: accelerated development, novel techniques used, potential side effects, uncertainty regarding the size and extent of their effectiveness, and limited production compared to the demand (Dubé and MacDonald, 2020).
Australia began its mass vaccination program in late February 2021 but was initially hampered by low vaccine uptake due to mistrust of vaccine effectiveness and the government, the blood clotting syndrome (TTS) associated with the AstraZeneca product, shipment delays, and misinformation (Kaufman et al., 2022b). Despite the fact that the present uptake is high (> 90% two doses for over 16 years), the coverage varies across age, jurisdictions, and the number of doses (Australian Government: Department of Health and Aged Care, 2022 August [cited 14 ). High coverage is primarily driven by travel desires (and travel vaccination requirements) and the need to mitigate risk. As new variants of SARS-CoV-2 are emerging and existing immunity waning in a short period of time, it is likely that people will need to get multiple booster doses. Consequently, identifying immediate and future public behaviour toward vaccination is important for public health authorities to combat possible challenges to reaching and maintaining herd immunity.
The 5C model provides five measures (Confidence, Complacency, Constraints, Calculation and Collective Responsibility) for assessing an individual's psychological reasoning towards vaccination (Betsch et al., 2018(Betsch et al., , 2020. Confidence associates the trust in vaccine effectiveness, safety, and the system that delivers it. People with low confidence mistrust the healthcare system, fall for misinformation, believe in conspiracies, and doubt the benefit of vaccines. Complacency exists when vaccination is viewed as a low priority or when vaccine-preventable diseases are not of a concern.
High complacency correlates with low uptake of a vaccine. Constraints refer to the structural and psychological barriers to getting vaccinated. For example, geographical constraints in accessibility, limited language, and health literacy, and cost may postulate high constraints. Calculation defines the engagement in extensive information searching, which may lead to lower vaccination willingness arising with exposure to a high volume of antivaccination content. Collective responsibility asserts an individual's willingness to protect others by getting vaccinated and contributes to herd immunity. See Table 1 for a summary of the description of the 5C model.
Most works in identifying vaccine behaviour use surveys that are costly, time-consuming, and don't scale to a large population. The use of social media for expressing vaccine-related opinion provides a great opportunity to use online social monitoring tools to guide health policy-making for vaccine adoption. There are few recent studies that adapt the 5C scheme for online health monitoring. Greyling and Rossouw (2022) focus on the analysis of tweets rather than building predictive models, Fues Wahl et al. (2022)'s uses 1794 tweets from Scandinavian users and manually categorise them into the 5C categories, and Boucher et al. (2021) focuses on the vaccine trials and uses unsupervised methods.
Our contributions are as follows: 1) we provide the first large-scale vaccine behaviour dataset in English, annotated with both 5c and sentiment; 2) we present two benchmark models and show the challenges of 5C predictive models given the highly imbalanced data; and 3) we analyse the data showing that changes in 5C distribution is not uniform across regions, indicating opportunities for targeted health messaging. We make the dataset and benchmarks available hoping to impact future work in online vaccine behaviour monitoring based on the 5C framework.

Related work
The 5C model has been widely applied to examine COVID-19 vaccination behaviour. In 2020, Kwok et al. (2021a) estimated Hong Kong nurses' intention to receive COVID-19 vaccine using the 5C model and examines the correlation of their vaccine behaviour to previous influenza vaccination. Thunström et al. (2021) applied the 5C scale to investigate psychological reasoning behind the previous uptake of measles and flu vaccines by adults in the United States and their intention to get COVID-19 vaccination for themselves and their children. Wismans et al. (2021) studied the psychological drivers of vaccination intention in university students across the Netherlands, Belgium and Portugal, using the 5C model. Gallant et al. (2021) implemented the 5C model to investigate older adults' vaccination behaviour in the United Kingdom over the first year of the pandemic. Lindholt et al. (2021) examined the levels and predictors of acceptance of an approved COVID-19 vaccine in eight Western countries by utilizing the 5C model. Rustagi et al. (2022) applied the 5C model to identify vaccine hesitancy among chronic disease patients availing care in a primary health facility in India.
Previous studies examined COVID-19 vaccination behavior utilizing traditional surveys (Seale et al., 2021;Trent et al., 2022;Rhodes et al., 2021;Edwards et al., 2021;Dodd et al., 2021;Kaufman et al., 2022a;Kwok et al., 2021a;Thunström et al., 2021;Sherman et al., 2021;Wismans et al., 2021;Paul et al., 2021;Sallam, 2021;Akarsu et al., 2021;Fisher et al., 2020;Freeman et al., 2020;Ward et al., 2020;Lazarus et al., 2021). However, such surveys are often costly in their design and implementation, time-consuming, produces limited data and represent comparatively short-term situation. Recently, Twitter has been increasingly applied in research concerning public attitude towards vaccination due to its advantages of availability of a large amount of real-time posts without any costs, ease of access and public searching facility. In spite of these advantages, there remains a gap in Australia for Twitter-based COVID-19 vaccination research. We found just one study (Kwok et al., 2021b), conducted during 2020, which addresses Australian public opinion towards COVID-19 vaccine solely from Australian Twitter users. Sentiment analysis of this study has shown that the majority of people expressed positive emotions towards vaccine with trust and anticipation as the most prominent behaviors associated with it, while fear being the top negative emotion.
An investigation of Twitter posts from 10 countries, including Australia, has revealed that more information about vaccines' safety and the expected side effects may increase public positive attitudes towards vaccination Greyling and Rossouw (2022). Similarly, sentiment analysis from 4 million tweets Collective responsibility Willingness to protect others  (Chopra et al., 2021;Lyu et al., 2021;Xue et al., 2020). Along with sentiment analysis, some studies have implemented the 5C model to examine COVID-19 vaccination emotions as well as attitudes towards other vaccines in tweets. Greyling and Rossouw (2022) constructed a multiple linear regression model to examine positive attitudes towards vaccines across ten countries. In their model, among other methods, the positive 5C categories were applied to identify the covariates. Boucher et al. (2021) applied a topic modeling approach to investigate the mistrust in Covid-19 vaccination based on English and French tweets. They then used the 5C scale to categorize the topics and found that all mistrusts fell into the Confidence category. Fues Wahl et al. (2022) applied the 5C model to map relevant predictors for several vaccination behaviours in Scandinavian Twitter users. Similar to our work, they manually labelled each tweet according to the 5C scale. However, unlike our dataset, their dataset was multi-labeled, as each tweet was assumed to have multiple 5C categories. They did not, however, include Covid-19 vaccination as a specific vaccination category in their analysis.

Dataset
We made use of the Twitter API to collect 60, 000 COVID-19 vaccine-related tweets restricted to Australia from 01 February 2020 to 30 October 2021, using keywords: vax, vaccine, vaccinate, vaccination, jab, pfizer, astrazeneca. We selected a weighted sample of 20, 000 tweets with weights proportional to the total number of tweets within the time frame of a week and annotated each tweet first based on its sentiment towards vaccination and then its associated 5C category if the tweet contains either a positive or negative stance. We note that some tweets do not provide a clear positive or negative stance and thus have been categorized accordingly if the stance is totally irrelevant to the topic of vaccination, or if the stance concerns vaccination, but does not belong to either the positive or negative category. We excluded tweets from the last two categories in our analysis. We labeled each tweet based on the most prominent 5C behaviour. Thus, our dataset has the structure of binary categories in terms of tweet's sentiment and multi-class categories with regard to 5C vaccine behaviour. Table 2 depicts a sample of labeled tweets. The final dataset consists of 9471 annotated tweets. The study has received clearance by the authors' organisation's human research ethics committee.
We note that, in the original 5C model (Betsch et al., 2018) much of the description was made for the negative aspects of the model, hence we made some discretion for labeling positive 5C categories, as, based on the tweet texts, these categories were not clearly identified. For example, tweets which mentioned only about taking vaccine without any further explanations (like 'got my first vaccine !!!') were labeled as positive Confidence. Tweets which talked about getting vaccine as a compulsory action (like 'no jab no job') were labeled as positive Constraints. Similarly, tweets which supported vac-

Label
Sample Tweet (rephrased for privacy reasons) + Complacency It takes just one COVID infected person to start a Pandemic. Please vaccinate. + Complacency Resp. Difficult to wear mask but we have to. Vaccinate, not just for yourself but for others too. + Calculation Vaccines don't prevent infections, they prevent severe disease and hospitalisation. + Constraints no jab no job. That's how it should be. -Complacency 99% of people recover from COVID, vaccines shouldn't be mandated.

-Confidence
A vaccine that might kill you for various reasons while big pharma benefits. -Calculation myocarditis risks is higher in teenage boys, we shouldn't rush into vaccinating them.
-Constraint AZ age limit should be 60, not 50 Table 2: A sample of eight tweets (rephrased for privacy reasons) and their labels. + andrefer to the positive and negative sentiments towards COVID-19 vaccination, respectively. The five C categories are also shown (see Table 1 for definition).
cine based on data as well as personal experience (like 'I took AZ vaccine 1 week ago, and there is no symptoms of blood clot') were categorized as positive Calculations. To check the validity of the annotations, we compared annotation agreements between two researchers from the team who independently labeled a random sample of 200 tweets. Using the Cohen's kappa (Cohen, 1960;McHugh, 2012) statistic, we found a strong (κ = 0.95) level of agreement between the researchers with regards to the sentiment labels (see Table 3 for contingency  table), and a strong (κ = 0.88) agreement for the 5C labels (see Table 4

Data Analysis
Overall, people expressed significantly high negative emotions in the Constraints category, similar levels of positive and negative emotions in the Com- placency category and have been otherwise always positive towards vaccination depicting high Confidence and Calculation behaviours (Figure 1). The most eminent concerns regarding negative emotions have been related to the constraints caused by vaccine roll out, Pfizer vaccine, aged care facilities, and the government and its leader (Table 5).
On the other hand AstraZeneca and Pfizer vaccines and COVID vaccine in general lead among positive topics, with high calculations and confidence associated with them (Table 5). A high number of tweets with the Positive Calculation category relate to the tweets in support of AstraZeneca with their own experience as evidence to disprove the tweets associating blood clot issues with that vaccine. This causes AstraZeneca to be a dominant unigram in both positive and negative instances. As the pandemic prevailed from 2020 to 2022, negative attitudes towards vaccination continuously varied in terms of prevalence and corresponding 5C reasoning. During 2020, the prevalence of negative stances has been relatively stable apart from the peaks around August and December, and the   shallow dips between the end of May to the end of July and in September (Figure 2 (left)). The shallow period from the end of May 2020 to the end of July 2020 coincides with the event of high approval towards the government for proactively closing the border, implementing lockdown, releasing super funds. Negative stances have been stable from early 2021 until around the end of March, and peaked during April, possibly due to the panic caused by the blood clot from AstraZeneca vaccine. Multiple peaks have then appeared with prominent ones occurring in July, August and December 2021 (Figure 2 (right)). The peaks in July and August correlate with the public outrage caused by the shortage of vaccine doses, the continuous emergence of blood clotting incidences from AstraZeneca vaccine, and the government ignoring to use the offer of Pfizer vaccine from that company. Consequently, the peak in December 2021 correlates to high displeasure towards the government at that time.
From the beginning of 2020 until around November that year, people voiced strong lack of confidence with regard to vaccination, but has then changed to the constraints from December 2021 onward until September 2021 where the notion has turned back to lack of confidence. Collective responsibility has not been consequential much to negative emotions except during April 2020 to March 2020 (Figure 2 (left)). This may be due to the fact that people resisting to lockdowns around April 2020, pointing their right to freedom. Such emotions have reduced after June 2020, with many deaths due to Covid occurring during April 2020. High level of negative confidence seen from October 2020 to the end of that year may be related to the events of the development of vaccines and people being doubtful of their effectiveness due to some vaccinated people in other countries getting re-infected. The emergence of negative Complacency and Collective responsibility from February 2021 to March 2021 may happen because of public resistance to the government's allocated vaccine, pointing out their right to choose the vaccine type. High level of negative Constraints emotion depicted from April 2021 onward may be a consequence of combination of several factors: shortage of vaccine, fear of blood clotting from the As-traZeneca vaccine, the failure of government to secure Pfizer vaccine when it was first available.
The bulk of opinions about vaccination has been arising from people residing in Sydney and Melbourne, the most populated two cities in Australia (Australian Bureau of Statistics, 2022 August [cited 24 August 2022](b)), with Melbourne slightly dominating in terms of negative stances and people from both cities displaying quite similar levels of positive emotions (Figure 3). Both cities have shown identical behavior in terms of the 5C scale, with negative attitudes attributed mostly to lack of confidence and calculative behaviour, while positive attitudes related mostly to constraints, collective responsibility and confidence behaviour. The prevalence of tweets and people's variable emotions in Melbourne and Sydney towards vaccination may also be related to long lockdown periods in Victoria and New South Wales where these two cities are located, respectively (Australian Bureau of Statistics, 2022 August [cited 24 August

Vaccine behaviour identification
We perform three predictive modeling tasks in our analysis: (1) predicting the sentiment of tweets, (2) predicting the 5C vaccination behaviour in tweets, (3) predicting the combined sentiment and 5C categories in tweets: Task 1 -Sentiment Identification: The first task is a binary classification problem to identify regardless of the type of behaviour if the post expresses positive or negative attitude towards vaccination. This task is framed as a binary classification problem.
Task 2 -5C Categorisation: In task 2 our goal is to identify one of the 5C categories from the tweet content regardless of its sentiment. This task is useful to practitioners where there is no sentiment expressed, however, one of the 5C vaccine behaviours are expressed through posts. For example, a vaccine researcher might be expressing scientific data regarding the risks versus benefits of a vaccine in certain population groups without necessarily expressing an opinion. This task is formulated as a multi-class classification trained using categorical cross-entropy loss.
Task 3 -Combined Sentiment and 5C Categorisation : In this task, we categorise a post into one of the ten categories resulting from the two sentiments and the five vaccine behaviour types (10 classes). We formulate this task as a multi-class problem. To identify if the information in the sentiment and the vaccine attitudes are complementary, we compare the results with a model where Task 1 and 2 are trained separately but the results are combined after prediction.
For our benchmark models, we use BERT (Devlin et al., 2018) using the base-uncased English version. To make sure the results are reasonable, we compare Task 3 with a Gradient Boosting Machine (GBM) (Hastie et al., 2009) baseline using TF-IDF features. Prior to fitting the models, we pre-processed the tweet texts by removing stop words (only for GBM), tokenizing sentences and encoding the words to integers. We then randomly partitioned the dataset into train/test/validation sets in the ratio 7 : 1 : 2.
GBM, in Task 3, is a tree-based ML model which sequentially fits new models to enhance the accuracy of the estimated response variable supervised learning tasks such as the classification problems we address here. For each predictive modeling task, we use 50 trees, each with a maximum depth of  Table 6: Performance of BERT in predicting the sentiment (task 1), 5C categories (task 2) and combined sentiment and 5C during prediction.
15 and the minimum number of observations at each leaf node being also 15. The learning rate and column sample rate have been set to 0.1 and 0.4, respectively. We use grid search with 1 to 70 trees, having maximum tree depth in the range 3 to 7, with column sample rate of 0.4 to 1 and minimum rows from 1 to 100 to search for the best parameters in GBM using cross-validation. For BERT, we use the default parameters of the pre-trained model. Additionally, the number of nodes in the output layer equals the total number of classes to be predicted, the activation function of the output layer is "softmax", with the loss function being "categorical cross-entropy". We evaluate the performance of the models using the precision, recall and F1 scores which are commonly applied measures in classification problems. Because of the imbalanced nature of our problem, we use macro-averaged F1 to evaluate across multiple classes.

Results
Our results assert that the largest F1 value corresponds to positive sentiments (88%) from the sentiment task (task 1) and the Constraint class (79%) in predicting the 5C task (task 2) ( Table 6). Our results further show that BERT has correctly predicted the 5C categories 1191 times (Table 7).
For the combined sentiment and 5C prediction task (task 3), our prediction depicts that the negative Constraints (80%) and positive Calculation and Collective Responsibility categories (59% for both) have the highest F1 measure (   be visualized from Figure 4, in which we can see that classes with lower proportions have been less likely predicted. As such, we fitted a GBM model to predict the combined sentiment and 5C task, and found that, overall, BERT performed better than GBM (F1 score of 48%). We stress that GBM uses boosting, so it oversamples difficult instances, for example, less frequent ones, and thus can handle class imbalance in an indirect way to some extent. However, both GBM and BERT were to be affected by the class imbalance. When the weights of the low-frequency categories were increased to fix this issue, they reduced the accuracy of other categories, and hence we did not implement class weights in the models.

Conclusion
We presented the first large dataset for online monitoring vaccination behaviour in Australia, which is annotated by public sentiments towards vaccination and their psychological reasoning by the 5C scale. Our analysis has shown a close correlation between the sentiments of the tweets and the prominent events during the pandemic. Our analysis of vaccination behaviour from this dataset showed Figure 4: Comparison of precision and recall scores from the sentiment and 5C prediction task (task 3) with the distribution of 5C categories.
Model P% R% F1% BERT 53 38 48 GBM 36 28 32 Table 9: Comparison of the two benchmark methods for the combined sentiment and 5C tweet classification task.
large amount of negative emotions towards vaccination due to the constraints related to vaccine rollout caused by its shortage, delay in securing Pfizer vaccine when it was available, administering vaccination to aged care facilities, and the government's handling of the vaccination program. Using this dataset, we predicted the sentiments towards vaccination, the 5C behaviour and the combined sentiment and 5C behaviour using BERT. All these predictive tasks are based on classification models, and showed variable performance across classes due to the imbalance proportion of data across classes. We compared performance of BERT with GBM in predicting the combined sentiment and the 5C categories and found that BERT has performed better than GBM.
Our work provides a proof of concept in the application of 5C scales to monitor vaccination behaviour using social media and can be extended to other domains such as Facebook and Google Trends.

Acknowledgements
This work is funded by The University of Queensland's AI4Pandemic group whose focus is the application of data science, machine learning, statistical learning, applied mathematics, and computation for managing pandemics both at the epidemic and clinical level. We are thankful for their support and hope that this work contributes to the aims of the group.