{'fraction_non_alphanumeric': 0.037257255073096227, 'fraction_numerical': 0.019419594152301985, 'mean_word_length': 5.096774193548387, 'pattern_counts': {'":': 0, '<': 2, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 3, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Post-marketing surveillance study on influenza vaccine in South Korea using a nationwide spontaneous reporting database with multiple data mining methods

Safety profiles of the influenza vaccine and its subtypes are still limited. We aimed to address this knowledge gap using multiple data mining methods and calculated performance measurements to evaluate the precision of different detection methods. We conducted a post-marketing surveillance study between 2005 and 2019 using the Korea Adverse Event Reporting System database. Three data mining methods were applied: (a) proportional reporting ratio, (b) information component, and (c) tree-based scan statistics. We evaluated the performance of each method in comparison with the known adverse events (AEs) described in the labeling information. Compared to other vaccines, we identified 36 safety signals for the influenza vaccine, and 7 safety signals were unlabeled. In subtype-stratified analyses, application site disorders were reported more frequently with quadrivalent and cell-based vaccines, while a wide range of AEs were noted for trivalent and egg-based vaccines. Tree-based scan statistics showed well-balanced performance. Among the detected signals of influenza vaccines, narcolepsy requires special attention. A wider range of AEs were detected as signals for trivalent and egg-based vaccines. Although tree-based scan statistics showed balanced performance, complementary use of other techniques would be beneficial when large noise due to false positives is expected.


Results
General characteristics. The basic characteristics of influenza vaccine-related AEs compared with those of other vaccines are provided in Table 1. More AEs were reported in females than in males for both influenza vaccines and all other vaccines, accounting for 68.77% and 57.30%, respectively. The proportion of children below 24 months in the influenza vaccine group was higher than that of other vaccines, while that of individuals 19 years or older in the influenza vaccine group was lower than that of other vaccines. When stratified by route of administration, most of the characteristics between the influenza vaccine and other vaccines were similar, except for the origin and year of the report: most came from a pharmaceutical company for all other vaccines (74.71%), while pharmaceutical companies with regional pharmacovigilance centers (RPVC) for influenza vaccines (52.84% and 42.57%, respectively) and the proportion of serious influenza vaccine-related AEs (5.18%) were less than that of other vaccines (8.06%).

Signal detection.
Compared to all other vaccines, we identified 36 AEs as safety signals of influenza vaccines detected by any data mining method at least once (see Supplementary Material S2 online). Among them, seven AEs (injection site inflammation, muscle weakness, quadriplegia, paraplegia, hyperventilation, cachexia, and narcolepsy) were not included in the labeling information and were detected by all data mining methods, except for paraplegia (not in TSS). For serious AEs of special interest, anaphylaxis, neuritis (a preferred term [PT] of GBS), and narcolepsy were detected as signals compared to all other vaccines by all data mining methods, and paralysis (a PT of Bell's palsy) was detected by IC and TSS (Table 2). Additionally, each method detected a similar number of safety signals (PRR, 29; IC, 30; TSS, 27).
Different findings have been reported for influenza vaccine subtypes using various methods. When using TIV as a comparator, 13 AEs were detected as safety signals by any data mining method at least once for QIV, and among them (see Supplementary Material S3 online), two safety signals (injection site bruising and cachexia) were not included in the labeling (Table 3). Moreover, each method generated a similar number of safety signals: 13 signals for PRR and TSS and 12 signals for IC. In contrast, we detected 59 AEs as safety signals at least once for TIV when compared to QIV; 26 safety signals were not included in labeling, but only 5 of these were overlapped by the three methods.
Compared to the egg-based vaccine, we identified 13 AEs as safety signals of cell-based vaccines using at least one data mining method (see Supplementary Material S4 online). Among them, four safety signals (injection site bruising, pneumonitis, encephalopathy, and tachycardia) were not included in the labeling, but only injection site bruising was detected by all three methods (Table 4). PRR, IC, and TSS detected 9, 13, and 6 safety signals, respectively. When using cell-based vaccines as a comparator, 30 AEs were detected as safety signals by any data mining method at least once for egg-based vaccines, 4 of which (leg pain, tremor, injection site mass, and cachexia) were not included in the labeling. PRR, IC, and TSS detected 20, 2, and 23 safety signals, respectively. Performance evaluation. We measured the performance indicators of each data mining method based on the confusion matrix, where all detected signals were categorized into true positive (TP), false positive (FP), true negative (TN), or false negative (FN) based on the reference standard. Overall, TSS demonstrated well balance. TSS showed the highest sensitivity (77.8%), positive predictive value (PPV, 77.8%), accuracy (72.1%), and area under the curve (AUC, 70.1%) and the lowest specificity (62.5%). In contrast, IC showed the highest specificity www.nature.com/scientificreports/ (94.7%) and negative predictive value (NPV, 64.3%). In addition, except for specificity, all PRR measurements were the lowest (Fig. 1).

Discussion
We conducted a safety surveillance study on influenza vaccines and their subtypes using a nationwide spontaneous reporting database with three different data mining methods. The total number of signals detected by disproportionality-based analysis and TSS for the influenza vaccine compared to other vaccines were similar, and few unexpected AEs for the overall influenza vaccine were observed. Additional signal detection according to subtype was also conducted, where we noted that a wider range of AEs were detected as signals for TIVs and egg-based vaccines. Regarding the evaluation for signal detection, TSS showed better performance than the other data mining methods, with the optimal balance among all performance measurements. From the results of the signal detection for the overall influenza vaccine, we identified that most signals were known AEs, such as application site disorders, body as a whole-general disorder, and musculoskeletal system disorders. All five serious AEs of special interest for influenza vaccine were detected as signals, except for febrile seizure. Similar findings were reported in two recent studies, both of which used two disproportionality-based methods (PRR and IC) for safety signal detection of influenza vaccines using the U.S. Vaccine Adverse Event Reporting System (VAERS), where GBS and paralysis were detected as signals by all three methods, respectively 20,21 . Among the seven detected signals not listed in the labeling in South Korea, injection-site inflammation and muscle weakness are reflected in the U.S. Food and Drug Administration (US-FDA) labeling. Accordingly, injection site inflammation and muscle weakness should be updated after signal evaluation. Of the Table 1. Characteristics of adverse event reports following influenza vaccination and all other vaccines between 2005 and 2019. Abbreviations: HCPs, healthcare professionals; PMS, post-marketing surveillance; RPVC, regional pharmacovigilance center. α P values indicate differences between influenza vaccine and all other vaccines. P < 0.05 was considered statistically significant.  www.nature.com/scientificreports/ remaining five unlabeled signals in both the U.S. and South Korea, narcolepsy seems to require urgent assessment. As mentioned earlier, narcolepsy cases grew rapidly following the 2009-2010 H1N1 vaccination in Finland and five other European countries (see Supplementary Material S5 online). Subsequent studies have shown that the AS03-adjuvanted pandemic vaccine Pandemrix (GlaxoSmithKline Biologicals, Wavre, Belgium), which was widely used in these countries during that period, contains a protein structurally similar to the orexin receptor, which is likely to induce an autoimmune response that destroys orexin cells in the brain and causes narcolepsy 22 . The H1N1 vaccine adjuvanted with AS03 was not used in Korea, but we identified narcolepsy as a safety signal for the influenza vaccine compared with other vaccines in this study, although there were only 11 reported cases. Therefore, signal validation through an in-depth evaluation of these cases is required. Furthermore, we noted that the distribution of AEs differed between QIVs and TIVs. Local reactions at the injection site were more frequently reported in QIVs than in TIVs in our studies, which is consistent with previous studies. A meta-analysis of 5 RCTs reported that the incidence of injection site pain was higher with QIVs than TIVs with a pooled risk ratio of 1.18 (95% confidence interval: 1.03-1.35) 23 . In addition, a wider range of AEs were detected as signals for TIVs in our study, which showed inconsistent results with a previous RCT 24 that demonstrated that QIVs and TIVs showed similar profiles. This may be attributed to the different distribution of age groups between QIVs and TIVs, with the proportion of the older group (65 years and above) being 0.47% and 10.1%, respectively. The aged population is vulnerable to adverse drug reactions due to age-related physiological changes 25 .
Moreover, we detected a wider range of AEs as signals with egg-based vaccines compared with cell-based vaccines, suggesting that safety profiles among these two subtypes may differ, which is inconsistent with the results of previous studies. Two RCTs demonstrated that cell-based and egg-based TIVs had comparable safety profiles in children and adolescents < 18 years of age 26,27 . We believe that this situation may be due to the following reasons. First, compared with cell-based vaccines, an inherent drawback of egg-based vaccines is that impurities or exogenous contamination may cause AEs 28 . In addition, Korea approved the first cell-based influenza vaccine in 2014, and egg-based influenza vaccines still dominate the majority of the market share 29 . In our database, cellbased vaccines only accounted for 15% of all cases; therefore, it is possible that some AEs of cell-based vaccines have not yet been revealed. Therefore, it is necessary to collect additional information.
In this study, TSS had the highest accuracy (72.1%), AUC (70.1%), PPV (77.8%), and sensitivity (77.8%). TSS allows the simultaneous evaluation of a large number of individual AE terms and related AE groups, thereby resolving multiple testing problems of disproportionality-based methods 30 . Two previous studies of the BCG vaccine and pneumococcal vaccine using the KAERS database also showed balanced performance in TSS 17,18 .    www.nature.com/scientificreports/ considering the highest sensitivity in TSS, which were about threefold higher compared with the other methods (TSS: 77.8%, PRR: 21.1%, IC: 25.5%). This result suggests that the utilization of multiple data mining methods is reasonable choice to detect safety signals conservatively. In addition, the low values of accuracy and AUC were observed across all methods (TSS: 72.1%, 70.1%; PRR: 63.2%, 57.1%; IC: 66.1%, 60.1%). These results might be influenced by the reference standard that was used for constructing a confusion matrix for performance measurement. However, as the bias from reference standard could affect the performance of all methods, we believe that this bias is unlikely to distort the results of performance. Another possibility is that these methods are not precise or advanced to detect safety signals appropriately. Recently, machine learning algorithms have been utilized in the pharmacovigilance field with relatively high performance, compared with the traditional data mining methods 33 . However, we did not utilize this method in this study. Therefore, further research involving a variety of new methods such as machine learning is needed.

Conclusions
In this post-marketing surveillance study, we identify 7 new safety signals for influenza vaccine including narcolepsy as a serious AE, and observed TIVs and egg-based influenza vaccine showed a wide range of new safety signals when compared to QIVs and cell-based influenza vaccine, respectively. Further studies are needed to confirm the potential relationship between influenza vaccine and new safety signals, and meanwhile, healthcare providers should evaluate whether vaccinees are susceptible to experiencing these safety signals by an in-depth review of comorbidities or AE-related history to prevent an avoidable risk. In addition, although TSS showed balanced performance in this study, there is still discrepancy in the identified safety signals among different methods, and some measurements were lower than the others. This result suggests that the complementary use of multiple data mining methods should be utilized to detect new safety signals conservatively.

Data source. We used the Korea Institute of Drug Safety and Risk Management-Korea Adverse Event
Reporting System Database (KIDS-KAERS database) 34 from 2005 to 2019. This database includes all spontaneous AE reports in South Korea and detailed information on demographics, vaccination, AEs, causality assessment, and clinical information. Information on the vaccine product was also provided. All drugs and AEs are coded according to the Anatomical Therapeutic Chemical Classification (ATC) system and the World Health Organization-Adverse Reaction Terminology (WHO-ART) dictionary 35,36 , respectively. WHO-ART is established as a tree structure, including system organ class (SOC), high-level terms (HLT), PTs, and included terms (IT) from top to bottom, and the signals were determined at the PT levels in this study. The need for informed consent was waived, as this study was conducted using a spontaneous reporting database which does not include personal information. This study was approved by the Institutional Review Board of Chung-Ang University in South Korea (1041078-202008-ZZ-209-01). The Institutional Review Board of Chung-Ang University has waived informed consent for this study. All methods were carried out in accordance with the relevant guidelines and regulations. Study vaccine. All vaccines available in Korea were included. The study vaccine was an influenza vaccine, and all other vaccines were used as comparators for signal detection. Nineteen influenza vaccine products have been approved and used in Korea, with 2 cell-based vaccines, 17 egg-based vaccines, 8 TIVs, and 11 QIVs. A full list of influenza vaccine products used in our study is given in Supplementary Material S1 online.
Selection of AE reports. From all AEs following vaccination collected in our database from 2005 to 2019, we excluded the following: (1) those that were reported as a concomitant vaccine, not a suspected vaccine; (2) those that were not final reports; (3) those that were invalid reports (missing or unspecified drugs or AE codes), and (4) those with report errors (logical error).
From a total of 42,211 reports in our database, we identified 38,221 reports after vaccination. Among these, 17,378 (45.5%) were reported for the influenza vaccine. We identified 17,280 reports, of which 7639 (44.2%) were reported for QIV, 9641 (55.8%) for TIV, 2744 (15.9%) for cell-based influenza vaccine, and 14,536 (84.1%) for egg-based influenza vaccine. A flowchart of the study is presented in Fig. 2.

Selection of five AEs of special interest.
Based on literature REVIEWS, we selected the following five possible serious and important AEs observed after influenza vaccination: GBS, febrile conversion, anaphylaxis, narcolepsy, and Bell's palsy. Detailed information is provided in Supplementary Material S5 online.

Statistical analyses. Descriptive analyses.
We calculated the frequency and proportion of baseline demographic characteristics (e.g., sex and age group) and reports (e.g., report type, source, reporter, seriousness, and causality assessment). Chi-squared tests were performed to compare the distribution of baseline characteristics between the influenza vaccine and all other vaccines or among individual subtypes of influenza vaccine (TIVs vs. QIVs and egg-based vs. cell-based vaccines).
Algorithms for signal detection. To detect the signals, three signal detection algorithms were used in this study, including PRR, IC, and TSS.  37 . The thresholds of signals for each algorithm were defined as follows: PRR ≥ 2, chi-squared test ≥ 4, number of reported AEs ≥ 3, IC with a lower limit of 95% confidence interval ≥ 0.
Tree-based scan statistic (TSS). TSS is based on log-likelihood ratio (LLR) statistics and is suitable for analyzing a hierarchical structure variable. In this study, a diagnosis tree defined by the WHO-ART hierarchical structure was constructed, and the signals were determined at the PT level to compare them with the signals detected by the other algorithms. The unconditional Bernoulli model was selected to compare the number of AE reports between the two drugs. The expected and observed counts at each leaf were generated under the null hypothesis of an independent relationship between the vaccine and AE. AEs were defined as signals when the p-value of LLR was lower than 0.05, and p-values were generated using a Monte Carlo simulation. The analysis was performed using TreeScan Software 38 .
Performance evaluation of signal detection algorithms. The performance of disproportionality analysis (PRR, IC) and TSS was evaluated based on performance indicators such as sensitivity, specificity, PPV, NPV, accuracy, and AUC to identify which algorithm was more appropriate for detecting safety signals for influenza vaccines. To evaluate the performance indicators, a reference standard was established based on the MFDS-approved package inserts downloaded on September 13, 2020, from the NeDRUG website of the MFDS 39 .

Data availability
The data underlying this study are from the KAERS which has been transferred to the KIDS.