{'fraction_non_alphanumeric': 0.024191908924578166, 'fraction_numerical': 0.015111472521515214, 'mean_word_length': 5.118573797678275, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Assessing the reporting quality of randomized controlled trials on COVID-19 vaccines: a systematic review

ABSTRACT This systematic review evaluated the reporting quality of COVID-19 vaccine randomized controlled trials (RCTs). Relevant RCTs published between July 20, 2020 and June 11, 2021 were identified in the PubMed database by two independent reviewers. Study quality was evaluated with the 2010 AND 2001 Consolidated Standards of Reporting Trials (CONSORT) adherence scores. A total of 22 RCTs were included. The median CONSORT adherence score according to the 2010 criteria was 21 (range, 12–25), thus indicating that 75% of the items in more than half of the RCTs had clear reports. Univariate analysis showed that CONSORT adherence scores were not predicted by category; analysis of variance also showed no significant difference between groups. Our results indicated that the overall quality of COVID-19 vaccine RCTs was very good. Current evidence indicates that a variety of COVID-19 vaccines are effective. No RCTs have reported serious adverse effects such as mortality.


Introduction
Evidence-based medicine is fundamentally dependent on the quality of available clinical evidence. The results of randomized controlled trials (RCTs) provide the highest level of primary evidence, and the use of large sample sizes improves the power of statistical tests and reduces the risk of bias. Due to the lack of targeted drugs for COVID-19, many countries began to concurrently develop COVID-19 vaccines in the early stages of the pandemic. 1 Vaccination is considered to be the most effective measure for preventing the further spread of COVID-19. 2 Vaccines stimulate the body's immune system to produce antibodies against a specific virus, thus reducing the probability of future infection. Vaccinations prevent 2-3 million deaths from infections annually. 3 The effectiveness and breadth of COVID-19 vaccination will be the main determinant of how long the pandemic will last. 4 The first approved COVID-19 vaccine was produced by Pfizer-BioNTech and has been widely administered in the UK. The need for the rapid development of vaccines to combat the COVID-19 pandemic has necessitated the introduction of temporary regulations to expedite the authorization of their use in humans. 5 As a result, the risk of side effects (e.g., serious disease, mortality) have only been based on experimental research results from the first three stages of vaccine development; epidemiological research results typically available in the fourth stage are currently lacking. 6 By April 2020, approximately 100 different COVID-19 vaccines had been developed by research and development departments in different countries all over the world, with some having proceeded to the human trial stage. 7 If sufficient protection can be obtained after the first vaccine dose, the second dose can be delayed; this would ensure that a greater number of people in regions with limited access to vaccines can receive the first dose. 8 Recently, an increasing number of RCTs have investigated the effectiveness of COVID-19 vaccines by comparing infection rates between vaccinated (experimental group) and unvaccinated (control group) individuals. 9 The purpose of this systematic review was to evaluate the quality of these RCTs and to summarize the effectiveness and adverse effects of currently available COVID-19 vaccines. The overarching aim was to provide a frame of reference to facilitate vaccination selection.

Search strategy
This review was conducted in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines that are in the supplemental materials. Relevant articles were identified by using the search terms "vaccines" and "COVID 19" in the PubMed database. The authors of the present review were not involved in the conduct of any previous RCTs pertaining to this topic.

Reporting quality assessment
All extracted data were independently compiled by two reviewers. The reporting quality of each study was evaluated using the 19-item 2001 Consolidated Standards of Reporting Trials (CONSORT) statement (Supplemental Table S1) and the 28-item 2010 CONSORT standardized evaluation checklist ( Table 1). The overall report quality score was referred to as the CONSORT adherence score.

Research selection and data extraction
The inclusion criteria comprised the following: (a) evaluation of a COVID-19 vaccine using a randomized controlled design; (b) use of a COVID-19 vaccine in the experimental group; and (c) articles published in English. Studies were excluded if they did not report safety or effectiveness data, or were duplicate publications or secondary reports of previously published RCTs. If the results of a single RCT were reported in multiple publications, the one with the most complete data was selected.
The difference between the level 1 screening (titles and abstracts), two reviewers were resolved through discussion.

Data collection
Two independent reviewers extracted the following data: first author name; year of publication; whether or not the term "RCT" was used in the study title; use of a structured or nonstructured abstract; experimental design and allocation ratio to the intervention and control groups; specific content recorded in the article or protocol; study setting; place of the data collection; drug information; primary and secondary outcomes; measurement information; methods used for sample size calculation and randomization; allocation concealment; blinding method; whether or not an intent-to-treat (ITT) analysis and subgroup analysis were performed; flowchart; recruitment and follow-up time; results for vaccine efficacy; experimental registration number; and source of funding. Any discrepancies were resolved by consensus between the two reviewers.

Statistical analysis
The main purpose of this study was to assess the quality of RCTs that have evaluated COVID-19 effectiveness and safety. Using CONSORT criteria, we assigned 1 point for each criterion and calculated the total score of each item. SPSS Statistics 25 was used to analyze the collected data, and descriptive statistics were used to calculate the median and mean. The linear regression coefficient generated by the CONSORT adherence scores was used as the dependent variable to obtain the regression coefficient and P value. The difference between the groups and whether the classification could predict the dependent variable were evaluated.

Results
As shown in the flow chart in Figure 1, a total of 11053 articles were retrieved from PubMed. Title and abstract screening excluded 10999 non-RCTs. Full texts of the remaining 54 studies were evaluated according to our predefined inclusion and exclusion criteria. Thirty-two studies were excluded, as they were either secondary reports of previous RCTs (29 articles) or duplicate studies (3 articles). Thus, a total of 22 studies were included in our analysis ( Table 2) 10-31 and their characteristics are summarized in Table 3; data are expressed as absolute counts and proportions.
The majority of the RCTs were conducted in countries in Europe and North America. The impact factor of most of the journals in which the studies were published (73%, n = 16) exceeded 30. Over half (63%, n = 14) of the studies included more than 500 participants. Most of the studies were either phase I or II vaccine trials; only 12% (n = 4) were phase III trials.
Inter-rater agreement for the 2010 CONSORT standardized evaluation checklist were classified via Cohen's κ statistic as substantial, good, or perfect (Table 1). CONSORT adherence scores ranged from 0 to 28. As the reported reference median CONSORT adherence score was 21 (range, 12-25), this  indicated that 75% of the items in more than half of the RCTs in the present review had clear reports. The results of the descriptive analysis showed that less than 10% of the studies had a score of <14, thus indicating that study quality was "very good" ( Table 1). The results showed the positive number of CONSORT in the frequency distribution diagram ( Figure 2). All studies provided detailed scientific background information and reported baseline participant characteristics in both the experimental and control groups. Clinical outcomes after vaccination were summarized and presented in the form of tables. Since univariate analysis did not show significant differences between categories, CONSORT adherence scores could not be predicted according to category. Analysis of variance showed that there was no significant difference between the groups ( Table 4).
We evaluated adverse event report scores based on the rating of the hazardous recommendations. Our analysis indicated that inactivated vaccines, nucleic acid vaccines, adenovirus vector vaccines, protein subunit vaccines, and other types of COVID-19 vaccines had good efficacy. These vaccines were also safe, as no serious adverse reactions such as death were reported (Supplementary Table S2).
The use of allocation concealment and blinding across the included studies is shown in Supplementary Table S3. Over half (59%, n = 13) of the studies used a centralized randomization method, and 82% (n = 18) used the blind method. Only 27% (n = 6) of the studies performed an ITT analysis. Studies that did not report the use of ITT were assumed to have not used this analysis method. Supplementary Table S4 summarizes the endpoints used in the 22 trials. Most trials ended with adverse reactions or immunogenicity; safety and adverse reactions were the most commonly reported outcomes (77%,     Therefore, CONSORT adherence scores were not predicted by category.

Discussion
The results of our review indicate that the quality of RCTs on COVID-19 vaccines was not affected by the specific stage of vaccine development that was under investigation. Indeed, the CONSORT adherence scores indicated that the reporting quality of these RCTs was very good. This finding is pertinent for governments worldwide, as they are responsible for the majority of funding for vaccine research and development.
Vaccines are one of the most effective and safest means for preventing the further spread of COVID-19. 32 A number of different factors may affect the quality of research reports. For example, study quality is often significantly associated with the type of funding source. Journals with more published papers have higher impact factors, which are often associated with increased study quality. Studies in such journals are more likely to have a large sample size and include a wide range of age groups, from adolescents to the elderly.
At present, the incidence of new COVID-19 cases has not plateaued in many countries. In addition, some countries have even reported mutated variants with increased transmissibility. The emergence of COVID-19 variants indicates that a second vaccine dose is necessary, as previous studies have found that vaccine effectiveness after the first dose decreases after a period of time. 33 A third dose can further maintain effectiveness over the long-term and should be considered in countries where the proportion of the population with both first and second doses has reached a certain threshold. 34 Different vaccine types can be selected by countries according to their actual situation. 35 The results of the present review indicate that the majority of the investigated vaccine types are very effective. From the conclusion that there is no significant difference in univariate analysis, it can be seen that the literature quality of different categories with different characteristics is similar. It can be concluded that the quality of these RCTs is very good; the reporting was very specific and detailed, regardless of journal impact factor, funding source, region in which trials were conducted, and sample size.
Some RCTs did not provide details regarding the random allocation of study participants, as well as whether allocation concealment was performed. Some RCTs did not report whether researchers or patients were blinded to treatment allocation. The current stage of COVID-19 vaccine research has mainly focused on outcomes pertaining to adverse reactions, immunogenicity, and vaccine effectiveness; the latter outcome has been limited by the inability to mass produce experimental vaccines for evaluation in clinical trials. Nevertheless, a plethora of studies on COVID-19 vaccines are planned or in progress, and their results will provide important data on actual vaccine effectiveness. To date, the completed RCTs on COVID-19 vaccines have been of very good quality; this may be attributed to the individual efforts of research personnel, as well as the large amount of invested human, material, and financial resources. High-quality RCTs not only provide a greater reference value for future studies, but also contribute more to the global efforts to combat the COVID-19 pandemic. This is pertinent, as greater challenges for vaccine development are expected with the continuous emergence of COVID-19 variants. Long-term studies are required to determine whether existing vaccines can effectively and safely prevent infection by different variant strains. 36,37 Nevertheless, we found that some studies omitted certain CONSORT checklist items, such as declaring that the study was a RCT in the title; this criterion was only satisfied in 59% of the included studies. The implementation of randomization was only described in 41% of the studies. Thus, the majority of studies did not adhere to the principles of randomization; alternatively, they may have followed these principles but failed to report it. This resulted in a reduced study quality to some degree.
Some limitations are acknowledged in the present review. For example, we did not perform a detailed analysis of study follow-up duration and specific types of adverse reactions. Furthermore, the included studies did not provide detailed data on participant race, sex, age, or other differences. In addition, as the univariate analyses did not yield statistically significant associations with CONSORT adherence scores, we were unable to conduct a multivariate analysis to adjust for confounding factors.
In conclusion, based on the use of the CONSORT criteria, we determined that the RCTs on COVID-19 vaccines that have been published to date are of very high quality. This may be attributed to not only the adherence of study authors to established research reporting guidelines, but also the strict evaluation of manuscripts by referees during the peer review stage.

Disclosure statement
No potential conflict of interest was reported by the author(s).

Data availability statement
Data are available upon reasonable request.