{'fraction_non_alphanumeric': 0.02255293739378072, 'fraction_numerical': 0.0061090441412888705, 'mean_word_length': 5.49910447761194, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Measuring patient satisfaction with video consultation: a systematic review of assessment tools and their measurement properties

Background Video consultation (VC) is considered promising in delivering healthcare closer to the patient and improving patient satisfaction. Indeed, providing care-at-distance via VC is believed to be promising for some situations and patients, serving their needs without associated concomitant costs. In order to assess implementation and perceived benefits, patient satisfaction is frequently measured. Measuring patient satisfaction with VC in healthcare is often performed using quantitative and qualitative outcome analysis. As studies employ different surveys, pooling of data on the topic is troublesome. This systematic review critically appraises, summarizes, and compares available questionnaires in order to identify the most suitable questionnaire for qualitative outcome research using VC in clinical outpatient care. Methods PubMed, Embase, and Cochrane were searched for relevant articles using predefined inclusion criteria. Methodological quality appraisal of yielded questionnaires to assess VC was performed using the validated COSMIN guideline. Results This systematic search identified twelve studies that used ten different patient satisfaction questionnaires. The overall quality of nine questionnaires was rated as “inadequate” to “doubtful” according to the COSMIN criteria. None of the questionnaires retrieved completed a robust validation process for the purpose of use. Conclusion and recommendations Although high-quality studies on measurement properties of these questionnaires are scarce, the questionnaire developed by Mekhjian has the highest methodological quality achieving validity on internal consistency and the use of a large sample size. Moreover, this questionnaire can be used across healthcare settings. This finding may be instrumental in further studies measuring patient satisfaction with VC.

Background. Video consultation (VC) is considered promising in delivering healthcare closer to the patient and improving patient satisfaction. Indeed, providing care-at-distance via VC is believed to be promising for some situations and patients, serving their needs without associated concomitant costs. In order to assess implementation and perceived benefits, patient satisfaction is frequently measured. Measuring patient satisfaction with VC in healthcare is often performed using quantitative and qualitative outcome analysis. As studies employ different surveys, pooling of data on the topic is troublesome. This systematic review critically appraises, summarizes, and compares available questionnaires in order to identify the most suitable questionnaire for qualitative outcome research using VC in clinical outpatient care. Methods. PubMed, Embase, and Cochrane were searched for relevant articles using predefined inclusion criteria. Methodological quality appraisal of yielded questionnaires to assess VC was performed using the validated COSMIN guideline.
Results. This systematic search identified twelve studies that used ten different patient satisfaction questionnaires. The overall quality of nine questionnaires was rated as "inadequate" to "doubtful" according to the COSMIN criteria. None of the questionnaires retrieved completed a robust validation process for the purpose of use.

Conclusion and recommendations.
Although high-quality studies on measurement properties of these questionnaires are scarce, the questionnaire developed by Mekhjian has the highest methodological quality achieving validity on internal consistency and the use of a large sample size. Moreover, this questionnaire can be used across healthcare settings. This finding may be instrumental in further studies measuring patient satisfaction with VC.
In short, "Telemedicine" (TM) is the delivery of healthcare services using technology while remaining at a physical distance (1). Video consultation (VC) is a form of TM, which allows healthcare providers to communicate with their patients over a real-time video connection (2). VC holds great potential for patients to receive care in the comfort of their own home, resulting in several benefits for both the patient and the healthcare provider (3). Despite its promise, the successful implementation of VC resulting into routine provision of care is scarce (4). Many initiatives appear not to be systematically embedded in care processes and risk becoming dormant once the initial funding to start up the initiative has ended.
In order to evaluate and validate the use of VC in health care, patient satisfaction with professional consultation via VC is regarded as an important driver. If patient satisfaction is well established, policy makers can be directed toward sustainable implementation. The literature on VC abounds with studies concerning patient satisfaction, generally suggesting favorable results concerning a decrease in patient-related expenditures while maintaining face-to-face interaction with a caregiver (5). The authors use a diverse range of questionnaires to measure patient satisfaction, resulting in heterogeneous data which make it difficult to compare and combine results. Cross-situational evidence is needed to support a stronger business case for policy makers. In other words, to determine patient satisfaction with VC, a valid and reliable tool which allows a consistent assessment is of importance.
Although reviews in literature have summarized the evaluation of available assessment tools to date (6)(7)(8), these studies focus on TM as a whole, not specifically on VC. Furthermore, available reviews focus on questionnaire development and validation studies only, without including other available studies on the evaluation of their measurement properties. In addition, a comprehensive overview of the evaluation of the quality of measurement properties is not available. Hence, an evidence-based recommendation in the selection of the most suitable questionnaire for patient satisfaction with VC is lacking.
In this systematic review, the COSMIN (Consensus-based Standards for the selection of health Measurements Instruments) methodology and guidelines are used to critically appraise and summarize the measurement properties of all available validated questionnaires that measure patient satisfaction with VC. The primary aim is to come to an evidence-based recommendation on the most suitable questionnaire for measuring patient satisfaction with VC which can be used across settings and by multidisciplinary teams.

Methods
This systematic review has been reported according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses Protocols (PRISMA-P) guidelines in combination with the protocol for systematic reviews of measurement properties recommended by the COSMIN panel. This study was registered with PROSPERO (CRD42016051841) (9).

Search Criteria
A systematic literature search was performed to provide an overview of questionnaires used to measure patient satisfaction regarding VC. Empirical studies reporting the development and validation of patient satisfaction questionnaires concerning VC were included. VC was classified as "any type of consultation facilitated or supported by using a real-time video connection between a healthcare provider and a patient." Store-and-Forward connections, and studies that shared information asynchronously, were excluded from the analysis. In addition, studies reporting the use of video with the main purpose of remote patient monitoring were excluded as well.
Patient satisfaction was defined as "patients' reported opinion regarding the use of VC in consultation in any medical setting." Articles solely measuring healthcare providers' satisfaction were excluded, as were studies conducting a standard patient satisfaction questionnaire without any relation to VC technology. The search strategy consisted of taxonomic matching terms of VC, physician-patient relations, and patient satisfaction (Supplementary Table 1).

Selection of Articles
The PubMed database, Embase database, and Cochrane Library were searched for relevant peer-reviewed articles. The search was conducted on 2 August 2019. Conference proceedings and reviews were not considered eligible for inclusion. Reports that did not relate to patient satisfaction or conducting patient satisfaction questionnaires and studies investigating the internal validity or technological aspects of a VC system were excluded from the analysis. Two reviewers screened all reports on title and abstract according to the aforementioned criteria. Reports deemed "relevant," "dubious," or "unknown" were examined in full text. The reference lists of the reports assessed for eligibility were searched for other relevant reports. Grey literature was searched using Google Scholar. None of the reports were excluded based on language. In case of missing data, the Internet was searched and study authors were contacted directly.

Data Extraction
The following general study characteristics were extracted from all reports: the name of the questionnaire, year of publication, study location, study design, and the purpose of using VC. Data concerning the questionnaires measurements included: type(s) of validity and reliability assessment, participants included in validation assessment, sample size, and other characteristics related to the measurement properties. All necessary data were collected by EZB and EVH.

Assessment of Methodological Quality of Included Studies
To evaluate the methodological quality of the questionnaires, the COSMIN Risk of Bias checklist was used (10). This checklist is a standardized tool to assess studies on measurement properties. It contains the assessment of several measurement properties on design aspects and statistical methods. For each study, two independent reviewers (EZB and EVH) assessed the methodological quality of items based on a four-point rating scale (inadequate, doubtful, adequate, and very good) (10). The overall score is determined by the lowest rating of any item on the checklist. In case of disagreement, there was a discussion to reach consensus.

Assessment of the Measurement Property of a Questionnaire
The result of each study was rated independently by EZB and EVH according to the updated criteria for good measurement properties (11). The measurement property of a questionnaire was labeled as either being sufficient ( + ), insufficient (−), or indeterminate (?).

Evidence Synthesis and Generating Recommendations
A summary of the strength of the evidence for the measurement properties for each questionnaire, including an overview of measurement properties, is provided. The quality of the evidence is qualitatively summarized and graded using the modified GRADE approach labeling findings as high, moderate, low, or very low in evidence. The modified GRADE approach uses the following three factors to determine the quality of the evidence: the risk of bias, the inconsistency of the results, and indirectness (12).

Study Selection
The systematic search identified 2,348 articles. Cross-reference search identified eighteen additional articles. After removing duplicates (n = 353), 1,995 articles remained and were screened for relevance on title and abstract. A total of 268 were eligible for full-text screening. A total of twelve articles described the development or description of measurement properties (3;13-23). Of the twelve articles included, ten different questionnaires were evaluated. Figure 1 illustrates the PRISMA flow chart of the process. An overview of the general characteristics of the included studies is reported in Table 1.
Lower quality ratings were mostly caused by not assessing or describing the dimensionality of a questionnaire and not assessing internal consistency for every subscale of a questionnaire separately. An overview of the methodological quality of the included studies is reported in Supplementary Tables 2a and 2b.

Assessment of Measurement Properties and Evidence Synthesis
A summary of the strength of the evidence for the measurement properties for each questionnaire, including an overview of measurement properties, is reported in Table 2.
Telehealth Usability Questionnaire (TUQ) The TUQ was developed by Parmanto et al. (21) and is based on several reported questionnaires in literature. The multidimensional questionnaire contains twenty-one items. The TUQ was assessed in one study for internal consistency in which the Cronbach's α was high for every subscale measured. However, the quality of the study was rated "doubtful" because the population used in the study did not match the target group of the questionnaire. The TUQ received an indeterminate rating because there was not enough information available on the structural validity. None of the articles retrieved by this systematic review mentioned the TUQ as a reference; however, two studies outside the scope of this review report the use of the TUQ (27;28).

Telemedicine Satisfaction and Usefulness Questionnaire (TSUQ)
The TSUQ developed in both English and Spanish was based on the TMPQ and includes fourteen items (14). The questionnaire was assessed on internal consistency and structural validity. Based on the COSMIN checklist, the risk of bias was rated as "very good" and a large sample size was used. Therefore, the quality of evidence for internal consistency was rated as "high." Several studies mentioned the TSUQ as a reference (16;29;30).

Telemedicine Satisfaction Questionnaire (TSQ)
The development of TSQ was based on previous literature after which a panel of doctors, nurses, and experts in TM reviewed the questionnaire (20). Changes were made accordingly. The TSQ and the translated Dutch version were assessed by two separate studies. Although the authors researched four measurement
properties, the risk of bias was assessed as "inadequate" to "doubtful" due to a small sample size and a lack of description of the methods used. The TSQ is mentioned by multiple studies inand outside the scope of this systematic review (20;31-33).

Telemedicine Satisfaction and Usefulness Questionnaire (TMPQ)
The TMPQ was developed based on published literature as well as focus groups with patients (15;23). The measurement properties were assessed in two studies. Both studies included small sample sizes and the risk of bias was assessed "inadequate" to "doubtful" mainly because of a lack of a description of the used methods. The TMPQ is mentioned by one other author within the scope of this systematic review (34).

Other
Several authors mentioned the lack of a validated questionnaire and therefore designed a specifically tailored questionnaire to use in their study (35)(36)(37). All reported questionnaires were developed based on previously used items in the literature (13;38;39). The risk of bias was rated "inadequate" to "doubtful," with the exception of the questionnaire developed by Mekhjian and et al. (19). This research group designed a questionnaire achieving validity on internal consistency and included a large sample size. The study was therefore assessed as of high methodological quality concerning internal validity.

Recommendations for the Most Suitable Questionnaire to Measure Patient Satisfaction with VC
The study performed by Mekhjian et al. (19) carries the best evidence for the validity of the measurement properties ( Table 2). Although other authors have adapted this questionnaire after publication, new validity studies have not yet been published. The TSUQ scored high quality on internal validity but the overall rating was indeterminate. The quality of the hypotheses testing for construct validity was considered moderate. The TSUQ is also reported by other authors. The questionnaire used by Mekhjian et al. (19) was designed for inmates of the Ohio prison. VC was used in order to prevent inmates from having to leave the prison to receive medical care. The received care was from multiple disciplines resulting in a more general focus on patient satisfaction with VC. Although the TSUQ scored second best, this questionnaire was specifically designed for diabetes care, making it more difficult to be used interdisciplinary.

Discussion
This systematic search retrieved twelve studies evaluating the measurement properties of ten different questionnaires to assess patient satisfaction with VC. A validated questionnaire that can be considered a gold standard when respecting the COSMIN criteria checklist was not identified to date. According to the COSMIN checklist for validation, the methodological quality of most studies was identified as "inadequate" to "poor." Main reasons for this were inadequate sample sizes and/or inadequate reporting of missing items. Evidence for the psychometric properties of the questionnaires was limited with few positive results on reliability and validity. The findings of this review are in agreement with previous systematic reviews, reporting the lack of validation (8).
An explanation for the lack of validated questionnaires in literature might be that the use of VC in clinical practice varies across healthcare settings. This makes the development of a patient satisfaction questionnaire that can be used across settings complex. As a result, questionnaires are designed to acquire knowledge for a specific specialty and tailored for a specific situation or indication to ensure their relevance. Furthermore, it results in a potpourri of questionnaires for single-use, not contributing to a robust body of evidence. Indeed, it is important that questionnaires are externally validated across settings. Therefore, it is encouraging that the recommended questionnaire based on this systematic review seems to be a good fit for multiple disciplines because of its interdisciplinary use.
This study systematically summarizes available evidence on the measurement properties of various questionnaires developed to assess patient satisfaction with VC. The use of the standardized COSMIN methodology for critical appraisal of the methodological quality is important in assessing outcome and has not been done before. However, one may comment that the strictness of the COSMIN methodology makes it hard for development studies to obtain a sufficient score when study quality is perceived to be sufficient but not excellent. In addition, questionnaires developed before the COSMIN guidelines were published may have not reported their methodology sufficiently to be ranked properly. Hence, it must be noted from our findings that we cannot conclude that questionnaires scoring low on the COSMIN criteria are in fact "inadequate." It merely demonstrates that the questionnaires have not been tested extensively or more importantly, that the methodology has not been reported accurately and findings cannot be assessed validly across situations. Furthermore, as Bagot et al. (40) pointed out, the selection of keywords when submitting an empirical paper is of great importance. Due to the omission of or the improper use of relevant keywords, it is possible that relevant articles were not retrieved via our search strategy. Although our effort to retrieve additional relevant articles through cross-referencing, this might have resulted in an incomplete overview of all the patient satisfaction questionnaires in the current medical literature.
As a rapid increase of technology leads to a predominance of pilot and feasibility studies with often small sample sizes, validating a specific questionnaire is time consuming, costly, and may not be a priority. The adaptation of a well-designed questionnaire to a specific situation is an elegant and smart solution when both information and evidence is needed. In that case, careful consideration of which questionnaire items to use might facilitate the combination of outcomes for meta-analyses in order to generalize results and improve the quality of the outcome. Collaborative research efforts to jointly use and collect the same outcome measurements may provide progress in increasing the quality of the assessment of patient satisfaction with VC.
Apart from future studies on the measurement properties of the questionnaire of Mekhjian et al. (19) and the TSUQ, a valuable suggestion is to define which domains and questionnaire items are important to measure patient satisfaction extensively.  Garcia and Adelakun (7) made the first attempt to identify contributing dimensions of patients' satisfaction with VC. They proposed a framework that provides guidance on which generic dimensions should be included to measure patient satisfaction appropriately. However, examples of questionnaire items to be used were not provided. Consensus on these items could establish a standardized collection of outcomes and might counteract heterogeneity in outcome measurements. This could facilitate a large step toward a uniform assessment of patient satisfaction toward VC.

Conclusion and Recommendations
This systematic review indicates that high-quality studies on measurement properties of patient satisfaction questionnaires with VC are scarce, and the need for such an instrument appeared to be much requested by diverse authors. The study reported by Mekhjian scores highest on the aspect of methodological quality using the COSMIN Risk of Bias checklist and seems to have the most potential for future and cross-sectional use. More studies on the measurement properties could further consolidate these recommendations, as additional validity studies on the questionnaire of Mekhjian need to be published hereafter.
Supplementary material. The supplementary material for this article can be found at https://doi.org/10.1017/S0266462320000367