{'fraction_non_alphanumeric': 0.031879083888852223, 'fraction_numerical': 0.007260246848392845, 'mean_word_length': 6.047441860465116, 'pattern_counts': {'":': 1, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
A Holistic Framework to Improve the Uptake and Impact of eHealth Technologies

Background Many eHealth technologies are not successful in realizing sustainable innovations in health care practices. One of the reasons for this is that the current development of eHealth technology often disregards the interdependencies between technology, human characteristics, and the socioeconomic environment, resulting in technology that has a low impact in health care practices. To overcome the hurdles with eHealth design and implementation, a new, holistic approach to the development of eHealth technologies is needed, one that takes into account the complexity of health care and the rituals and habits of patients and other stakeholders. Objective The aim of this viewpoint paper is to improve the uptake and impact of eHealth technologies by advocating a holistic approach toward their development and eventual integration in the health sector. Methods To identify the potential and limitations of current eHealth frameworks (1999–2009), we carried out a literature search in the following electronic databases: PubMed, ScienceDirect, Web of Knowledge, PiCarta, and Google Scholar. Of the 60 papers that were identified, 44 were selected for full review. We excluded those papers that did not describe hands-on guidelines or quality criteria for the design, implementation, and evaluation of eHealth technologies (28 papers). From the results retrieved, we identified 16 eHealth frameworks that matched the inclusion criteria. The outcomes were used to posit strategies and principles for a holistic approach toward the development of eHealth technologies; these principles underpin our holistic eHealth framework. Results A total of 16 frameworks qualified for a final analysis, based on their theoretical backgrounds and visions on eHealth, and the strategies and conditions for the research and development of eHealth technologies. Despite their potential, the relationship between the visions on eHealth, proposed strategies, and research methods is obscure, perhaps due to a rather conceptual approach that focuses on the rationale behind the frameworks rather than on practical guidelines. In addition, the Web 2.0 technologies that call for a more stakeholder-driven approach are beyond the scope of current frameworks. To overcome these limitations, we composed a holistic framework based on a participatory development approach, persuasive design techniques, and business modeling. Conclusions To demonstrate the impact of eHealth technologies more effectively, a fresh way of thinking is required about how technology can be used to innovate health care. It also requires new concepts and instruments to develop and implement technologies in practice. The proposed framework serves as an evidence-based roadmap.

The framework proposes a user-centered design approach for telemedicine systems by taking the first step of mapping the underlying theoretical dimensions relevant for teleconsultations, taking the patient-provider interaction as the starting point. Usercentered design is a design approach in which the needs and requirements of users are considered at each stage of the design process.
User-centered design approach Users: patients and healthcare providers Theoretical dimensions relevant for teleconsultations (1) individual context (patient, provider, disease characteristics) (2) organizational context (eg, compatibility, facilitating conditions) (3) technological context (eg, medium characteristics, mode/interactivity) (4) teleconsultation process: communications & perceptions (5) process evaluation: health outcomes, satisfaction, adoption 2 Catwell & Sheikh, 2009, Evaluating eHealth interventions: the need for continuous systemic evaluation [2] The framework proposes a comprehensive overall evaluation approach, one that encourages a multifaceted, multidisciplined approach and facilitates continuous systematic evaluations throughout the lifecycle of an eHealth intervention. The authors state that RCTs alone fail to take sufficient account of the contextual considerations; these design methodologies alone are often less well suited to evaluate the impact of eHealth interventions in a complex environment. According to the authors, design teams need to gain a thorough understanding of the stakeholders' needs, concerns, values, and beliefs, and define (as far as possible) what the eventual system will be expected to provide.
Multifaceted, multidisciplined approach The authors state that it is important that design teams take a multifaceted and multidisciplined approach to document the complex relationships between the (1) political, Structure a debating tool that stakeholders can access in order to know their own health system better. The framework provides evaluation dimensions for addressing the fit between human, organization, and technology factors. The HOT-fit framework can and should be applied in a flexible way, taking into account different contexts and visions, stakeholders' point of views, phases in the system development life cycle, and evaluation methods.

Stakeholders participation
The term "stakeholders" is mentioned, but not specified by the authors. The term "user" refers to: Evaluation methods This framework can be applied using qualitative, quantitative or a combination of both approaches. Methods are presented via a case study: -A formative evaluation was undertaken of the adoption of FIS to identify system problems as they emerged and to improve the system as it was developed.
-Qualitative methods were employed to generate a fuller description of the healthcare setting and its cultural issues and to understand why the system functioned well or poorly in a particular setting.
-Snowball sampling method was used in order to gain in-depth information from key informants about the development of the FIS.
-During observations and face-to-face interviews, individuals including users, clinicians and IT staff that were involved with the system were queried about their system use and patient pathways. 4 Hamid & Sarmad, 2008, Evaluation of e-health services: user"s perspective criteria [4] The framework proposes user-centered evaluation criteria for eHealth services. The authors state that the evaluation criteria can serve as part of an eHealth evaluation framework. A sequential multi-method research approach is adopted by the authors. The framework only considers one stakeholder or a group of stakeholders with a common perspective in an evaluation process; in this study, it is the user's perspective.

User"s perspective Users not specified
User-centered evaluation criteria (1) costs (money and time saving) (2) benefits (effort saving, quality, access) (3) easy to learn/use (to work with a service) (4) accessibility (of content and user interface) (5) compatibility (fit into the healthcare system) (6) functionality (eg, information accuracy, technical functionality) (7) user satisfaction (utility, reliability, efficiency, customization, flexibility) Sequential multi-method research approach Methods not mentioned 5 Pagliari, 2007, Design & evaluation in eHealth: challenges and implications for an interdisciplinary field [5] Framework to facilitate interdisciplinary collaboration between software developers and health services researchers. The author discusses the importance of research for ensuring that new eHts are adopted and effective. Evaluation should ideally be approached as a longitudinal process occurring through a series of overlapping and iterative stages relevant to the maturity of the technology in its lifecycle, from initial conception to rollout. The framework presents the evaluation research methods during the development and implementation process.
Interdisciplinary collaboration Disciplines: (1) software designers (2) researchers: medical, social, management, legal scientists Iterative evaluation stages (1) evaluation of concepts and prototypes (lab): drafting new interventions based on an assessment of stakeholder needs and theory (2) evaluation of impacts: assessing the impact of the innovations on the processes and outcomes of care in selected target settings (experimental studies) (3) pragmatic evaluation: evaluating systems after roll-out (assess impact) Evaluation research methods (1) Longitudinal process studies (2) Multiple methods: rigorous qualitative methods (eg, ethnographic studies), and quantitative methods (clinical trials). The author states that controlled trials may be ideal for studying the impact of eHealth systems on measures of clinical outcome or efficiency, but they are poorly suited to exploring social, contextual, or technical barriers to adoption and certainly will have little to offer developers designing a new Web interface. Conversely, think aloud methods may be extremely useful for assessing the usability of a decision-support tool but say very little about its clinical validity or effectiveness. 6 Kaufman et al., 2006, Evaluation framework for health information system design, development and implementation [6] The framework provides a heuristic for matching the stage of system design and the level of evaluation (continuous evaluation). A usercentered approach to design is presented. The authors state that the incorporation of sound evaluation methodologies throughout the stages of system development is necessary to increase the potential of information systems in order to influence healthcare processes and outcomes positively. The framework proposes determinants of success of in-patient clinical information systems. The authors state that the framework is useful in evaluating patient care information systems, with modifications to include contingent factors, such as user involvement during system development and implementation and organizational culture. The authors also state that an evaluation should start before the development and should have no fixed end (continuous formative evaluation). In evaluations of information systems that employ multiple methods, the data from different sources complement each other to provide a more complete picture.

Multiple methods
The integration of qualitative (observations, interviews) and quantitative (questionnaires, work sampling) data collection methods provides an opportunity to improve the quality of the results through triangulation. 9 Shaw, 2002, "CHEATS": a generic information communication technology (ICT) evaluation framework [9] The framework (guideline for gathering information) provides a comprehensive evaluation strategy and a multidisciplinary approach. The CHEATS framework comprises of six evaluation aspects involved in systems design, implementation and use should be taken into account. Comprehensive evaluation strategy (methods) This involves a continuing process of semi-structured interviews with key participants (qualitative data), as well as the collection of quantitative data, from questionnaires and existing data, about service use and clinical effectiveness (beyond RCTs, ICT is not a drug and should not be evaluated as such). 10 Kazanjian & Green, 2002, Beyond effectiveness: the evaluation of information systems using a comprehensive health technology assessment framework [10] The framework provides guidelines for information seeking during development (four key dimensions) for decision-making about the adoption of health information technologies; identifying stakeholders, needs-assessment (problems, solutions), value specification (beneficiaries, benefits of technology). Identifying relevant interest groups, wider social and political impact of technologies. A multidisciplinary approach (inclusion of all stakeholders) is presented.
Multidisciplinary development approach Stakeholders: (1) technology producers (2) providers (3) patients and society (primary stakeholders) (4) third-party payers (to know the impact of technology on resource use, implications for accountability) Key dimensions for decision-making (1) population at risk, population impact (disability, quality of life) (2) social context (ethical, legal, political concerns) (3) economic concerns (eg, optimization of total social returns by weighting estimated costs and perceived benefits) (4) technology assessment (eg, increased understanding of conflicting interests) 11 Kushniruk, 2002, Evaluation in the design of health information systems: application of approaches emerging from usability engineering [11] The framework underlines the importance of evaluation throughout the process of software development (continual evaluation). The framework provides continual evaluation methods (formative) from project planning to design and implementation.

Continual evaluation Evaluation phases:
(1) planning (needs analysis; eg, workflow analysis) (2) analysis (requirements, eg, interviews) (3) design (eg, usability testing) (4) implementation (eg, programming, usability testing) (5) support (eg, maintenance; outcome-based evaluations) Continual evaluation methods The framework considers evaluation methods ranging from controlled experimental approaches to naturalistic approaches (ethnographic). Usability testing is presented as a key method for conducting evaluations during iterative system development.
Integration of data collection from multiple methods (process outcomes + summative outcomes) 12 Hebert, 2001, Telehealth success: evaluation framework development [12] The framework provides performance indicators to assess telehealth success. Similar studies (eg, diabetic homecare) can be examined using the framework to extract commonalities and differences in where telehealth is effective as well as what variables demonstrate "success" (eg, satisfaction). The framework provides quality indicators for health information and services to patients on the Internet. Quality is classified as structural quality (the communication setting, infrastructure, and resources), process quality (the communication process itself), and outcome quality (the effect of communication The conceptual framework proposes key-enablers for successful deliverance of e-health services; the author states that eHealth programs should be based on a sound economic framework and deliver significant value for the investment. User-centered design is advantageous to provide services that are valuable to users. Multidisciplinary collaboration is necessary to assist in the development of effective and sustainable eHealth programs. User-centered design Users are defined as key healthcare players (the health triangle): (1) patients (2) practitioners: any healthcare professional eg, general practitioner or specialist (3) providers: healthcare service providers (eg, hospitals, medical and academic research institutions), diagnostic equipment providers, informatics and computer suppliers, professional associations, health management organizations, insurance companies, the Ministry of Health, pharmaceutical companies These key healthcare players should work together to develop, promote and deliver healthcare services. Technology is the linking factor between these key players.