Mapping Theories, Models, and Frameworks to Evaluate Digital Health Interventions: Scoping Review.

BACKGROUND: Digital health interventions (DHIs) are a central focus of health care transformation efforts, yet their uptake in practice continues to fall short of their potential. In order to achieve their desired outcomes and impact, DHIs need to reach their target population and need to be used. Many factors can rapidly intersect between this dynamic of users and interventions. The application of theories, models, and frameworks (TMFs) can facilitate the systematic understanding and explanation of the complex interactions between users, practices, technology, and health system factors that underpin research questions. There remains a gap in our understanding of how TMFs have been applied to guide the evaluation of DHIs with real-world health system operations. OBJECTIVE: This study aims to map TMFs used in studies to guide the evaluation of DHIs. The objectives are to (1) describe the TMFs and the constructs they target, (2) identify how TMFs have been prospectively used (ie, their roles) in primary studies to evaluate DHIs, and (3) to reflect on the relevance and utility of our findings for knowledge users. METHODS: This scoping review was conducted in partnership with knowledge users using an integrated knowledge translation approach. We included papers (eg, reports; empirical quantitative, qualitative, and mixed methods studies; conference proceedings; and dissertations) if primary insights resulting from the application of TMFs were presented. Any type of DHI was eligible. Papers published from 2000 and onward were mainly identified from the following databases: MEDLINE (Ovid), CINAHL Complete (EBSCOhost), PsycINFO (Ovid), EBM Reviews (Ovid), and Embase (Ovid). RESULTS: A total of 156 studies published between 2000 and 2022 were included. A total of 68 distinct TMFs were identified across 85 individual studies. In more than half (85/156, 55%) of the included studies, 1 of following 6 prevailing TMFs were reported: Consolidated Framework for Implementation Research (n=39); the Reach, Effectiveness, Adoption, Implementation, and Maintenance Framework (n=17); the Technology of Acceptance Model (n=16); the Unified Theory on Acceptance and Use of Technology (n=12); the Diffusion of Innovation Theory (n=10); and Normalization Process Theory (n=9). The most common intended roles of the 6 TMFs were to inform data collection (n=86), to inform data analysis (n=69), and to identify key constructs that may serve as barriers and facilitators (n=52). CONCLUSIONS: As TMFs are most often reported to be applied to support data collection and analysis, researchers should consider more clearly synthesizing key insights as practical use cases to both increase the relevance and digestibility of their findings. There is also a need to adapt or develop guidelines for better reporting DHIs and the use of TMFs to guide evaluation. Hence, it would contribute to ensuring ongoing technology transformation efforts are evidence and theory informed rather than anecdotally driven.

What We Have Heard: Next Steps for Long-Term Care Pandemic Preparedness in Canada.

In this concluding article, Healthcare Excellence Canada and the Canadian Institutes of Health Research reflect upon and respond to the lessons learned from the contributing articles in the special issue and summarize key takeaways for the next steps in evidence-informed pandemic preparedness in long-term care in Canada. The implications of their cross-organizational partnership for achieving collective impact now and in the future are also discussed.

The value of connected health information: perceptions of electronic health record users in Canada.

BACKGROUND: As health care becomes more complex, it becomes more important for clinicians and patients to share information. Electronic health information exchange can help address this need. To this end, all provinces and territories (PTs) in Canada have created interoperable electronic health records (iEHRs). These secure systems offer authorized users an integrated view of a person's healthcare history across the continuum of care. They include information such as lab results, medications, diagnostic images, clinical reports and immunization profiles. This study explores user experiences and perceived outcomes of iEHR use. METHODS: Surveys conducted between 2006 and 2014 asked iEHR users in six Canadian PTs about system, information and service quality; iEHR use and user satisfaction; and net quality and productivity benefits. The surveys had a core set of questions that used Likert-type scales. Results were synthesized across surveys for each evaluative dimension. Consensus among researchers and subject matter experts on whether to classify the outcomes as positive, mixed/neutral, or negative was established using a modified Delphi technique. RESULTS: A total of 2316 iEHR users responded to the six surveys. Information quality was the most studied area. Results varied across PTs, but positive outcomes were more common than mixed/neutral or negative outcomes by a 19:1:1 ratio across this dimension. The next most frequently studied aspects were user satisfaction, the impact of iEHR use on quality of care, and the impact on productivity. In all three areas, there were more positive than mixed/neutral or /negative results (ratios of 13:1:1, 14:3:1, and 15:2:1respectively). CONCLUSIONS: Overall, users of iEHRs that provide secure access to patient information collated from across the health system tend to report positive outcomes, including quality of care and productivity. This study is an important first step in understanding user perspectives on iEHRs and health information exchange more broadly.

The Effects of Web-Based Patient Access to Laboratory Results in British Columbia: A Patient Survey on Comprehension and Anxiety.

BACKGROUND: Web-based patient access to personal health information is limited but increasing in Canada and internationally. OBJECTIVE: This exploratory study aimed to increase understanding of how Web-based access to laboratory test results in British Columbia (Canada), which has been broadly available since 2010, affects patients' experiences. METHODS: In November 2013, we surveyed adults in British Columbia who had had a laboratory test in the previous 12 months. Using a retrospective cohort design, we compared reported wait-time for results, test result comprehension, and anxiety levels of "service users" who had Web-based access to their test results (n=2047) with those of a general population panel that did not have Web-based access (n=1245). RESULTS: The vast majority of service users (83.99%, 95% CI 82.31-85.67) said they received their results within "a few days", compared to just over a third of the comparison group (37.84%, 95% CI 34.96-40.73). Most in both groups said they understood their test results, but the rate was lower for service users than the comparison group (75.55%, 95% CI 73.58-77.49 vs 84.69%, 95% CI 82.59-86.81). There was no significant difference between groups in levels of reported anxiety after receiving test results. CONCLUSIONS: While most patients who received their laboratory test results online reported little anxiety after receiving their results and were satisfied with the service, there may be opportunities to improve comprehension of results.

Understanding the gap between desire for and use of consumer health solutions.

Modern healthcare is more complex than ever before, with a broader range of care providers, organizations, diagnostic approaches and treatments. The result is that accurate and timely information is more important than ever. In response, clinical use of health information technology has grown significantly in recent years and there is growing interest in the use of consumer health solutions. In this article, the authors discuss the current landscape of the latter in Canada, enablers and barriers to their adoption and our readiness for change.

Drug information systems: evolution of benefits with system maturity.

Benefits from information and communication technology tend to grow over time as system use matures. This study examines pharmacists' experiences with provincial drug information systems (DIS) across Canada. At the time of survey, two provinces had more mature DIS (more than five years) and three provinces had less mature DIS (five years or less).

Opportunities for Action toward a Sustainable Health System.

The current pandemic is a stark reminder that crises bring to light society's vulnerabilities. In the lead paper of this issue of Healthcare Papers, Miller and Xie (2020) argue that the same is - and will be - true for climate change. They make a compelling and urgent case for its importance to health and healthcare in Canada and around the world. Opportunities to advance the multiple interrelated dimensions of sustainability in the health sector include understanding and mitigating the health implications of climate change; preparing the health sector for climate change; and accelerating the health sector's contribution to society-wide net-zero targets. High-performing, resilient health systems with their capacity to deeply engage with communities, and to respond dynamically to changing circumstances, will be key to proactively addressing climate change, just as they are proving to be in pandemic preparedness and response.

Identifying optimal frameworks to implement or evaluate digital health interventions: a scoping review protocol.

INTRODUCTION: Digital health interventions (DHIs) are defined as health services delivered electronically through formal or informal care. DHIs can range from electronic medical records used by providers to mobile health apps used by consumers. DHIs involve complex interactions between user, technology and the healthcare team, posing challenges for implementation and evaluation. Theoretical or interpretive frameworks are crucial in providing researchers guidance and clarity on implementation or evaluation approaches; however, there is a lack of standardisation on which frameworks to use in which contexts. Our goal is to conduct a scoping review to identify frameworks to guide the implementation or evaluation of DHIs. METHODS AND ANALYSIS: A scoping review will be conducted using methods outlined by the Joanna Briggs Institute reviewers' manual and will conform to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses Extension for Scoping Reviews. Studies will be included if they report on frameworks (ie, theoretical, interpretive, developmental) that are used to guide either implementation or evaluation of DHIs. Electronic databases, including MEDLINE, EMBASE, CINAHL and PsychINFO will be searched in addition to grey literature and reference lists of included studies. Citations and full text articles will be screened independently in Covidence after a reliability check among reviewers. We will use qualitative description to summarise findings and focus on how research objectives and type of DHIs are aligned with the frameworks used. ETHICS AND DISSEMINATION: We engaged an advisory panel of digital health knowledge users to provide input at strategic stages of the scoping review to enhance the relevance of findings and inform dissemination activities. Specifically, they will provide feedback on the eligibility criteria, data abstraction elements, interpretation of findings and assist in developing key messages for dissemination. This study does not require ethical review. Findings from review will support decision making when selecting appropriate frameworks to guide the implementation or evaluation of DHIs.

Nothing About Me Without Me.

Active and effective partnerships with patients are increasingly recognized as key to improving the design, delivery and organization of health services and policy. Co-designing for improvement yields new insights and tends to lead to better results than healthcare providers, researchers, or policy makers acting on their own. (For convenience, I use the term "patient" here since it has been identified as a term of choice in several Canadian focus groups. I intend it in an inclusive sense, not only referring to those who have a particular health problem or who use specific health services, but also family, friends, and other caregivers, as well as those who bring other important perspectives from their lived experience of interactions with the health and social sectors.).

When Small Changes Add Up to System Transformation.

Just as sandy beaches are the result of uncountable individual waves crashing on a shore, fundamental shifts in the health sector often reflect the cumulative result of hundreds - perhaps thousands - of individual decisions. Each, on its own, may seem to have only local effect. But together, they represent a transformation in how care is delivered.

Reflections on a Decade of Healthcare Policy/Politiques de Santé.

It has been a fantastic opportunity to be the editor for this journal over the last decade. I have very much enjoyed being exposed to the broad range of health services and policy research papers that have been submitted to the journal - more than 1,000 manuscripts that reflect a diverse set of questions, approaches, contributors and conclusions. The articles in this issue exemplify that diversity. Some themes continue to resonate strongly: health human resources, primary healthcare, and healthcare financing have been important topics since the journal was founded, for instance. But there have been changes too. Some topics - such as medical assistance in dying - have emerged as central questions in healthcare policy. We now see more research being undertaken with patients and communities. And the ratio of qualitative and mixed-methods research to quantitative analysis has risen.

Understanding and using the hospital standardized mortality ratio in Canada: challenges and opportunities.

In 2005, the Canadian Institute for Health Information (CIHI) began a methodological journey to develop a Canadian version of the hospital standardized mortality ratio (HSMR). For two years, CIHI worked with hospitals, regional authorities and measurement experts to define the most appropriate methodology given Canadian datasets and systems of care. In November 2007, we made the findings publicly available for regional health authorities and larger facilities. In their lead article, Penfold et al. discuss their views regarding some methodological issues and potential limitations of the HSMR to monitor quality of care and, in particular, as a patient safety indicator. Here we respond to their specific concerns and maintain that the HSMR remains an important tool in the arsenal of information hospitals can use to focus the discussion of patient safety/quality improvement, monitor the provision of care over time and identify opportunities for improvement.

Illuminating the Consequences of Policy Change.

The ripple effects of major policy changes are difficult to predict in advance. That is why the ability to track consequences of decisions - intended and unintended - is an important role of health services and policy research. In this issue of the journal, Janine Brown, Lilian Thorpe and Donna Goodridge discuss our ability to track medically assisted death (MAiD), one of the most significant health policy changes in Canada in recent history. Whatever one's views on MAiD, there is consensus on the need to understand how often, and in what circumstances, such deaths take place. Being able to track deaths in a consistent way also matters. Comparisons between different population groups, across regions of the country, and with other nations can be helpful in illuminating the effects of this policy and its application.

An Assessment Framework for e-Mental Health Apps in Canada: Results of a Modified Delphi Process.

BACKGROUND: The number of e-mental health apps is increasing rapidly. Studies have shown that the use of some apps is beneficial, whereas others are ineffective or do not meet users' privacy expectations. Individuals and organizations that curate, recommend, host, use, or pay for apps have an interest in categorizing apps according to the consensus criteria of usability and effectiveness. Others have previously published recommendations for assessing health-related apps; however, the extent to which these recommendations can be generalized across different population groups (eg, culture, gender, and language) remains unclear. This study describes an attempt by Canadian stakeholders to develop an e-mental health assessment framework that responds to the unique needs of people living in Canada in an evidence-based manner. OBJECTIVE: The objective of our study was to achieve consensus from a broad group of Canadian stakeholders on guiding principles and criteria for a framework to assess e-mental health apps in Canada. METHODS: We developed an initial set of guiding principles and criteria from a rapid review and environmental scan of pre-existing app assessment frameworks. The initial list was refined through a two-round modified Delphi process. Participants (N=25) included app developers and users, health care providers, mental health advocates, people with lived experience of a mental health problem or mental illness, policy makers, and researchers. Consensus on each guideline or criterion was defined a priori as at least 70% agreement. The first round of voting was conducted electronically. Prior to Round 2 voting, in-person presentations from experts and a persona empathy mapping process were used to explore the perspectives of diverse stakeholders. RESULTS: Of all respondents, 68% (17/25) in Round 1 and 100% (13/13) in Round 2 agreed that a framework for evaluating health apps is needed to help Canadian consumers identify high-quality apps. Consensus was reached on 9 guiding principles: evidence based, gender responsive, culturally appropriate, user centered, risk based, internationally aligned, enabling innovation, transparent and fair, and based on ethical norms. In addition, 15 informative and evaluative criteria were defined to assess the effectiveness, functionality, clinical applicability, interoperability, usability, transparency regarding security and privacy, security or privacy standards, supported platforms, targeted users, developers' transparency, funding transparency, price, user desirability, user inclusion, and meaningful inclusion of a diverse range of communities. CONCLUSIONS: Canadian mental health stakeholders reached the consensus on a framework of 9 guiding principles and 15 criteria important in assessing e-mental health apps. What differentiates the Canadian framework from other scales is explicit attention to user inclusion at all stages of the development, gender responsiveness, and cultural appropriateness. Furthermore, an empathy mapping process markedly influenced the development of the framework. This framework may be used to inform future mental health policies and programs.

International health IT benchmarking: learning from cross-country comparisons.

OBJECTIVE: To pilot benchmark measures of health information and communication technology (ICT) availability and use to facilitate cross-country learning. MATERIALS AND METHODS: A prior Organization for Economic Cooperation and Development-led effort involving 30 countries selected and defined functionality-based measures for availability and use of electronic health records, health information exchange, personal health records, and telehealth. In this pilot, an Organization for Economic Cooperation and Development Working Group compiled results for 38 countries for a subset of measures with broad coverage using new and/or adapted country-specific or multinational surveys and other sources from 2012 to 2015. We also synthesized country learnings to inform future benchmarking. RESULTS: While electronic records are widely used to store and manage patient information at the point of care-all but 2 pilot countries reported use by at least half of primary care physicians; many had rates above 75%-patient information exchange across organizations/settings is less common. Large variations in the availability and use of telehealth and personal health records also exist. DISCUSSION: Pilot participation demonstrated interest in cross-national benchmarking. Using the most comparable measures available to date, it showed substantial diversity in health ICT availability and use in all domains. The project also identified methodological considerations (e.g., structural and health systems issues that can affect measurement) important for future comparisons. CONCLUSION: While health policies and priorities differ, many nations aim to increase access, quality, and/or efficiency of care through effective ICT use. By identifying variations and describing key contextual factors, benchmarking offers the potential to facilitate cross-national learning and accelerate the progress of individual countries.

Applying the Resilient Health System Framework for Universal Health Coverage.

Since the 1978 Declaration of Alma-Ata affirming health as a fundamental human right, policy-makers and stakeholders have proposed many different strategies to achieve the goal of 'health for all'. However, globally there still remains a lack of access to health information and quality health care, especially in low- and middle-income countries (LMIC). Digital health holds great promise to improve access and quality of care. We propose using the "resilient health system framework" as a guide to scale-up digital health as a means to achieve universal health care (UHC) and health for all. This article serves as a call to action for all governments to include population-based digital health tools as a foundational element in on-going health system priorities and service delivery.

Valuing national effects of digital health investments: an applied method.

This paper describes an approach which has been applied to value national outcomes of investments by federal, provincial and territorial governments, clinicians and healthcare organizations in digital health. Hypotheses are used to develop a model, which is revised and populated based upon the available evidence. Quantitative national estimates and qualitative findings are produced and validated through structured peer review processes. This methodology has applied in four studies since 2008.