An Assessment Framework for e-Mental Health Apps in Canada: Results of a Modified Delphi Process.

BACKGROUND: The number of e-mental health apps is increasing rapidly. Studies have shown that the use of some apps is beneficial, whereas others are ineffective or do not meet users' privacy expectations. Individuals and organizations that curate, recommend, host, use, or pay for apps have an interest in categorizing apps according to the consensus criteria of usability and effectiveness. Others have previously published recommendations for assessing health-related apps; however, the extent to which these recommendations can be generalized across different population groups (eg, culture, gender, and language) remains unclear. This study describes an attempt by Canadian stakeholders to develop an e-mental health assessment framework that responds to the unique needs of people living in Canada in an evidence-based manner. OBJECTIVE: The objective of our study was to achieve consensus from a broad group of Canadian stakeholders on guiding principles and criteria for a framework to assess e-mental health apps in Canada. METHODS: We developed an initial set of guiding principles and criteria from a rapid review and environmental scan of pre-existing app assessment frameworks. The initial list was refined through a two-round modified Delphi process. Participants (N=25) included app developers and users, health care providers, mental health advocates, people with lived experience of a mental health problem or mental illness, policy makers, and researchers. Consensus on each guideline or criterion was defined a priori as at least 70% agreement. The first round of voting was conducted electronically. Prior to Round 2 voting, in-person presentations from experts and a persona empathy mapping process were used to explore the perspectives of diverse stakeholders. RESULTS: Of all respondents, 68% (17/25) in Round 1 and 100% (13/13) in Round 2 agreed that a framework for evaluating health apps is needed to help Canadian consumers identify high-quality apps. Consensus was reached on 9 guiding principles: evidence based, gender responsive, culturally appropriate, user centered, risk based, internationally aligned, enabling innovation, transparent and fair, and based on ethical norms. In addition, 15 informative and evaluative criteria were defined to assess the effectiveness, functionality, clinical applicability, interoperability, usability, transparency regarding security and privacy, security or privacy standards, supported platforms, targeted users, developers' transparency, funding transparency, price, user desirability, user inclusion, and meaningful inclusion of a diverse range of communities. CONCLUSIONS: Canadian mental health stakeholders reached the consensus on a framework of 9 guiding principles and 15 criteria important in assessing e-mental health apps. What differentiates the Canadian framework from other scales is explicit attention to user inclusion at all stages of the development, gender responsiveness, and cultural appropriateness. Furthermore, an empathy mapping process markedly influenced the development of the framework. This framework may be used to inform future mental health policies and programs.

How funding agencies can support research use in healthcare: an online province-wide survey to determine knowledge translation training needs.

BACKGROUND: Health research funding agencies are increasingly promoting evidence use in health practice and policy. Building on work suggesting how agencies can support such knowledge translation (KT), this paper discusses an online survey to assess KT training needs of researchers and research users as part of a Canadian provincial capacity-building effort. METHODS: The survey comprised 24 multiple choice and open-ended questions including demographics, interest in learning KT skills, likelihood of participating in training, and barriers and facilitators to doing KT at work. More than 1,200 people completed the survey. The high number of responses is attributed to an engagement strategy involving partner organizations (health authorities, research institutes, universities) in survey development and distribution. SPSS was used to analyze quantitative results according to respondents' primary role, geographic region, and work setting. Qualitative results were analyzed in NVivo. RESULTS: Over 85 percent of respondents are interested in learning more about the top KT skills identified. Research producers have higher interest in disseminating research results; research users are more interested in the application of research results. About one-half of respondents require beginner-level training in KT skills; one-quarter need advanced training. Time and cost constraints are the biggest barriers to participating in KT training. More than one-half of respondents have no financial support for travel and almost one-half lack support for registration fees. Time is the biggest challenge to integrating KT into work. CONCLUSIONS: Online surveys are useful for determining knowledge translation training needs of researchers, research users and ultimately organizations. In this case, findings suggest the importance of considering all aspects of KT in training opportunities, while taking into account different stakeholder interests. Funders can play a role in developing new training opportunities as part of a broad effort, with partners, to build capacity for the use of health research evidence. Survey results would ideally be complemented with an objective needs assessment based on core competencies, and should be acted on in a way that acknowledges the complexity of knowledge translation in healthcare, existing training activities, and the expertise stakeholders already have but may not refer to as knowledge translation.

Translating evidence into practice: the role of health research funders.

BACKGROUND: A growing body of work on knowledge translation (KT) reveals significant gaps between what is known to improve health, and what is done to improve health. The literature and practice also suggest that KT has the potential to narrow those gaps, leading to more evidence-informed healthcare. In response, Canadian health research funders and agencies have made KT a priority. This article describes how one funding agency determined its KT role and in the process developed a model that other agencies could use when considering KT programs. DISCUSSION: While 'excellence' is an important criterion by which to evaluate and fund health research, it alone does not ensure relevance to societal health priorities. There is increased demand for return on investments in health research in the form of societal and health system benefits. Canadian health research funding agencies are responding to these demands by emphasizing relevance as a funding criterion and supporting researchers and research users to use the evidence generated.Based on recommendations from the literature, an environmental scan, broad circulation of an iterative discussion paper, and an expert working group process, our agency developed a plan to maximize our role in KT. Key to the process was development of a model comprising five key functional areas that together create the conditions for effective KT: advancing KT science; building KT capacity; managing KT projects; funding KT activities; and advocating for KT. Observations made during the planning process of relevance to the KT enterprise are: the importance of delineating KT and communications, and information and knowledge; determining responsibility for KT; supporting implementation and evaluation; and promoting the message that both research and KT take time to realize results. SUMMARY: Challenges exist in fulfilling expectations that research evidence results in beneficial impacts for society. However, health agencies are well placed to help maximize the use of evidence in health practice and policy. We propose five key functional areas of KT for health agencies, and encourage partnerships and discussion to advance the field.