Mapping review of 'proof-of-concept' in mental health implementation research using the TRL framework: a need for a better focus and conceptual clarification.

BACKGROUND: Proof-of-concept (PoC) development is a key step in implementation sciences. However, there is a dearth of studies in this area and the use of this term in health and social sciences is ambiguous. OBJECTIVE: The objective was to remove the ambiguity surrounding the PoC and pilot study stage in the research development process using a standard system to rate the development of projects and applications provided by the Technology Readiness Levels (TRL) framework. DESIGN: Mapping review and critical analysis using TRL as the standard measure. SEARCH STRATEGY AND CHARTING METHOD: PubMed and PsycInfo databases were searched for papers that reported PoC studies of mental health interventions up to August 2023. Data were extracted, described and tabulated. ELIGIBILITY CRITERIA: Included were PoC studies in mental health implementation research. Exclusion criteria were research relating to biomedical (drugs) development, neurocognitive tools, neuropsychology, medical devices, literature reviews or discussion papers or that did not include the term 'proof-of-concept' in the title, abstract or text. RESULTS: From the 83 citations generated from the database search, 22 studies were included in this mapping review. Based on the study title, abstract and text, studies were categorised by research development stage according to the TRL framework. This review showed 95% of the studies used PoC incorrectly to describe the development stage of their research but which were not at this specific level of project development. CONCLUSIONS: The TRL was a useful reference framework to improve terminological clarity around the term 'proof-of-concept' in implementation research. To extend the use of TRL in implementation sciences, this framework has now been adapted and validated to a health and social science-related research context accompanied by a health-related glossary of research process terms and definitions to promote a common vocabulary and shared understanding in implementation sciences.

Adaptation of the technology readiness levels for impact assessment in implementation sciences: The TRL-IS checklist.

Background: Intervention development is a critical process in implementation research. There are key stages involved in the process to design, pilot, demonstrate and release a technology or an intervention. The Technology Readiness Level (TRL) is a globally accepted instrument for assessing the maturity of research development. However, the original levels do not fit all, and some adjustments are required for its applicability in implementation sciences. Aims: This study aimed to gather the prior knowledge base on TRL in public and population health research; to develop a standard definition of readiness, and to adapt and validate the TRL to an implementation science context (TRL-IS). Materials and methods: A Mixed methods approach has been followed in this study. A scoping review using the PRISMA extension (PRISMA-ScR) informed a nominal expert panel for developing a standard definition of readiness and to modify the TRL following an ontoterminology approach. Then the maturity of six practical case study examples were rated by ten researchers using the modified TRL to estimate inter-rater reliability, and a group of experts provided final content and face validity and feasibility.This mixed methods study included 1) a scoping review to examine the current literature and develop a knowledge base, identify knowledge gaps and to clarify concepts; 2) the development of a standard definition of 'Readiness' and related terms; and 3) adaptation of the TRL to implementation science and development of a checklist to rate the maturity of applications.A standard definition of readiness and related terms was produced by the core team, and an international nominal group (n = 30) was conducted to discuss and validate the definition and terms, and the location of 'Readiness' in the initiation and early development phases of implementation.Following feedback from the nominal group, the development of the TRL-IS was finalised and a TRL-IS rating checklist was developed to rate the maturity of applications. The TRL-IS checklist was tested using six cases based on real world studies on implementation research.The inter-rater reliability of the TRL-IS was evaluated by ten raters and finally six raters evaluated the content and face validity, and feasibility, of the TRL-IS checklist using the System Usability Scale (SUS). Results: Few papers (n = 11) utilised the TRL to evaluate the readiness of readiness of health and social science implementation research. The main changes in the adaption of the TRL-IS included the removal of laboratory testing, limiting the use of "operational" environment and a clearer distinction between level 6 (pilot in a relevant environment) and 7 (demonstration in the real world prior to release). The adapted version was considered relevant by the expert panel. The TRL-IS checklist showed evidence of good inter-rater reliability (ICC = 0.90 with 95 % confident interval = 0.74-0.98, p < .001) and provides a consistent metric. Conclusions: In spite of recommendations made by national and international research funding agencies, few health and social science implementation studies include the TRL as part of their evaluation protocol. The TRL-IS offers a high degree of conceptual clarity between scientific maturity phases or readiness levels, and good reliability among raters of varying experience. This study highlights that adoption of the TRL-IS framework in implementation sciences will bolster the scientific robustness and comparability of research maturity in this domain.

The EMPOWER Occupational e-Mental Health Intervention Implementation Checklist to Foster e-Mental Health Interventions in the Workplace: Development Study.

BACKGROUND: Occupational e-mental health (OeMH) interventions significantly reduce the burden of mental health conditions. The successful implementation of OeMH interventions is influenced by many implementation strategies, barriers, and facilitators across contexts, which, however, are not systematically tracked. One of the reasons is that international consensus on documenting and reporting the implementation of OeMH interventions is lacking. There is a need for practical guidance on the key factors influencing the implementation of interventions that organizations should consider. Stakeholder consultations secure a valuable source of information about these key strategies, barriers, and facilitators that are relevant to successful implementation of OeMH interventions. OBJECTIVE: The objective of this study was to develop a brief checklist to guide the implementation of OeMH interventions. METHODS: Based on the results of a recently published systematic review, we drafted a comprehensive checklist with a wide set of strategies, barriers, and facilitators that were identified as relevant for the implementation of OeMH interventions. We then used a 2-stage stakeholder consultation process to refine the draft checklist to a brief and practical checklist comprising key implementation factors. In the first stage, stakeholders evaluated the relevance and feasibility of items on the draft checklist using a web-based survey. The list of items comprised 12 facilitators presented as statements addressing "elements that positively affect implementation" and 17 barriers presented as statements addressing "concerns toward implementation." If a strategy was deemed relevant, respondents were asked to rate it using a 4-point Likert scale ranging from "very difficult to implement" to "very easy to implement." In the second stage, stakeholders were interviewed to elaborate on the most relevant barriers and facilitators shortlisted from the first stage. The interview mostly focused on the relevance and priority of strategies and factors affecting OeMH intervention implementation. In the interview, the stakeholders' responses to the open survey's questions were further explored. The final checklist included strategies ranked as relevant and feasible and the most relevant facilitators and barriers, which were endorsed during either the survey or the interviews. RESULTS: In total, 26 stakeholders completed the web-based survey (response rate=24.8%) and 4 stakeholders participated in individual interviews. The OeMH intervention implementation checklist comprised 28 items, including 9 (32.1%) strategies, 8 (28.6%) barriers, and 11 (39.3%) facilitators. There was widespread agreement between findings from the survey and interviews, the most outstanding exception being the idea of proposing OeMH interventions as benefits for employees. CONCLUSIONS: Through our 2-stage stakeholder consultation, we developed a brief checklist that provides organizations with a guide for the implementation of OeMH interventions. Future research should empirically validate the effectiveness and usefulness of the checklist.

A Realist Evaluation of Case Management Models for People with Complex Health Conditions Using Novel Methods and Tools-What Works, for Whom, and under What Circumstances?

Case management developed from a generalist model to a person-centred model aligned with the evidence-informed evolution of best practice people-centred integrated care. Case management is a multidimensional and collaborative integrated care strategy where the case manager performs a set of interventions/actions to support the person with a complex health condition to progress in their recovery pathway and participate in life roles. It is currently unknown what case management model works in real life for whom and under what circumstances. The purpose of this study was to answer these questions. The study methods used realistic evaluation framework, examined the patterns and associations between case manager actions (mechanisms), the person's characteristics and environment (context), and recovery (outcomes) over 10 years post severe injury. There was mixed methods secondary analysis of data extracted via in-depth retrospective file reviews (n = 107). We used international frameworks and a novel approach with multi-layered analysis including machine learning and expert guidance for pattern identification. The study results confirm that when provided, a person-centred case management model contributes to and enhances the person's recovery and progress towards participation in life roles and maintaining well-being after severe injury.Furthermore, the intensity of case management for people with traumatic brain injury, and the person-centred actions of advising, emotional and motivational support, and proactive coordination contribute to the person achieving their goals. The results provide learnings for case management services on the case management models, for quality appraisal, service planning, and informs further research on case management.

Digital mental health and employment - Lessons from the Australian experience

Background and objectives: This paper reviews Australia's take-up of digital mental health interventions, including some specific reference to their application in relation to employment. Use of these interventions in Australia was already significant. Under COVID-19 they have exploded. The Australian experience offers useful lessons for European and other countries, and these are summarised.MethodsThis paper presents a narrative review of key texts, resources, policies and reports, from government and other sources. It also presents data reflecting the take-up of digital mental health services and the employment of people with a mental illness in Australia.ResultsWe present data showing the explosion in uptake of digital mental health services. There is very limited evidence about the impact of these services in improving employment outcomes for people with a mental illness in Australia. The Government has moved to make them a permanent feature of mental health care, in addition to traditional face-to-face care.ConclusionKey lessons emerge from the Australian experience, including the need for target clarity; the importance of blending digital services into broader frameworks of mental health care; the need for quality and safety standards to be developed and applied to digital services; and the need for better evaluation of the outcomes of digital interventions in the workplace.The digital mental health genie is out of the bottle. New capacity for evaluation of the outcomes of digital mental health services is vital to ensure value and quality of such investments. (AU)

What Does It Take to Get Somebody Back to Work after Severe Acquired Brain Injury? Service Actions within the Vocational Intervention Program (VIP 2.0).

Little is known about service actions delivered in the complex intervention of vocational rehabilitation (VR) for people with severe acquired brain injury (ABI). Scale-up of the Vocational Intervention Program (VIP) across the 12 Community teams of the NSW Brain Injury Rehabilitation Program provided an opportunity to analyse the intensity and profile of actions delivered in providing VR programs. Seventy-two participants with severe TBI were supported in returning to either pre-injury employment (FastTrack, FT, n = 27) or new employment (NewTrack, NT, n = 50), delivered by two types of VR providers (Disability Employment Service DES; private providers). VR providers documented their service actions in hours and minutes, using the Case Management Taxonomy, adapted to VR. The NT pathway required significantly higher levels of intervention in comparison to FT (25 h, five minutes vs. 35 h, 30 min, p = 0.048, W = 446). Case coordination was the most frequent service action overall (41.7% of total time for FT, 42.3% for NT). DES providers recorded significantly greater amounts of time undertaking engagement, assessment and planning, and emotional/motivational support actions compared to private providers. Overall duration of the programs were a median of 46 weeks (NT) and 36 weeks (FT), respectively. This study helps illuminate the profile of VR interventions for people with severe TBI.

Strategies for Implementing Occupational eMental Health Interventions: Scoping Review.

BACKGROUND: The implementation of eMental health interventions, especially in the workplace, is a complex process. Therefore, learning from existing implementation strategies is imperative to ensure improvements in the adoption, development, and scalability of occupational eMental health (OeMH) interventions. However, the implementation strategies used for these interventions are often undocumented or inadequately reported and have not been systematically gathered across implementations in a way that can serve as a much-needed guide for researchers. OBJECTIVE: The objective of this scoping review was to identify implementation strategies relevant to the uptake of OeMH interventions that target employees and detail the associated barriers and facilitation measures. METHODS: A scoping review was conducted. The descriptive synthesis was guided by the RE-AIM (reach, effectiveness, adoption, implementation, and maintenance) framework and the Consolidated Framework for Implementation Research. RESULTS: A total of 31 of 32,916 (0.09%) publications reporting the use of the web-, smartphone-, telephone-, and email-based OeMH interventions were included. In all, 98 implementation strategies, 114 barriers, and 131 facilitators were identified. The synthesis of barriers and facilitators produced 19 facilitation measures that provide initial recommendations for improving the implementation of OeMH interventions. CONCLUSIONS: This scoping review represents one of the first steps in a research agenda aimed at improving the implementation of OeMH interventions by systematically selecting, shaping, evaluating, and reporting implementation strategies. There is a dire need for improved reporting of implementation strategies and combining common implementation frameworks with more technology-centric implementation frameworks to fully capture the complexities of eHealth implementation. Future research should investigate a wider range of common implementation outcomes for OeMH interventions that also focus on a wider set of common mental health problems in the workplace. This scoping review's findings can be critically leveraged by discerning decision-makers to improve the reach, effectiveness, adoption, implementation, and maintenance of OeMH interventions.

Impact of Ed-LinQ: A Public Policy Strategy to Facilitate Engagement between Schools and the Mental Health Care System in Queensland, Australia.

Ed-LinQ is a mental health policy initiative to enhance the early detection and treatment of children with mental illness by improving the liaison between schools and health services in Queensland, Australia. We measured its impact from policy to practice to inform further program developments and public strategies. We followed a mixed quantitative/qualitative approach. The Adoption Impact Ladder (AIL) was used to analyse the adoption of this initiative by end-users (decision makers both in the health and education sectors) and the penetration of the initiative in the school sector. Survey respondents included representatives of schools (n = 186) and mental health providers (n = 78). In total, 63% of the school representative respondents were at least aware of the existence of the Ed-LinQ initiative, 74% were satisfied with the initiative and 28% of the respondent schools adopted the initiative to a significant extent. Adoption was higher in urban districts and in the health sector. The overall level of penetration in the school sector of Queensland was low (3%). The qualitative analysis indicated an improvement in the referral and communication processes between schools and the health sectors and the importance of funding in the implementation of the initiative. Mapping of existing programs is needed to assess the implementation of a new one as well as the design of different implementation strategies for urban and rural areas. Assessing the adoption of health policy strategies and their penetration in a target audience is critical to understand their proportional impacts across a defined ecosystem and constitutes a necessary preliminary step for the evaluation of their quality and efficiency.

Vagueness and Ambiguity in Communication of Case Management: A Content Analysis in the Australian National Disability Insurance Scheme.

Case management (CM) is an integrated care strategy, characterised by a set of actions to support person-centred planning, coordination of health and social services. Decades of CM, organisational psychology and occupational research highlight how vagueness and ambiguity in role communication can create role conflict and job stress, negatively impacts staff turnover, intra-organisational collaboration, job performance, and that poor communication of CM impedes policy, quality analysis service development and practice. We conducted a detailed top-down hierarchical, quality analysis of communication about CM roles and responsibilities in a Scheme for people with disability in Australia. The study used content analysis methods and the main actions as defined in a validated CM taxonomy (Appendix 1). We systematically searched and analysed 53 Scheme policy and practice documents of CM from 2013-2019. The results showed poor role communication with vagueness, ambiguity, gaps in the description of CM roles and responsibilities. Poor role communication has contributed to negative experiences and outcomes of CM actions of planning and coordination, as reported by CM users in many Scheme-related parliamentary inquiries, research, formal complaints, and decision appeals. The results reinforce the importance of an ontological approach in communication of CM roles and actions and provides learnings for integrated care roles across countries and contexts.

Disability-Related Questions for Administrative Datasets.

High rates of unemployment among people with disability are long-standing and persistent problems worldwide. For public policy, estimates of prevalence and population profiles are required for designing support schemes such as Australia's National Disability Insurance Scheme; for monitoring implementation of the United Nations Convention on Rights of Persons with Disabilities; and for monitoring service access, participation, and equity for people with disability in mainstream systems including employment. In the public sector, creating a succinct identifier for disability in administrative systems is a key challenge for public policy design and monitoring. This requires concise methods of identifying people with disability within systems, producing data comparable with population data to gauge accessibility and equity. We aimed to create disability-related questions of value to the purposes of an Australian state and contribute to literature on parsimonious and respectful disability identification for wider application. The research, completed in 2017, involved mapping and identification of key disability concepts for inclusion in new questions, focus groups to refine wording of new questions, and online surveys of employees evaluating two potential new question sets on the topic of disability and environment. Recommendations for new disability-related questions and possible new data collection processes are being considered and used by the leading state authority.

A new framing approach in guideline development to manage different sources of knowledge.

RATIONALE: Contemporary guideline methodology struggles to consider context and information from different sources of knowledge besides quantitative research. Return to work programmes involve multiple components and stakeholders. If the guideline is to be relevant and practical for a complex intervention such as return to work, it is essential to use broad sources of knowledge. AIMS AND OBJECTIVE: This paper reports on a new method in guideline development to manage different sources of knowledge. METHODS: The method used framing for the return-to-work guidance within the Clinical Practice Guidelines for the Management of Rotator Cuff Syndrome in the Workplace. The development involved was a multi-disciplinary working party of experts including consumers. The researchers considered a broad range of research, expert (practice and experience) knowledge, the individual's and workplace contexts, and used framing with the International Classification of Functioning, Disability and Health. Following a systematic database search on four clinical questions, there were seven stages of knowledge management to extract, unpack, map and pack information to the ICF domains framework. Companion graded recommendations were developed. RESULTS: The results include practical examples, user and consumer guides, flow charts and six graded or consensus recommendations on best practice for return to work intervention. CONCLUSIONS: Our findings suggest using framing in guideline methodology with internationally accepted frames such as the ICF is a reliable and transparent framework to manage different sources of knowledge. Future research might examine other examples and methods for managing complexity and using different sources of knowledge in guideline development.

What Is Case Management? A Scoping and Mapping Review.

The description of case management in research and clinical practice is highly variable which impedes quality analysis, policy and planning. Case management makes a unique contribution towards the integration of health care, social services and other sector services and supports for people with complex health conditions. There are multiple components and variations of case management depending on the context and client population. This paper aims to scope and map case management in the literature to identify how case management is described in the literature for key complex health conditions (e.g., brain injury, diabetes, mental health, spinal cord injury). Following literature searches in multiple databases, grey literature and exclusion by health condition, community-based and adequate description, there were 661 potential papers for data extraction. Data from 79 papers (1988-2013) were analysed to the point of saturation (no new information) and mapped to the model, components and activities. The results included 22 definitions, five models, with 69 activities or tasks of case managers mapped to 17 key components (interventions). The results confirm the significant terminological variance in case management which produces role confusion, ambiguity and hinders comparability across different health conditions and contexts. There is an urgent need for an internationally agreed taxonomy for the coordination, navigation and management of care.

The brain injury case management taxonomy (BICM-T); a classification of community-based case management interventions for a common language.

BACKGROUND: Case management is a complex intervention. Complexity arises from the interaction of different components: the model (theoretical basis), implementation context (service), population and health condition, focus for the intervention (client and/or their family), case manager's actions (interventions) and the target of case management (integrated care and support, client's community participation). There is a lack of understanding and a common language. To our knowledge there is no classification (taxonomy) for community-based case management. OBJECTIVE: To develop a community-based case management in brain injury taxonomy (BICM-T), as a common language and understanding of case management for use in quality analysis, policy, planning and practice. METHODS: The mixed qualitative methods used multiple sources of knowledge including scoping, framing and a nominal group technique to iteratively develop the Beta version (draft) of the taxonomy. A two part developmental evaluation involving case studies and mapping to international frameworks assessed the applicability and acceptability (feasibility) before finalization of the BICM-T. RESULTS: The BICM-T includes a definition of community-based case management, taxonomy trees, tables and a glossary. The interventions domain tree has 9 main actions (parent category): engagement, holistic assessment, planning, education, training and skills development, emotional and motivational support, advising, coordination, monitoring; 17 linked actions (children category); 8 related actions; 63 relevant terms defined in the glossary. CONCLUSIONS: The BICM-T provides a knowledge map with the definitions and relationships between the core actions (interventions domain). Use of the taxonomy as a common language will benefit practice, quality analysis, evaluation, policy, planning and resource allocation.

Evidence-based medicine: is it a bridge too far?

AIMS: This paper aims to describe the contextual factors that gave rise to evidence-based medicine (EBM), as well as its controversies and limitations in the current health context. Our analysis utilizes two frameworks: (1) a complex adaptive view of health that sees both health and healthcare as non-linear phenomena emerging from their different components; and (2) the unified approach to the philosophy of science that provides a new background for understanding the differences between the phases of discovery, corroboration, and implementation in science. RESULTS: The need for standardization, the development of clinical epidemiology, concerns about the economic sustainability of health systems and increasing numbers of clinical trials, together with the increase in the computer's ability to handle large amounts of data, have paved the way for the development of the EBM movement. It was quickly adopted on the basis of authoritative knowledge rather than evidence of its own capacity to improve the efficiency and equity of health systems. The main problem with the EBM approach is the restricted and simplistic approach to scientific knowledge, which prioritizes internal validity as the major quality of the studies to be included in clinical guidelines. As a corollary, the preferred method for generating evidence is the explanatory randomized controlled trial. This method can be useful in the phase of discovery but is inadequate in the field of implementation, which needs to incorporate additional information including expert knowledge, patients' values and the context. CONCLUSION: EBM needs to move forward and perceive health and healthcare as a complex interaction, i.e. an interconnected, non-linear phenomenon that may be better analysed using a variety of complexity science techniques.

In search of an integrative measure of functioning.

International trends towards people-centred, integrative care and support require any measurement of functioning and disability to meet multiple aims. The information requirements of two major Australian programs for disability and rehabilitation are outlined, and the findings of two searches for suitable measures of functioning and disability are analysed. Over 30 current measures of functioning were evaluated in each search. Neither search found a generic measure of functioning suitable for these multibillion dollar programs, relevant to a wide range of people with a variety of health conditions and functioning experiences, and capable of indicating support needs, associated costs, progress and outcomes. This unsuccessful outcome has implications internationally for policy-relevant information for disability, rehabilitation and related programs. The paper outlines the features of an Integrative Measure of Functioning (IMF) based on the concepts of functioning and environmental factors in the International Classification of Functioning, Disability and Health (ICF). An IMF would be applicable across a variety of health conditions, settings and purposes, ranging from individual assessment to public health. An IMF could deliver person-centred, policy-relevant information for a range of programs, promoting harmonised language and measurement and supporting international trends in human services and public health.

Supporting a person-centred approach in clinical guidelines. A position paper of the Allied Health Community - Guidelines International Network (G-I-N).

BACKGROUND: A person-centred approach in the context of health services delivery implies a biopsychosocial model focusing on all factors that influence the person's health and functioning. Those wishing to monitor change should consider this perspective when they develop and use guidelines to stimulate active consideration of the person's needs, preferences and participation in goal setting, intervention selection and the use of appropriate outcome measures. OBJECTIVE: To develop a position paper that promotes a person-centred approach in guideline development and implementation. DESIGN, SETTING AND PARTICIPANTS: We used three narrative discussion formats to collect data for achieving consensus: a nominal group technique for the Allied Health Steering Group, an Internet discussion board and a workshop at the annual G-I-N conference. We analysed the data for relevant themes to draft recommendations. RESULTS: We built the position paper on the values of the biopsychosocial model. Four key themes for enhancing a person-centred approach in clinical guidelines emerged: (i) use a joint definition of health-related quality of life as an essential component of intervention goals, (ii) incorporate the International Classification of Functioning, Disability and Health (ICF) as a framework for considering all domains related to health, (iii) adopt a shared decision-making method, and (iv) incorporate patient-reported health outcome measures. The position statement includes 14 recommendations for guideline developers, implementers and users. CONCLUSION: This position paper describes essential elements for incorporating a person-centred approach in clinical guidelines. The consensus process provided information about barriers and facilitators that might help us develop strategies for implementing person-centred care.

Framing of scientific knowledge as a new category of health care research.

RATIONALE: The new area of health system research requires a revision of the taxonomy of scientific knowledge that may facilitate a better understanding and representation of complex health phenomena in research discovery, corroboration and implementation. METHOD: A position paper by an expert group following and iterative approach. RESULTS: 'Scientific evidence' should be differentiated from 'elicited knowledge' of experts and users, and this latter typology should be described beyond the traditional qualitative framework. Within this context 'framing of scientific knowledge' (FSK) is defined as a group of studies of prior expert knowledge specifically aimed at generating formal scientific frames. To be distinguished from other unstructured frames, FSK must be explicit, standardized, based on the available evidence, agreed by a group of experts and subdued to the principles of commensurability, transparency for corroboration and transferability that characterize scientific research. A preliminary typology of scientific framing studies is presented. This typology includes, among others, health declarations, position papers, expert-based clinical guides, conceptual maps, classifications, expert-driven health atlases and expert-driven studies of costs and burden of illness. CONCLUSIONS: This grouping of expert-based studies constitutes a different kind of scientific knowledge and should be clearly differentiated from 'evidence' gathered from experimental and observational studies in health system research.