Adaptation of the technology readiness levels for impact assessment in implementation sciences: The TRL-IS checklist.

Background: Intervention development is a critical process in implementation research. There are key stages involved in the process to design, pilot, demonstrate and release a technology or an intervention. The Technology Readiness Level (TRL) is a globally accepted instrument for assessing the maturity of research development. However, the original levels do not fit all, and some adjustments are required for its applicability in implementation sciences. Aims: This study aimed to gather the prior knowledge base on TRL in public and population health research; to develop a standard definition of readiness, and to adapt and validate the TRL to an implementation science context (TRL-IS). Materials and methods: A Mixed methods approach has been followed in this study. A scoping review using the PRISMA extension (PRISMA-ScR) informed a nominal expert panel for developing a standard definition of readiness and to modify the TRL following an ontoterminology approach. Then the maturity of six practical case study examples were rated by ten researchers using the modified TRL to estimate inter-rater reliability, and a group of experts provided final content and face validity and feasibility.This mixed methods study included 1) a scoping review to examine the current literature and develop a knowledge base, identify knowledge gaps and to clarify concepts; 2) the development of a standard definition of 'Readiness' and related terms; and 3) adaptation of the TRL to implementation science and development of a checklist to rate the maturity of applications.A standard definition of readiness and related terms was produced by the core team, and an international nominal group (n = 30) was conducted to discuss and validate the definition and terms, and the location of 'Readiness' in the initiation and early development phases of implementation.Following feedback from the nominal group, the development of the TRL-IS was finalised and a TRL-IS rating checklist was developed to rate the maturity of applications. The TRL-IS checklist was tested using six cases based on real world studies on implementation research.The inter-rater reliability of the TRL-IS was evaluated by ten raters and finally six raters evaluated the content and face validity, and feasibility, of the TRL-IS checklist using the System Usability Scale (SUS). Results: Few papers (n = 11) utilised the TRL to evaluate the readiness of readiness of health and social science implementation research. The main changes in the adaption of the TRL-IS included the removal of laboratory testing, limiting the use of "operational" environment and a clearer distinction between level 6 (pilot in a relevant environment) and 7 (demonstration in the real world prior to release). The adapted version was considered relevant by the expert panel. The TRL-IS checklist showed evidence of good inter-rater reliability (ICC = 0.90 with 95 % confident interval = 0.74-0.98, p < .001) and provides a consistent metric. Conclusions: In spite of recommendations made by national and international research funding agencies, few health and social science implementation studies include the TRL as part of their evaluation protocol. The TRL-IS offers a high degree of conceptual clarity between scientific maturity phases or readiness levels, and good reliability among raters of varying experience. This study highlights that adoption of the TRL-IS framework in implementation sciences will bolster the scientific robustness and comparability of research maturity in this domain.

Evaluating Aboriginal and Torres Strait Islander Social and Emotional Wellbeing services: A collective case study in Far North Queensland.

BACKGROUND: Access to a coordinated range of strengths-based, culturally appropriate community-led primary mental health and Social and Emotional Wellbeing services is critical to the mental health and wellbeing of young Aboriginal and Torres Strait Islander people, and is a policy commitment of the Australian government. However, complex and fragmented service networks and a lack of standardised service data are barriers in identifying what services are available and what care they provide. METHOD: A standardised service classification tool was used to assess the availability and characteristics of Social and Emotional Wellbeing services for young Aboriginal and Torres Strait Islander people in two regions in Queensland, Australia. RESULTS: We identified a complex pattern of service availability and gaps in service provision. Non-Indigenous non-governmental organisations provided a significant proportion of services, particularly 'upstream' support, while Aboriginal Community Controlled Organisations were more likely to provide 'downstream' crisis type care. Most services provided by the public sector were through Child Safety and Youth Justice departments. CONCLUSIONS: Our findings demonstrate the complexity of current networks, and show that non-Indigenous organisations are disproportionately influential in the care received by young Aboriginal and Torres Strait Islander people, despite community goals of self-determination, and government commitment to increasing capacity of Aboriginal Community Controlled Organisations to support their local communities. These findings can be used to support decision making and planning.

The EMPOWER Occupational e-Mental Health Intervention Implementation Checklist to Foster e-Mental Health Interventions in the Workplace: Development Study.

BACKGROUND: Occupational e-mental health (OeMH) interventions significantly reduce the burden of mental health conditions. The successful implementation of OeMH interventions is influenced by many implementation strategies, barriers, and facilitators across contexts, which, however, are not systematically tracked. One of the reasons is that international consensus on documenting and reporting the implementation of OeMH interventions is lacking. There is a need for practical guidance on the key factors influencing the implementation of interventions that organizations should consider. Stakeholder consultations secure a valuable source of information about these key strategies, barriers, and facilitators that are relevant to successful implementation of OeMH interventions. OBJECTIVE: The objective of this study was to develop a brief checklist to guide the implementation of OeMH interventions. METHODS: Based on the results of a recently published systematic review, we drafted a comprehensive checklist with a wide set of strategies, barriers, and facilitators that were identified as relevant for the implementation of OeMH interventions. We then used a 2-stage stakeholder consultation process to refine the draft checklist to a brief and practical checklist comprising key implementation factors. In the first stage, stakeholders evaluated the relevance and feasibility of items on the draft checklist using a web-based survey. The list of items comprised 12 facilitators presented as statements addressing "elements that positively affect implementation" and 17 barriers presented as statements addressing "concerns toward implementation." If a strategy was deemed relevant, respondents were asked to rate it using a 4-point Likert scale ranging from "very difficult to implement" to "very easy to implement." In the second stage, stakeholders were interviewed to elaborate on the most relevant barriers and facilitators shortlisted from the first stage. The interview mostly focused on the relevance and priority of strategies and factors affecting OeMH intervention implementation. In the interview, the stakeholders' responses to the open survey's questions were further explored. The final checklist included strategies ranked as relevant and feasible and the most relevant facilitators and barriers, which were endorsed during either the survey or the interviews. RESULTS: In total, 26 stakeholders completed the web-based survey (response rate=24.8%) and 4 stakeholders participated in individual interviews. The OeMH intervention implementation checklist comprised 28 items, including 9 (32.1%) strategies, 8 (28.6%) barriers, and 11 (39.3%) facilitators. There was widespread agreement between findings from the survey and interviews, the most outstanding exception being the idea of proposing OeMH interventions as benefits for employees. CONCLUSIONS: Through our 2-stage stakeholder consultation, we developed a brief checklist that provides organizations with a guide for the implementation of OeMH interventions. Future research should empirically validate the effectiveness and usefulness of the checklist.

The Integrated Atlas of Dementia Care in the Australian Capital Territory: A Collective Case Study of Local Service Provision.

Background: This study evaluates the dementia care system in a local area and aimed to include all specialised services designed to provide health and social services to people with dementia or age-related cognitive impairment, as well as general services with a high or very high proportion of clients with dementia. Methods: The study used an internationally standardised service classification instrument called Description and Evaluation of Services and DirectoriEs for Long Term Care (DESDE-LTC) to identify and describe all services providing care to people with dementia in the Australian Capital Territory (ACT). Results: A total of 47 service providers were eligible for inclusion. Basic information about the services was collected from their websites, and further information was obtained through interviews with the service providers. Of the 107 services offered by the 47 eligible providers, 27% (n = 29) were specialised services and 73% (n = 78) were general services. Most of the services were residential or outpatient, with a target population mostly of people aged 65 or older, and 50 years or older in the case of Aboriginal and Torres Strait Islander Australians. There were government supports available for most types of care through various programmes. Conclusions: Dementia care in the ACT relies heavily on general services. More widespread use of standardised methods of service classification in dementia will facilitate comparison with other local areas, allow for monitoring of changes over time, permit comparison with services provided for other health conditions and support evidence-informed local planning.

Impact of the workforce allocation on the technical performance of mental health services: the collective case of Helsinki-Uusimaa (Finland).

BACKGROUND: Long-term mental health (MH) policies in Finland aimed at investing in community care and promoting reforms have led to a reduction in the number of psychiatric hospital beds. However, most resources are still allocated to hospital and community residential services due to various social, economic and political factors. Despite previous research focussing on the number and cost of these services, no study has evaluated the emerging patterns of use, their technical performance and the relationship with the workforce structure. OBJECTIVE: The purpose of this study was to observe the patterns of use and their technical performance (efficiency) of the main types of care of MH services in the Helsinki-Uusimaa region (Finland), and to analyse the potential relationship between technical performance and the corresponding workforce structure. METHODS: The sample included acute hospital residential care, non-hospital residential care and outpatient care services. The analysis was conducted using regression analysis, Monte Carlo simulation, fuzzy inference and data envelopment analysis. RESULTS: The analysis showed a statistically significant linear relationship between the number of service users and the length of stay, number of beds in non-hospital residential care and number of contacts in outpatient care services. The three service types displayed a similar pattern of technical performance, with high relative technical efficiency on average and a low probability of being efficient. The most efficient acute hospital and outpatient care services integrated multidisciplinary teams, while psychiatrists and nurses characterized non-hospital residential care. CONCLUSIONS: The results indicated that the number of resources and utilization variables were linearly related to the number of users and that the relative technical efficiency of the services was similar across all types. This suggests homogenous MH management with small variations based on workforce allocation. Therefore, the distribution of workforce capacity should be considered in the development of effective policies and interventions in the southern Finnish MH system.

The identification of economically relevant health and social care services for mental disorders in the PECUNIA project.

BACKGROUND: Health economic research is still facing significant problems regarding the standardization and international comparability of health care services. As a result, comparative effectiveness studies and cost-effectiveness analyses are often not comparable. This study is part of the PECUNIA project, which aimed to improve the comparability of economic evaluations by developing instruments for the internationally standardized measurement and valuation of health care services for mental disorders. The aim of this study was to identify internationally relevant services in the health and social care sectors relevant for health economic studies for mental disorders. METHODS: A systematic literature review on cost-of-illness studies and economic evaluations was conducted to identify relevant services, complemented by an additional grey literature search and a search of resource use measurement (RUM) questionnaires. A preliminary long-list of identified services was explored and reduced to a short-list by multiple consolidation rounds within the international research team and an external international expert survey in six European countries. RESULTS: After duplicate removal, the systematic search yielded 15,218 hits. From these 295 potential services could be identified. The grey literature search led to 368 and the RUM search to 36 additional potential services. The consolidation process resulted in a preliminary list of 186 health and social care services which underwent an external expert survey. A final consolidation step led to a basic list of 56 services grouped into residential care, daycare, outpatient care, information for care, accessibility to care, and self-help and voluntary care. CONCLUSIONS: The initial literature searches led to an extensive number of potential service items for health and social care. Many of these items turned out to be procedures, interventions or providing professionals rather than services and were removed from further analysis. The resulting list was used as a basis for typological coding, the development of RUM questionnaires and corresponding unit costs for international mental health economic studies in the PECUNIA project.

Mental health systems modelling for evidence-informed service reform in Australia.

Australia's Fifth National Mental Health Plan required governments to report, not only on the progress of changes to mental health service delivery, but to also plan for services that should be provided. Future population demand for treatment and care is challenging to predict and one solution involves modelling the uncertain demands on the system. Modelling can help decision-makers understand likely future changes in mental health service demand and more intelligently choose appropriate responses. It can also support greater scrutiny, accountability and transparency of these processes. Australia has an emerging national capacity for systems modelling in mental health which can enhance the next phase of mental health reform. This paper introduces concepts useful for understanding mental health modelling and identifies where modelling approaches can support health service planners to make evidence-informed decisions regarding planning and investment for the Australian population.

Standardized Systematic Description of Provision of Care for Multiple Sclerosis at a Local Level: A Demonstration Study.

BACKGROUND: This study evaluates and describes the pattern of services provided for people living with multiple sclerosis (MS) in a local area as a starting point for a more global assessment. METHODS: A health care ecosystem approach has been followed using an internationally standardized service classification instrument-the Description and Evaluation of Services and DirectoriEs for Long Term Care (DESDE-LTC)-to identify and describe all services providing care to people with MS in the Australian Capital Territory, Australia. Available services were classified according to the target population into those specifically dedicated to people living with MS and those providing general neurologic services, both public and private, and across both social and health sectors. RESULTS: A limited range of services was available. There were no local facilities providing or coordinating multidisciplinary integrated care specific to people with MS. Subspecialty services specific to MS were limited in number (6 of the 28 services), and use of specialist services provided in neighboring states was frequently reported. Overall, very few services were provided outside the core health sector (4%). CONCLUSIONS: The provision of care to people living with MS in the Australian Capital Territory is fragmented and relies heavily on generic neurology services in the public and private sectors. More widespread use of the DESDE-LTC as a standardized method of service classification in MS will facilitate comparison with other local areas, allow monitoring of changes over time, and permit comparison with services provided for other health conditions (eg, dementia, mental disorders).

Concepts of good mental health and wellbeing in people with intellectual disability: Study protocol for a systematic review.

Introduction: Good mental health is an indispensable aspect of good general health and different definitions of good mental health have been developed for the general population. However, it is not clear how these definitions can be applied to people with intellectual disabilities (ID), as they may have difficulties in many facets encompassed in existing definitions. Yet, people with ID can be mentally healthy or mentally ill just as people without ID. Objective: The aim of this systematic review is to summarize existing concepts, definitions, and measurement approaches of good mental health and wellbeing for people with ID. Methods: A comprehensive, systematic literature review will be conducted in 11 databases, including ASSIA, CINAHL, Cochrane Library, ERIC, MedRxiv, OSF preprints, ProQuest Dissertations & Theses Global, PsycINFO, PubMed, Scopus, and Web of Science. By including preprints and theses servers in the search strategy, we will also consider grey and unpublished literature. The quality of included studies will be rated using standardized checklists. All steps of the literature search, data extraction, and quality rating will be performed independently by two trained researchers. Disagreements will be resolved through discussion between these researchers and, if required, by consulting a third team member. In a narrative synthesis, existing approaches to good mental health and wellbeing for people with ID will be systematically described and linked to current research in mental health for people with and without ID. Discussion: The findings of this study will provide researchers and practitioners with an evidence-based overview of current approaches to good mental health and wellbeing of people with ID. We will explore and discuss the individual facets of the definitions, concepts, and measurement approaches and identify possible gaps which need to be addressed. This will strengthen future research on this topic and focus research activities on important and unresolved themes that need to be targeted to promote health equity for people with ID.

Patterns of mental health care provision in urban areas: A comparative analysis for local policy in the ACT.

Urbanisation presents specific mental health challenges, requiring a better understanding of service availability in urban areas for mental health care planning. Our objective is to analyse patterns of urban mental healthcare provision in Australia, and compare these with relevant national and international regions to inform urban mental healthcare policy and planning. Following a health ecosystems approach, we use a standardised service classification instrument, the Description and Evaluation of Services and DirectoriEs (DESDE), and Mental Health Care Atlases, to compare the availability, bed capacity and diversity of services providing specialised mental health or psychosocial care that are universally accessible (ie provided at no or low cost only in all relevant care sectors in four Australian and three international urban regions. We used a heuristics approach and an homogeneity test. Applicability to local policy was assessed using the Adoption Impact Ladder. Community care was less developed in Australia than internationally, except in the case of residential care in Australian Capital Territory, our reference area. Alternatives to hospitalisation were scarce across all regions. The Atlas was applicable to regional and local mental health planning. Differences in pattern of care between regions has implications for planning, equality of access to care and prioritisation of resources. An ecosystems approach is relevant to service planning in mental healthcare at local level.

A Realist Evaluation of Case Management Models for People with Complex Health Conditions Using Novel Methods and Tools-What Works, for Whom, and under What Circumstances?

Case management developed from a generalist model to a person-centred model aligned with the evidence-informed evolution of best practice people-centred integrated care. Case management is a multidimensional and collaborative integrated care strategy where the case manager performs a set of interventions/actions to support the person with a complex health condition to progress in their recovery pathway and participate in life roles. It is currently unknown what case management model works in real life for whom and under what circumstances. The purpose of this study was to answer these questions. The study methods used realistic evaluation framework, examined the patterns and associations between case manager actions (mechanisms), the person's characteristics and environment (context), and recovery (outcomes) over 10 years post severe injury. There was mixed methods secondary analysis of data extracted via in-depth retrospective file reviews (n = 107). We used international frameworks and a novel approach with multi-layered analysis including machine learning and expert guidance for pattern identification. The study results confirm that when provided, a person-centred case management model contributes to and enhances the person's recovery and progress towards participation in life roles and maintaining well-being after severe injury.Furthermore, the intensity of case management for people with traumatic brain injury, and the person-centred actions of advising, emotional and motivational support, and proactive coordination contribute to the person achieving their goals. The results provide learnings for case management services on the case management models, for quality appraisal, service planning, and informs further research on case management.

Patterns of mental healthcare provision in rural areas: A demonstration study in Australia and Europe.

Introduction: Mental healthcare systems are primarily designed to urban populations. However, the specific characteristics of rural areas require specific strategies, resource allocation, and indicators which fit their local conditions. This planning process requires comparison with other rural areas. This demonstration study aimed to describe and compare specialized rural adult mental health services in Australia, Norway, and Spain; and to demonstrate the readiness of the healthcare ecosystem approach and the DESDE-LTC mapping tool (Description and Evaluation of Services and Directories of Long Term Care) for comparing rural care between countries and across areas. Methods: The study described and classified the services using the DESDE-LTC. The analyses included context analysis, care availability, placement capacity, balance of care, and diversity of care. Additionally, readiness (Technology Readiness Levels - TRL) and impact analyses (Adoption Impact Ladder - AIL) were also assessed by two independent raters. Results: The findings demonstrated the usability of the healthcare ecosystem approach and the DESDE-LTC to map and identify differences and similarities in the pattern of care of highly divergent rural areas. Day care had a greater weight in the European pattern of care, while it was replaced by social outpatient care in Australian areas. In contrast, care coordination was more common in Australia, pointing to a more fragmented system that requires navigation services. The share between hospital and community residential care showed no differences between the two regions, but there were differences between catchment areas. The healthcare ecosystem approach showed a TRL 8 (the tool has been demonstrated in a real-world environment and it is ready for release and general use) and an AIL of 5 (the target public agencies provided resources for its completion). Two experts evaluated the readiness of the use of DESDE-LTC in their respective regional studies. All of them were classified using the TRL. Discussion: In conclusion, this study strongly supports gathering data on the provision of care in rural areas using standardized methods to inform rural service planning. It provides information on context and service availability, capacity and balance of care that may improve, directly or through subsequent analyses, the management and planning of services in rural areas.

Development of an Instrument for the Assessment of Health-Related Multi-sectoral Resource Use in Europe: The PECUNIA RUM.

BACKGROUND: Measuring objective resource-use quantities is important for generating valid cost estimates in economic evaluations. In the absence of acknowledged guidelines, measurement methods are often chosen based on practicality rather than methodological evidence. Furthermore, few resource-use measurement (RUM) instruments focus on the measurement of resource use in multiple societal sectors and their development process is rarely described. Thorn and colleagues proposed a stepwise approach to the development of RUM instruments, which has been used for developing cost questionnaires for specific trials. However, it remains unclear how this approach can be translated into practice and whether it is applicable to the development of generic self-reported RUM instruments and instruments measuring resource use in multiple sectors. This study provides a detailed description of the practical application of this stepwise approach to the development of a multi-sectoral RUM instrument developed within the ProgrammE in Costing, resource use measurement and outcome valuation for Use in multi-sectoral National and International health economic evaluAtions (PECUNIA) project. METHODS: For the development of the PECUNIA RUM, the methodological approach was based on best practice guidelines. The process included six steps, including the definition of the instrument attributes, identification of cost-driving elements in each sector, review of methodological literature and development of a harmonized cross-sectorial approach, development of questionnaire modules and their subsequent harmonization. RESULTS: The selected development approach was, overall, applicable to the development of the PECUNIA RUM. However, due to the complexity of the development of a multi-sectoral RUM instrument, additional steps such as establishing a uniform methodological basis, harmonization of questionnaire modules and involvement of a broader range of stakeholders (healthcare professionals, sector-specific experts, health economists) were needed. CONCLUSION: This is the first study that transparently describes the development process of a generic multi-sectoral RUM instrument in health economics and provides insights into the methodological aspects and overall validity of its development process.

Digital mental health and employment - Lessons from the Australian experience

Background and objectives: This paper reviews Australia's take-up of digital mental health interventions, including some specific reference to their application in relation to employment. Use of these interventions in Australia was already significant. Under COVID-19 they have exploded. The Australian experience offers useful lessons for European and other countries, and these are summarised.MethodsThis paper presents a narrative review of key texts, resources, policies and reports, from government and other sources. It also presents data reflecting the take-up of digital mental health services and the employment of people with a mental illness in Australia.ResultsWe present data showing the explosion in uptake of digital mental health services. There is very limited evidence about the impact of these services in improving employment outcomes for people with a mental illness in Australia. The Government has moved to make them a permanent feature of mental health care, in addition to traditional face-to-face care.ConclusionKey lessons emerge from the Australian experience, including the need for target clarity; the importance of blending digital services into broader frameworks of mental health care; the need for quality and safety standards to be developed and applied to digital services; and the need for better evaluation of the outcomes of digital interventions in the workplace.The digital mental health genie is out of the bottle. New capacity for evaluation of the outcomes of digital mental health services is vital to ensure value and quality of such investments. (AU)