The Diabetes Evaluation Framework for Innovative National Evaluations (DEFINE): Construct and Content Validation Using a Modified Delphi Method.

OBJECTIVES: In order to scale-up successful innovations, more evidence is needed to evaluate programs that attempt to address the rising prevalence of diabetes and the associated burdens on patients and the healthcare system. This study aimed to assess the construct and content validity of the Diabetes Evaluation Framework for Innovative National Evaluations (DEFINE), a tool developed to guide the evaluation, design and implementation with built-in knowledge translation principles. METHODS: A modified Delphi method, including 3 individual rounds (questionnaire with 7-point agreement/importance Likert scales and/or open-ended questions) and 1 group round (open discussion) were conducted. Twelve experts in diabetes, research, knowledge translation, evaluation and policy from Canada (Ontario, Quebec and British Columbia) and Australia participated. Quantitative consensus criteria were an interquartile range of ≤1. Qualitative data were analyzed thematically and confirmed by participants. An importance scale was used to determine a priority multi-level indicator set. Items rated very or extremely important by 80% or more of the experts were reviewed in the final group round to build the final set. RESULTS: Participants reached consensus on the content and construct validity of DEFINE, including its title, overall goal, 5-step evaluation approach, medical and nonmedical determinants of health schematics, full list of indicators and associated measurement tools, priority multi-level indicator set and next steps in DEFINE's development. CONCLUSIONS: Validated by experts, DEFINE has the right theoretic components to evaluate comprehensively diabetes prevention and management programs and to support acquisition of evidence that could influence the knowledge translation of innovations to reduce the burden of diabetes.

The FORGE AHEAD clinical readiness consultation tool: a validated tool to assess clinical readiness for chronic disease care mobilization in Canada's First Nations.

BACKGROUND: Given the astounding rates of diabetes and related complications, and the barriers to providing care present in Indigenous communities in Canada, intervention strategies that take into account contextual factors such as readiness to mobilize are needed to maximize improvements and increase the likelihood of success and sustainment. As part of the national FORGE AHEAD Program, we sought to develop, test and validate a clinical readiness consultation tool aimed at assessing the readiness of clinical teams working on-reserve in First Nations communities to participate in quality improvement (QI) to enhance diabetes care in Canada. METHODS: A literature review was conducted to identify existing readiness tools. The ABCD - SAT was adapted using a consensus approach that emphasized a community-based participatory approach and prioritized the knowledge and wisdom held by community members. The tool was piloted with a group of 16 people from 7 provinces and 11 partnering communities to assess language use, clarity, relevance, format, and ease of completion using examples. Internal reliability analysis and convergence validity were conducted with data from 53 clinical team members from 11 First Nations communities (3-5 per community) who have participated in the FORGE AHEAD program. RESULTS: The 27-page Clinical Readiness Consultation Tool (CRCT) consists of five main components, 21 sub-components, and 74 items that are aligned with the Expanded Chronic Care Model. Five-point Likert scale feedback from the pilot ranged from 3.25 to 4.5. Length of the tool was reported as a drawback but respondents noted that all the items were needed to provide a comprehensive picture of the healthcare system. Results for internal consistency showed that all sub-components except for two were within acceptable ranges (0.77-0.93). The Team Structure and Function sub-component scale had a moderately significant positive correlation with the validated Team Climate Inventory, r = 0.45, p < 0.05. CONCLUSIONS: The testing and validation of the FORGE AHEAD CRCT demonstrated that the tool is acceptable, valid and reliable. The CRCT has been successfully used to support the implementation of the FORGE AHEAD Program and the health services changes that partnering First Nations communities have designed and undertaken to improve diabetes care. TRIAL REGISTRATION NUMBER: Current ClinicalTrial.gov protocol ID NCT02234973 . Date of Registration: July 30, 2014.

Impact of a primary healthcare quality improvement program on diabetes in Canada: evaluation of the Quality Improvement and Innovation Partnership (QIIP).

OBJECTIVE: Primary healthcare (PHC) quality improvement (QI) initiatives are designed to improve patient care and health outcomes. We evaluated the Quality Improvement and Innovation Partnership (QIIP), an Ontario-wide PHC QI program on access to care, diabetes management and colorectal cancer screening. This manuscript highlights the impact of QIIP on diabetes outcomes and associated vascular risk factors. RESEARCH DESIGN AND METHODS: A cluster matched-control, retrospective prechart and postchart audit was conducted. One physician per QIIP-PHC team (N=34) and control (N=34) were recruited for the audit. Eligible charts were reviewed for prespecified type 2 diabetes mellitus clinical process and outcome data at baseline, during (intervention range: 15-17.5 months) and post. Primary outcome measures were the A1c of patients above study target and proportion of patients with an annual foot exam. Secondary outcome measures included glycemic, hypertension and lipid outcomes and management, screening for diabetes-related complications, healthcare utilization, and diabetes counseling, education and self-management goal setting. RESULTS: More patients in the QIIP group achieved statistically improved lipid testing, eye examinations, peripheral neuropathy exams, and documented body mass index. No statistical differences in A1c, low-density lipoprotein or systolic/diastolic blood pressure values were noted, with no significant differences in medication prescription, specialist referrals, or chart-reported diabetes counseling, education or self-management goals. Patients of QIIP physicians had significantly more PHC visits. CONCLUSION: The QIIP-learning collaborative program evaluation using stratified random selection of participants and the inclusion of a control group makes this one of the most rigorous and promising efforts to date evaluating the impact of a QI program in PHC. The chart audit component of this evaluation highlighted that while QIIP improved some secondary diabetes measures, no improvements in clinical outcomes were noted. This study highlights the importance of formalized evaluation of QI initiatives to provide an evidence base to inform future program planning and scale-up.

Developing community-driven quality improvement initiatives to enhance chronic disease care in Indigenous communities in Canada: the FORGE AHEAD program protocol.

BACKGROUND: Given the dramatic rise and impact of chronic diseases and gaps in care in Indigenous peoples in Canada, a shift from the dominant episodic and responsive healthcare model most common in First Nations communities to one that places emphasis on proactive prevention and chronic disease management is urgently needed. METHODS: The Transformation of Indigenous Primary Healthcare Delivery (FORGE AHEAD) Program partners with 11 First Nations communities across six provinces in Canada to develop and evaluate community-driven quality improvement (QI) initiatives to enhance chronic disease care. FORGE AHEAD is a 5-year research program (2013-2017) that utilizes a pre-post mixed-methods observational design rooted in participatory research principles to work with communities in developing culturally relevant innovations and improved access to available services. This intensive program incorporates a series of 10 inter-related and progressive program activities designed to foster community-driven initiatives with type 2 diabetes mellitus as the action disease. Preparatory activities include a national community profile survey, best practice and policy literature review, and readiness tool development. Community-level intervention activities include community and clinical readiness consultations, development of a diabetes registry and surveillance system, and QI activities. With a focus on capacity building, all community-level activities are driven by trained community members who champion QI initiatives in their community. Program wrap-up activities include readiness tool validation, cost-analysis and process evaluation. In collaboration with Health Canada and the Aboriginal Diabetes Initiative, scale-up toolkits will be developed in order to build on lessons-learned, tools and methods, and to fuel sustainability and spread of successful innovations. DISCUSSION: The outcomes of this research program, its related cost and the subsequent policy recommendations, will have the potential to significantly affect future policy decisions pertaining to chronic disease care in First Nations communities in Canada. TRIAL REGISTRATION: Current ClinicalTrial.gov protocol ID NCT02234973 . Date of Registration: July 30, 2014.

Case study of evaluations that go beyond clinical outcomes to assess quality improvement diabetes programmes using the Diabetes Evaluation Framework for Innovative National Evaluations (DEFINE).

RATIONALE, AIMS AND OBJECTIVES: Investments in efforts to reduce the burden of diabetes on patients and health care are critical; however, more evaluation is needed to provide evidence that informs and supports future policies and programmes. The newly developed Diabetes Evaluation Framework for Innovative National Evaluations (DEFINE) incorporates the theoretical concepts needed to facilitate the capture of critical information to guide investments, policy and programmatic decision making. The aim of the study is to assess the applicability and value of DEFINE in comprehensive real-world evaluation. METHOD: Using a critical and positivist approach, this intrinsic and collective case study retrospectively examines two naturalistic evaluations to demonstrate how DEFINE could be used when conducting real-world comprehensive evaluations in health care settings. RESULTS: The variability between the cases and the evaluation designs are described and aligned to the DEFINE goals, steps and sub-steps. The majority of the theoretical steps of DEFINE were exemplified in both cases, although limited for knowledge translation efforts. Application of DEFINE to evaluate diverse programmes that target various chronic diseases is needed to further test the inclusivity and built-in flexibility of DEFINE and its role in encouraging more comprehensive knowledge translation. CONCLUSIONS: This case study shows how DEFINE could be used to structure or guide comprehensive evaluations of programmes and initiatives implemented in health care settings and support scale-up of successful innovations. Future use of the framework will continue to strengthen its value in guiding programme evaluation and informing health policy to reduce the burden of diabetes and other chronic diseases.

Impact of a quality improvement program on primary healthcare in Canada: a mixed-method evaluation.

PURPOSE: Rigorous comprehensive evaluations of primary healthcare (PHC) quality improvement (QI) initiatives are lacking. This article describes the evaluation of the Quality Improvement and Innovation Partnership Learning Collaborative (QIIP-LC), an Ontario-wide PHC QI program targeting type 2 diabetes management, colorectal cancer (CRC) screening, access to care, and team functioning. METHODS: This article highlights the primary outcome results of an external retrospective, multi-measure, mixed-method evaluation of the QIIP-LC, including: (1) matched-control pre-post chart audit of diabetes management (A1c/foot exams) and rate of CRC screening; (2) post-only advanced access survey (third-next available appointment); and (3) post-only semi-structured interviews (team functioning). RESULTS: Chart audit data was collected from 34 consenting physicians per group (of which 88% provided access data). Between-group differences were not statistically significant (A1c [p=0.10]; foot exams [p=0.45]; CRC screening [p=0.77]; advanced access [p=0.22]). Qualitative interview (n=42) themes highlighted the success of the program in helping build interdisciplinary team functioning and capacity. CONCLUSION: The rigorous design and methodology of the QIIP-LC evaluation utilizing a control group is one of the most significant efforts thus far to demonstrate the impact of a QI program in PHC, with improvements over time in both QIIP and control groups offering a likely explanation for the lack of statistically significant primary outcomes. Team functioning was a key success, with team-based chronic care highlighted as pivotal for improved health outcomes. Policy makers should strive to endorse QI programs with proven success through rigorous evaluation to ensure evidence-based healthcare policy and funding.

Improving chronic care through continuing education of interprofessional primary healthcare teams: a process evaluation.

Process evaluations assess program structures and implementation processes so that outcomes can be accurately interpreted. This article reports the results of a process evaluation of Partnerships for Health, an initiative targeting interprofessional primary healthcare teams to improve chronic care in Southwestern Ontario, Canada. Program documentation, participant observation, and in-depth interviews were used to capture details about the program structure, implementation process, and experience of implementers and participants. Results suggest that the intended program was modified during implementation to better meet the needs of participants and to overcome participation barriers. Elements of program activities perceived as most effective included series of off-site learning/classroom sessions, practice-based/workplace information-technology (IT) support, and practice coaching because they provided: dedicated time to learn how to improve chronic care; team-building/networking within and across teams; hands-on IT training/guidance; and flexibility to meet individual practice needs. This process evaluation highlighted key program activities that were essential to the continuing education (CE) of interprofessional primary healthcare teams as they attempted to transform primary healthcare to improve chronic care.

Results of a mixed-methods evaluation of partnerships for health: a quality improvement initiative for diabetes care.

PURPOSE: Quality improvement (QI) initiatives have been implemented to facilitate transition to a chronic disease management approach in primary health care. However, the effect of QI initiatives on diabetes clinical processes and outcomes remains unclear. This article reports the effect of Partnerships for Health, a QI program implemented in Southwestern Ontario, Canada, on diabetes clinical process and outcome measures and describes program participants' views of elements that influenced their ability to reach desired improvements. METHODS: Part of an external, concurrent, comprehensive, mixed-methods evaluation of Partnerships for Health, a before/after audit of 30 charts of patient of program physicians (n = 35) and semistructured interviews with program participants (physicians and allied health providers) were conducted. RESULTS: The proportion of patients (n = 998) with a documented test/examination for the following clinical processes significantly improved (P ≤ .005): glycosylated hemoglobin (A1c), cholesterol, albumin-to-creatinine ratio, serum creatinine, glomerular filtration rate, electrocardiogram, foot/eye/neuropathy examination, body mass index, waist circumference, and depression screening. Data showed intensification of treatment and significant improvement in the number of patients at target for low-density lipoprotein (LDL) and blood pressure (BP) (P ≤ .001). Mean LDL and BP values decreased significantly (P ≤ .01), and an analysis of patients above glycemic targets (A1c >7% at baseline) showed a significant decrease in mean A1c values (P ≤ .01). Interview participants (n = 55) described using a team approach, improved collaborative and proactive care through better tracking of patient data, and increased patient involvement as elements that positively influenced clinical processes and outcomes. CONCLUSIONS: QI initiatives like Partnerships for Health can result in improved diabetes clinical process and outcome measures in primary health care.

The mentoring needs of trainees in family practice.

This qualitative study examined medical students' and family practice residents' ideas, perceptions, and experiences of being mentored and their expectations of the mentoring experience. Eight focus groups and 16 individual interviews were used to collect data from 49 medical students and 29 family practice residents. Interviews and focus groups were audiotaped and transcribed verbatim. The analysis was iterative and interpretive, using both individual and team analyses. The analysis of the data revealed two central but related themes. The first theme reflected participants' overall experiences with mentors composed of three distinct elements: mentor roles (e.g. coach, advisor) and attributes (e.g. openness and approachability), interactions with mentors, and early exposure to family practice mentors (e.g. observing patient encounters). The second theme explicated the trainees' specific learning needs to be addressed by mentors that were categorised into three distinct yet overlapping levels: 1 practice level (i.e. guidance regarding the logistics of practice management) 2 system level (i.e. knowledge about the medical community as well as community resources) 3 personal level (i.e. guidance in balancing personal and professional responsibilities). Having the option of selecting multiple mentors to address unique aspects of the mentees' personal and professional development is critical in respecting the evolutionary nature and fluidity of the mentoring experience.

What do practitioners think? A qualitative study of a shared care mental health and nutrition primary care program.

OBJECTIVE: To develop an in-depth understanding of a shared care model from primary mental health and nutrition care practitioners with a focus on program goals, strengths, challenges and target population benefits. DESIGN: Qualitative method of focus groups. SETTING/PARTICIPANTS: The study involved fifty-three practitioners from the Hamilton Health Service Organization Mental Health and Nutrition Program located in Hamilton, Ontario, Canada. METHOD: Six focus groups were conducted to obtain the perspective of practitioners belonging to various disciplines or health care teams. A qualitative approach using both an editing and template organization styles was taken followed by a basic content analysis. MAIN FINDINGS: Themes revealed accessibility, interdisciplinary care, and complex care as the main goals of the program. Major program strengths included flexibility, communication/collaboration, educational opportunities, access to patient information, continuity of care, and maintenance of practitioner and patient satisfaction. Shared care was described as highly dependent on communication style, skill and expertise, availability, and attitudes toward shared care. Time constraint with respect to collaboration was noted as the main challenge. CONCLUSION: Despite some challenges and variability among practices, the program was perceived as providing better patient care by the most appropriate practitioner in an accessible and comfortable setting.