Evaluation metrics for biostatistical and epidemiological collaborations.

Increasing demands for evidence-based medicine and for the translation of biomedical research into individual and public health benefit have been accompanied by the proliferation of special units that offer expertise in biostatistics, epidemiology, and research design (BERD) within academic health centers. Objective metrics that can be used to evaluate, track, and improve the performance of these BERD units are critical to their successful establishment and sustainable future. To develop a set of reliable but versatile metrics that can be adapted easily to different environments and evolving needs, we consulted with members of BERD units from the consortium of academic health centers funded by the Clinical and Translational Science Award Program of the National Institutes of Health. Through a systematic process of consensus building and document drafting, we formulated metrics that covered the three identified domains of BERD practices: the development and maintenance of collaborations with clinical and translational science investigators, the application of BERD-related methods to clinical and translational research, and the discovery of novel BERD-related methodologies. In this article, we describe the set of metrics and advocate their use for evaluating BERD practices. The routine application, comparison of findings across diverse BERD units, and ongoing refinement of the metrics will identify trends, facilitate meaningful changes, and ultimately enhance the contribution of BERD activities to biomedical research.

Factors associated with alcohol use, depression, and their co-occurrence during pregnancy.

BACKGROUND: Alcohol use and depression each adversely affect birth outcomes, but the impact of their co-occurrence among pregnant women is not well understood. In this study, we examined factors associated with alcohol use, depression, and their co-occurrence during pregnancy. METHODS: We analyzed datasets from 2 longitudinal studies conducted nearly 20 years apart in the same outpatient prenatal clinic of an urban women's hospital. Participants included 278 women recruited from 1982 to 1985 for the Maternal Health Practices and Child Development (MHPCD) Study and 209 women recruited from 2000 to 2002 for the Health Outcomes from Prenatal Education (HOPE) Study. Both studies selected women on the basis of their level of alcohol use early in pregnancy. We used multinomial logistic regression models to test multiclassification prediction of alcohol use, depression, and their co-occurrence during pregnancy. RESULTS: In the second and third trimesters, more MHPCD participants than HOPE participants consumed alcohol (67% vs. 20%), experienced depression (85% vs. 34%), and had co-occurring drinking and depression (56% vs. 10%) (p < 0.001 for each). For the MHPCD cohort, smoking predicted alcohol use. There were no significant predictors for depression alone or the co-occurrence. For the HOPE cohort, older age and smoking were predictors of alcohol use, smoking and less education were predictors of depression, and illicit drug use was a predictor of the co-occurrence of alcohol use and depression (p < 0.05 for all relationships). CONCLUSIONS: Smoking, older age, lower education, and illicit substance use predicted alcohol and/or probable depression in the second and third trimesters among women who drank in the first trimester.

A risk stratification tool to assess commercial influences on continuing medical education.

INTRODUCTION: Heightened concerns about industry influence on continuing medical education (CME) have prompted tighter controls on the management of commercial funding and conflict of interest. As a result, CME providers must closely monitor their activities and intervene if bias or noncompliance with accreditation standards is likely. Potential for industry influence can be difficult to assess at a stage in the planning process when mitigation strategies can assure balance and content validity. Few tools exist to aid providers in this regard. METHODS: A 12-item instrument was designed to assess risk for commercial influence on CME. To determine reliability and validity, a cohort of experienced CME professionals applied the tool to standardized "cases" representing CME activities in the early stages of planning. Results were compared with the experts' assignment of the same cases to one of four risk categories. A survey of study participants was conducted to ascertain usefulness and potential applications of the tool. RESULTS: Analysis demonstrated strong intraclass correlation across cases (0.90), interrater reliability (94%), and correlation between assessment of risk with and without the tool (Spearman coefficient, 0.93, p < 0.01; weighted kappa, 0.59). Participants found the tool easy to use and of potential benefit to their CME office. DISCUSSION: The Consortium for Academic Continuing Medical Education (CACME) risk stratification tool can help CME providers identify activities that must be closely monitored for potential industry influence, remain aware of factors that place programming at risk for noncompliance with accreditation standards, and substantiate the allocation of resources by the CME office.

Creating effective career development programs.

This paper is the fourth in a 5-part series that focuses on educating and training the clinical and translational science workforce. The goal of this paper is to delineate components of effective career development programs that go beyond didactic training. All academic health centers with a Clinical and Translational Science Award have a KL2 career development award for junior faculty, and many also have a TL1 training program for predoctoral and postdoctoral fellows. The training across these programs varies, however junior investigators across the United States experience similar challenges. Junior investigators can get overwhelmed with the demands of building their own research program, particularly in academia. 1Often, they are sidetracked by competing demands that can derail their progress. In these situations, junior investigators experience frustration and may search for alternative career paths. By providing them with additional professional skills in the 5 domains of: (1) self-awareness; (2) selecting the right topic and securing funding; (3) getting adequate support; (4) working with others; and (5) managing yourself, your career, and your demands. We will give junior investigators additional tools to manage these demands and facilitate their own career success.

Factor analysis of the Alcohol Expectancy Questionnaire in a middle-aged community sample.

OBJECTIVE: The purpose of this study is to evaluate the psychometric properties of the Alcohol Expectancy Questionnaire (AEQ) in a large community sample. METHOD: This study is a cross-sectional survey design that utilized face-to-face interviews. Participants were recruited through their participation in the St. Louis Epidemiological Catchment Area study. Subjects (N = 753) were re-interviewed 11 years after their initial ECA interview, with the goal of detection of new cases of alcoholism. Of the 733 who completed the AEQ, 62% were women and 52% were married. Ages ranged from 29 to 62, with an avenge (SD) age of 43 (8.18). The racial breakdown of the sample was diverse, with 57% white, 38% black and 5% of another ethnicity. RESULTS: The AEQ is comprised of six factors that explain 47% of the variance in the AEQ. Five of the factors have strong internal consistency, ranging from 0.78 to 0.93. The sixth factor has an internal consistency of 0.51. CONCLUSIONS: Using a large community sample, the results reported herein replicate those of the original researchers.

Assessing the health of adult daughter former caregivers for elders with Alzheimer's disease.

Past research is varied in assessing the effect of caregiving on health, particularly caregivers in the postcaregiving phase. The variation may be due, in part, to methodological issues, including the use of health measures not psychometrically tested. The study examines the Medical Outcomes Study Short Form 36 (SF-36) health survey with 102 former caregivers whose family member was deceased for at least one year at the time of the study and had been identified as having Alzheimer's disease or a related disorder. The SF-36 measures eight dimensions of physical and mental health and has been tested on a variety of populations, though not with former daughter caregivers. Confirmatory Factor Analysis supported the factorial validity of the SF-36 for this population, indicating it is a promising tool for understanding postcaregiver health.

Home-based treatment, rates of ambulatory follow-up, and psychiatric rehospitalization in a Medicaid managed care population.

This study reports on the effect of home-based mental health treatment following psychiatric hospitalization on ambulatory follow-up rates and readmission rates in a Medicaid managed care population. Logistic regression models were used to predict the odds of ambulatory treatment after hospitalization and to predict rehospitalization. A consumer who received in-home treatment was 22 times more likely to follow-through with aftercare treatment of more than one visit than were those who did not receive in-home treatment. However, in-home treatment, age, gender, and previous hospitalization did not significantly predict the odds of rehospitalization. While home-based services did not reduce the incidence of rehospitalization, providing services within the home has the potential to increase attendance in aftercare services by Medicaid managed care recipients.

Development of a healthcare quality improvement measurement tool: results of a content validity study.

Current methods of measuring continuous quality improvement (CQI) implementation are too long and not comprehensive. A new survey for CQI implementation was developed and tested for content validity using a panel of 8 experts--7 from the United States and 1 from England. The survey was reduced from 70 items to 22. The resultant survey had a clarity interrater agreement (IR) of .91, a representativeness IR of .93, a clarity content validity index (CVI) of .73, and a representativeness CVI of .91. Content validity served as an excellent data reduction method in building a valid, concise, and comprehensive measure of CQI implementation.

Shortening the Work Preference Inventory for use with physician scientists: WPI-10.

The Work Preference Inventory (WPI) is a four-factor, 30-item measure that assesses work motivation. Used to help individuals choose appropriate career paths, its length contributes to response burden, especially when combined with other measures. We aimed to develop a shortened, valid, and reliable version of the WPI. Trainees at the University of Pittsburgh's Institute for Clinical Research Education completed the 30-item WPI between 2007 and 2012. We conducted exploratory and confirmatory factor analyses to reduce the number of items. Of the 402 eligible trainees, 371 (92%) provided data for the exploratory factor analysis (EFA), and 134 of the eligible 144 trainees (93%) provided data for the confirmatory factor analysis (CFA). EFA revealed four factors that were roughly equivalent to those of the original. CFA used the three items with the highest loadings on each factor, with two items removed due to low loadings and R-squareds, resulting in a 10-item scale. Cronbach's alpha for each of the four factors ranged from 0.68 to 0.76. Factors in the WPI-10 were strongly and significantly associated with factors in the original WPI, indicating strong validity of the shortened measure. The WPI-10 shows evidence for similar validity and reliability to the original instrument while reducing respondent burden.

Brief motivational enhancement intervention to prevent or reduce postpartum alcohol use: a single-blinded, randomized controlled effectiveness trial.

AIMS: The aim of this study is to assess the effect of brief motivational enhancement intervention postpartum alcohol use. DESIGN: This study is a single-blinded, randomized controlled effectiveness trial in which pregnant women were assigned to receive usual care or up to 5 face-to-face brief motivational enhancement sessions lasting 10-30 minutes each and occurring at study enrollment, 4 and 8 weeks after enrollment, 32 weeks of gestation, and 6 weeks postpartum. SETTING: The setting is in a large, urban, obstetrics clinic. PARTICIPANTS: Participants were women who were ≥ 18 years old, <20 weeks of gestation, and consumed alcohol during pregnancy. Of 3438 women screened, 330 eligible women were assigned to usual care (n = 165) or intervention (n=165). Due to missing data, we analyzed 125 in the intervention group and 126 in the usual care group. MEASUREMENTS: The measurements were the proportion of women with any alcohol use and the number of drinks per day, reported via follow-up telephone interviews at 4 and 8 weeks after enrollment, 32 weeks of gestation, and 6 weeks, 6 months, and 12 months postpartum. FINDINGS: In random effects models adjusted for confounders, the intervention group was less likely to use any alcohol (odds ratio 0.50; 95% confidence interval [CI], 0.23-1.09; P=0.08) and consumed fewer drinks per day (coefficient -0.11; 95% CI -0.23-0.01; P=0.07) than, the usual care group in the postpartum period but these differences were non-significant. Missing data during the prenatal period prevented us from modeling prenatal alcohol use. CONCLUSIONS: Brief motivational enhancement intervention delivered in an obstetrical outpatient setting did not conclusively decrease alcohol use during the postpartum period.

Common metrics to assess the efficiency of clinical research.

The Clinical and Translational Science Award (CTSA) program was designed by the National Institutes of Health (NIH) to develop processes and infrastructure for clinical and translational research throughout the United States. The CTSA initiative now funds 61 institutions. In 2012, the National Center for Advancing Translational Sciences, which administers the CTSA Program, charged the Evaluation Key Function Committee of the CTSA Consortium to develop common metrics to assess the efficiency of clinical research processes and outcomes. At this writing, the committee has identified 15 metrics in six categories. It has also developed a standardized protocol to define, pilot test, refine, and implement the metrics. The ultimate goal is to learn critical lessons about how to evaluate the processes and outcomes of clinical research within the CTSAs and beyond. This article describes the work involved in developing and applying common metrics and benchmarks of evaluation.

A shortened version of the Clinical Research Appraisal Inventory: CRAI-12.

PURPOSE: The original Clinical Research Appraisal Inventory (CRAI), which assesses the self-confidence of trainees in performing different aspects of clinical research, comprises 92 items. Completing the lengthy CRAI is time-consuming and represents a considerable burden to respondents, yet the CRAI provides useful data for evaluating research training programs. The purpose of this study is to develop a shortened version of the CRAI and to test its validity and reliability. METHOD: Trainees in clinical research degree and career development programs at the University of Pittsburgh's Institute for Clinical Research Education completed the 92-item CRAI between 2007 and 2012, inclusive. The authors conducted, first, exploratory factor analysis on a training dataset (2007-2010) to reduce the number of items and, then, confirmatory factor analyses on a testing dataset (2011-2012) to test the psychometric properties of the shortened version. RESULTS: Of 546 trainees, 394 (72%) provided study data. Exploratory factor analysis revealed six distinct factors, and confirmatory factor analysis identified the two items with the highest loadings per factor, for a total of 12 items. Cronbach alpha for the six new factors ranged from 0.80 to 0.94. Factors in the 12-item CRAI were strongly and significantly associated with factors in the 92-item CRAI; correlations ranged from 0.82 to 0.96 (P < .001 for each). CONCLUSIONS: The 12-item CRAI is faster and less burdensome to complete but retains the strong psychometric properties of the original CRAI.

Psychometric properties of the mentor role instrument when used in an academic medicine setting.

The Ragins and McFarlin Mentor Role Instrument (RMMRI) was originally developed to measure perceptions of mentoring relationships in research and development organizations. The current study was designed to evaluate the RMMRI's reliability and validity when the instrument was administered to clinical and translational science trainees at an academic medical center. The 33-item RMMRI was administered prospectively to a cohort of 141 trainees at the University of Pittsburgh in 2007-2008. Likert-scale items focused on perceptions of five mentoring roles in the career dimension (sponsor, coach, protector, challenger, and promoter) and six mentoring roles in the psychosocial dimension (friend, social associate, parent, role model, counselor, and acceptor). Outcome items included overall perceptions of mentoring satisfaction and effectiveness. Of 141 trainees, 53% were male, 66% were white, 22% were Asian, and 59% were medical doctors. Mean age was 32 years. Analyses showed strong within-factor inter-item correlations (Pearson Coefficients of 0.57-0.93); strong internal consistency (Cronbach alphas of 0.82-0.97); confirmatory factorial validity, as demonstrated by confirmatory factor analysis of the two mentoring dimensions, 11 mentoring roles, and 33 RMMRI items; and concurrent validity, as demonstrated by strong correlations (Pearson Coefficients of 0.56-0.71) between mentoring dimensions, satisfaction, and effectiveness. This article concludes that the RMMRI shows reliability and validity in capturing the multidimensional nature of mentoring when administered to clinical and translational science trainees in the academic setting.

Defining translational research: implications for training.

Because translational research is not clearly defined, developers of translational research programs are struggling to articulate specific program objectives, delineate the knowledge and skills (competencies) that trainees are expected to develop, create an appropriate curriculum, and track outcomes to assess whether program objectives and competency requirements are being met. Members of the Evaluation Committee of the Association for Clinical Research Training (ACRT) reviewed current definitions of translational research and proposed an operational definition to use in the educational framework. In this article, the authors posit that translational research fosters the multidirectional and multidisciplinary integration of basic research, patient-oriented research, and population-based research, with the long-term aim of improving the health of the public. The authors argue that the approach to designing and evaluating the success of translational training programs must therefore be flexible enough to accommodate the needs of individual institutions and individual trainees within the institutions but that it must also be rigorous enough to document that the program is meeting its short-, intermediate-, and long-term objectives and that its trainees are meeting preestablished competency requirements. A logic model is proposed for the evaluation of translational research programs.