Evaluation logic in practice: Findings from two empirical investigations of American Evaluation Association members.

Surveys of two independent random samples of American Evaluation Association (AEA) members were conducted to investigate application of the logic of evaluation in their evaluation practice. This logic consists of four parts: (1) establish criteria, (2) set standards, (3) measure performance on criteria and compare to standards, and (4) synthesize into a value judgment. Nearly three-fourths (71.84% ±â€¯5.98%) of AEA members are unfamiliar with this logic, yet a majority also indicate its importance and utility for evaluation practice. Moreover, and despite unfamiliarity with the four steps of the logic of evaluation, many AEA members identify evaluative criteria (82.41% ±â€¯3.34%), set performance standards (60.55% ±â€¯7.39%), compare performance to standards (62.14% ±â€¯5.98%), and synthesize into an evaluative conclusion (75.00% ±â€¯5.80%) in their evaluation practice. Much like the working logic of evaluation, however, application of the general logic varies widely.

Social capital, substance use disorder and depression among youths.

BACKGROUND: Social capital - the network of social connections that exists among people - is known to be related to depression and substance use among adults. However, little is known about these relationships among adolescents, even though this age group is vulnerable due to factors of peer pressure, family, neighborhood, and maturational changes. OBJECTIVES: To evaluate the associations among social capital, substance use disorder and depression on a sample of 17 705 respondents between the ages of 12 and 17 in the 2009 National Survey of Drug Use and Health. METHODS: Structural equation modeling was used to examine social capital; responses to 48 items differentiated into two factors that measured structural social and cognitive social capital. Adolescent depression and substance use disorder were measured as past-year major depressive episodes and substance use disorder according to DSM-IV criteria. RESULTS: Structural social capital was associated with substance use disorder (ß = -0.12; p = 0.001) and depression (ß = -0.19; p = 0.001). Cognitive social capital was associated with substance use disorder (ß = -0.17; p = 0.001), but not with depression (ß = -0.002; p > 0.005). Substance use disorder mediated the association between structural and cognitive social capital and depression (ß = 0.06; p = 0.001). CONCLUSION: There was support for associations among youth structural and cognitive social capital, substance use disorder and depression. These findings suggest that additional research of a longitudinal nature is needed to determine causal connections among social capital, depression and substance use disorder for adolescents.

An overview of evidence-based program registers (EBPRs) for behavioral health.

Evaluations of behavioral health interventions have identified many that are potentially effective. However, clinicians and other decision makers typically lack the time and ability to effectively search and synthesize the relevant research literature. In response to this opportunity, and to increasing policy and funding pressures for the use of evidence-based practices, a number of "what works" websites have emerged to assist decision makers in selecting interventions with the highest probability of benefit. However, these registers as a whole are not well understood. This article, which represents phase one of a concurrent mixed methods study, presents a review of the scopes, structures, dissemination strategies, uses, and challenges faced by evidence-based registers in the behavioral health disciplines. The major findings of this study show that in general, registers of evidence-based practices are able, to a degree, to identify the most effective practices meet this need to a degree. However, much needs to be done to improve the ability of the registers to fully realize their purpose.

Comparing rating paradigms for evidence-based program registers in behavioral health: evidentiary criteria and implications for assessing programs.

Decision makers need timely and credible information about the effectiveness of behavioral health interventions. Online evidence-based program registers (EBPRs) have been developed to address this need. However, the methods by which these registers determine programs and practices as being "evidence-based" has not been investigated in detail. This paper examines the evidentiary criteria EBPRs use to rate programs and the implications for how different registers rate the same programs. Although the registers tend to employ a standard Campbellian hierarchy of evidence to assess evaluation results, there is also considerable disagreement among the registers about what constitutes an adequate research design and sufficient data for designating a program as evidence-based. Additionally, differences exist in how registers report findings of "no effect," which may deprive users of important information. Of all programs on the 15 registers that rate individual programs, 79% appear on only one register. Among a random sample of 100 programs rated by more than one register, 42% were inconsistently rated by the multiple registers to some degree.

An application of a new evidence grading system to research on the chronic care model.

Methodological quality undergirds all evidence-based medicine because without strong evidence supporting or refuting the efficacy of an intervention, the movement toward basing medical decisions and practice on scientific evidence is not sustainable. Recently, the consensus that had existed regarding the hierarchy of evidence produced by a study design was challenged on the basis that existing guidelines failed to properly define key terms, weight the merits of certain non-randomized controlled trials, and employ a comprehensive list of study design limitations to render evaluative conclusions, to name a few of the challenges. The present study introduces a new grading system that overcomes, or at the very least greatly diminishes, these challenges. This new method is applied to the literature on the Chronic Care Model and the results are then compared to several of the most popular grading guidelines currently in use. These results revealed substantial differences between the guidelines in accordance with previous research that challenged existing methods. Furthermore, the present study lends support to the proposed grading guideline although further research into its validity and reliability is needed.

Structure and measurement properties of the Patient Assessment of Chronic Illness Care instrument.

OBJECTIVE: In this study, the authors investigated the psychometric properties of a modified version of the Patient Assessment of Chronic Illness Care (PACIC) - a self-report instrument designed to measure the extent to which patients with chronic illness receive care congruent with the chronic care model. RESEARCH DESIGN AND METHOD: Five hundred and twenty-nine (529) type 2 diabetics were surveyed with a modified-PACIC. RESULTS: An exploratory and confirmatory factor analysis was conducted that accounted for the ordinal nature of the PACIC items. Ordinal alpha (0.972) and omega (0.973) reliability coefficients were calculated on the latent factor underlying the modified-PACIC. Correlation analyses were used to investigate the relationship between the modified-PACIC and numerous clinical labs related to diabetes. Validity for the original five-factor structure proposed by developers of the PACIC could not be confirmed. CONCLUSIONS: More research focusing on the instrument's psychometric properties is necessary before researchers use it to investigate the chronic care model and patient health. Researchers will need to ensure that the composite scores underlying the instrument are normally distributed and correlated with behavioural and health indexes.

Development and evaluation of the short version of the Patient Assessment of Chronic Illness Care instrument.

OBJECTIVES: The Patient Assessment of Chronic Illness Care (PACIC) was developed for measuring the extent to which patients receive care congruent with the chronic care model (CCM). The purpose of this study was to develop a short version of the PACIC with better psychometric properties than the original instrument. METHODS: Two samples of 529 and 361 type 2 diabetic patients completed a modified PACIC. A short-version PACIC instrument was developed and validated using parallel analysis to determine the number of factors, confirmatory factor analysis (CFA) within an exploratory factor analysis framework (E/CFA) was conducted to explore the measurement structure of the full instrument, and a CFA was performed to confirm the hypothesized structure. RESULTS: The results demonstrated that the PACIC is unidimensional and that it can be reduced to 11 items with no loss in psychometric properties. No demographic variables or clinical assays were found to be related to the PACIC. DISCUSSION: A short-version PACIC is now available and ready for use in research with diabetic patients. Its use is encouraged in future research, particularly in the exploration of its validity against actual CCM services delivered and long-term clinical outcomes.