The development and assessment of valid measures of service delivery to enhance inference in outcome-based research: measuring attendance at self-help group meetings.

The article focuses on the validity of measures of service delivery; any attempt to establish a linkage between service and outcomes follows the assumption that services have been given and received. Claims relating to the benefits of attending self-help groups on the mental health functioning of individuals with depression or manic depression were examined in the context of a well-controlled study and a validated measure of individual attendance. A test was conducted to probe the conjecture that the greater the degree of involvement in self-help groups, the greater the improvement in functioning. Additional questions related to individual patterns of attendance, generalization of the intervention's effects, and migration from "home" to other meeting sites.

Exploratory research synthesis--methodological considerations for addressing limitations in data quality.

Exploratory meta-analysis or research synthesis has been advocated as a way of developing important hypotheses for further study. An exploratory research synthesis was conducted on the carotid endarterectomy (CE) literature to illustrate this method. The CE scientific literature is similar to that of many other new medical interventions because it contains numerous limitations to data quality. Exploratory research synthesis of such literature necessitates a number of methodological and statistical considerations to address these limitations, including the problems of missing data, appropriate unit of analysis, nonnormal distribution of outcomes, and lack of controlled studies. Strengths and limitations of the exploratory research synthesis approach are discussed within the context of public policy decisions for assessing medical technologies.

Teaching pedestrian safety skills to young children: an analysis and one-year followup.

Twenty-four elementary school children in grades K-3 participated in a study to teach six street-crossing skills: (1) wait at curb, (2) look all ways, (3) watch vehicle distance, (4) walk, (5) continue to look, and (6) use crosswalk. The effects of an instructional package implemented on the street corner were evaluated using a multiple-baseline design across two groups of six children at each of two schools. Rapid acquisition of pedestrian skills was evident at both schools. Average skill levels improved from 44% during baseline to 97% after training at School A and from 21% to 86% at School B. Data taken at a second street at each school were used to assess setting generality of safety behaviors. A one-year followup of 14 children indicated that pedestrian safety skills either maintained at high levels or could be quickly recovered from intermediate levels after remedial training. This research represents a first step in the solution of just one of the many community problems involving safety-deficient settings.

Utilization analysis of a pedestrian safety training program.

A previously developed and analyzed pedestrian safety training program was used to teach appropriate street-crossing behaviors to kindergarteners and first graders. Adult crossing guards were trained to provide pedestrian safety instruction. Trained observers monitored the quality of instructions given by crossing guards and the pedestrian behavior of young children as they crossed the street. A multiple-baseline analysis of the effects of two training programs indicated that guards were able to deliver the pedestrian safety program to several groups of children with a high degree of competence after receiving a single videotape and role playing training session. Furthermore, children's level of appropriate street crossing increased contemporaneous to the change in guard behavior both on the street where training was delivered and on a second street where no training was previously delivered. Utilization analysis of the guard training program indicated that one cannot expect to produce consistently high levels of street-crossing behavior by implementing only the "show and tell" portions of the training package. Similarly, results suggested that one is unlikely to produce consistently high quality guard training behavior by only giving written instructions describing how pedestrian training should be administered.

Cumulating quality of life results in controlled trials of coronary artery bypass graft surgery.

Many studies evaluating the effectiveness of coronary artery bypass graft surgery allude to the quality of life benefit resulting from surgery. However, no comprehensive empirical estimate of the absolute or relative magnitude of this benefit is currently available. This paper presents a data synthesis of the research literature on bypass surgery to derive such an estimate. It uses follow-up measures of the percent of patients who were angina-free within both the surgical and medical groups of 14 controlled trials to estimate the quality of life benefit following surgery. Results based on the longest reported follow-up period suggest that the chances are approximately 25 to 40% greater that patients will be angina-free if they receive surgery rather than medical treatment. Estimates of benefit are about 15% less in randomized controlled trials compared to controlled trials that used a matching strategy. These results are unlikely to be affected by related factors such as the percentage of patients who crossover from the medical group to the surgical group or the specific method of calculating anginal relief used in this research report. However, differential patients selection may account for the observed design effect.

Using research synthesis in medical technology assessment.

This article describes the use of research synthesis procedures (e.g., meta-analysis) in evaluating medical technologies. The synthesis process involves retrieval, extraction, and analysis of pertinent information from a set a research studies. There are several advantages of research synthesis: determining overall effectiveness, explaining seemingly contradictory findings, providing a standard measure of effect, and increasing statistical power. In addition, such syntheses provide timely results that can assist various policy decisions such as initiating clinical trials at NIH, reimbursement at HCFA, and answering NIH consensus conference questions. The authors' synthesis of the research literature on coronary artery bypass graft surgery (CABGS) illustrates how this method can address the various elements of a technology assessment such as safety, efficacy, and cost-effectiveness. The paper concludes by noting how research synthesis can be particularly advantageous in assessing technologies that are rapidly evolving (e.g., lytic agents for acute myocardial infarction).

Evaluating understanding of popular press reports of health research.

This research assessed the ability of a sample of persons on a college campus to understand media reports of health research. Three or four articles on each of five contemporary health topics (dietary cholesterol and heart disease, treatment for breast cancer, starch blockers, drug treatment for heart disease, test tube skin) were selected from widely circulated newspapers (e.g., New York Times) and magazines (e.g., Newsweek). A sample of 144 college students responded to content-based and application-based questions derived from photocopies of these popular press articles. The overall rate of reader misunderstanding approached 40% and generally fell between one third and one half for each of 16 articles representing five health topics. Several strengths and weaknesses of the research are considered as they relate to the accuracy of estimated error rates and to the generality of study findings. The implications of these findings for other areas of health (e.g., AIDS risk factor research) are also discussed.

Carotid endarterectomy for asymptomatic patients. Assessing results of a quantitative synthesis.

Carotid endarterectomy (CE) surgery for asymptomatic patients remains controversial despite hundreds of published studies and recent randomized trials. Safety and efficacy are assessed using a quantitative synthesis method derived from meta-analysis and a "critical multiplist" inference approach. In addition, multivariate analyses reveal that use of a surgical shunt could further improve CE outcomes. Methods are examined for both their "confirmatory" and "exploratory" value.

Consensus among experts and research synthesis. A comparison of methods.

A comparison of two assessment methods, consensus among experts and research synthesis of the scientific literature, was performed using a surgical procedure, carotid endarterectomy (CE), as an example. These two methods have been widely advocated as being scientifically valid. While the comparison revealed a number of areas of general agreement, important differences between the two methods emerged. For example, 30-day mortality for asymptomatic patients was considered an effective outcome (ranked first) by the synthesis, but only "equivocal" (ranked third) of six major indicators reported by the consensus method. The synthesis results are also consistent with other literature reviews as well as with recent large-scale randomized trial results. A number of factors that could account for differences between the two methods were examined. Overall, use of consensus panels may be appropriate early in the development of an intervention where the evidence is sparse, while quantitative research synthesis is preferable when a number of high-quality studies have been performed.