Who reviews the reviewers? Feasibility of using a fictitious manuscript to evaluate peer reviewer performance.

STUDY OBJECTIVE: To determine whether a fictitious manuscript into which purposeful errors were placed could be used as an instrument to evaluate peer reviewer performance. METHODS: An instrument for reviewer evaluation was created in the form of a fictitious manuscript into which deliberate errors were placed in order to develop an approach for the analysis of peer reviewer performance. The manuscript described a double-blind, placebo control study purportedly demonstrating that intravenous propranolol reduced the pain of acute migraine headache. There were 10 major and 13 minor errors placed in the manuscript. The work was distributed to all reviewers of Annals of Emergency Medicine for review. RESULTS: The manuscript was sent to 262 reviewers; 203 (78%) reviews were returned. One-hundred ninety-nine reviewers recommended a disposition for the manuscript: 15 recommended acceptance, 117 rejection, and 67 revision. The 15 who recommended acceptance identified 17.3% (95% confidence interval [CI] 11.3% to 23.4%) of the major and 11.8% (CI 7.3% to 16.3%) of the minor errors. The 117 who recommended rejection identified 39.1 % (CI 36.3% to 41.9%) of the major and 25.2% (CI 23.0% to 27.4%) of the minor errors. The 67 who recommended revision identified 29.6% (CI 26.1% to 33.1%) of the major and 22.0% (CI 19.3% to 24.8%) of the minor errors. The number of errors identified differed significantly across recommended disposition. Sixty-eight percent of the reviewers did not realize that the conclusions of the work were not supported by the results. CONCLUSION: These data suggest that the use of a preconceived manuscript into which purposeful errors are placed may be a viable approach to evaluate reviewer performance. Peer reviewers in this study failed to identify two thirds of the major errors in such a manuscript.

Reliability of editors' subjective quality ratings of peer reviews of manuscripts.

CONTEXT: Quality of reviewers is crucial to journal quality, but there are usually too many for editors to know them all personally. A reliable method of rating them (for education and monitoring) is needed. OBJECTIVE: Whether editors' quality ratings of peer reviewers are reliable and how they compare with other performance measures. DESIGN: A 3.5-year prospective observational study. SETTING: Peer-reviewed journal. PARTICIPANTS: All editors and peer reviewers who reviewed at least 3 manuscripts. MAIN OUTCOME MEASURES: Reviewer quality ratings, individual reviewer rate of recommendation for acceptance, congruence between reviewer recommendation and editorial decision (decision congruence), and accuracy in reporting flaws in a masked test manuscript. INTERVENTIONS: Editors rated the quality of each review on a subjective 1 to 5 scale. RESULTS: A total of 4161 reviews of 973 manuscripts by 395 reviewers were studied. The within-reviewer intraclass correlation was 0.44 (P<.001), indicating that 20% of the variance seen in the review ratings was attributable to the reviewer. Intraclass correlations for editor and manuscript were only 0.24 and 0.12, respectively. Reviewer average quality ratings correlated poorly with the rate of recommendation for acceptance (R=-0.34) and congruence with editorial decision (R=0.26). Among 124 reviewers of the fictitious manuscript, the mean quality rating for each reviewer was modestly correlated with the number of flaws they reported (R=0.53). Highly rated reviewers reported twice as many flaws as poorly rated reviewers. CONCLUSIONS: Subjective editor ratings of individual reviewers were moderately reliable and correlated with reviewer ability to report manuscript flaws. Individual reviewer rate of recommendation for acceptance and decision congruence might be thought to be markers of a discriminating (ie, high-quality) reviewer, but these variables were poorly correlated with editors' ratings of review quality or the reviewer's ability to detect flaws in a fictitious manuscript. Therefore, they cannot be substituted for actual quality ratings by editors.

Prospective validation of artificial neural network trained to identify acute myocardial infarction.

BACKGROUND: Artificial neural networks apply non-linear statistics to pattern recognition problems. One such problem is acute myocardial infarction (AMI), a diagnosis which, in a patient presenting as an emergency, can be difficult to confirm. We report here a prospective comparison of the diagnostic accuracy of a network and that of physicians, on the same patients with suspected AMI. METHODS: Emergency department physicians who evaluated 1070 patients 18 years or older presenting to the emergency department of a teaching hospital in California, USA with anterior chest pain indicated whether they thought these patients had sustained a myocardial infarction. The network analysed the patient data collected by the physicians during their evaluations and also generated a diagnosis. FINDINGS: The physicians had a diagnostic sensitivity and specificity for myocardial infarction of 73.3% (95% confidence interval 63.3-83.3%) and 81.1% (78.7-83.5%), respectively, while the network had a diagnostic sensitivity and specificity of 96.0% (91.2-100%) and 96.0% (94.8-97.2%), respectively. Only 7% of patients had had an AMI, a low frequency but typical for anterior chest pain. INTERPRETATION: The application of non-linear neural computational analysis via an artificial neural network to the clinical diagnosis of myocardial infarction appears to have significant potential.

A neural network trained to identify the presence of myocardial infarction bases some decisions on clinical associations that differ from accepted clinical teaching.

An artificial neural network trained to identify the presence of myocardial infarction has been shown to function with a high degree of accuracy. The effects on network diagnosis of some of the clinical input variables used by this network have previously been shown to be distributed over two distinct maxima. Analysis of the basis for this distribution by studying the specific patterns in which these variables had significantly different impacts on network diagnosis revealed that the differential impacts were due to the contexts in which the variables whose effects were bimodally distributed were placed. These contexts were defined by the values of the other input data used by the network. In a number of instances, the clinical relationships implied by these associations were divergent from prior knowledge about factors predictive of myocardial infarction. One implication of these findings is that this network, which has been shown to perform with a high degree of diagnostic accuracy, may be doing so by identifying relationships between inputted information that are divergent from accepted teaching.

Complexity, chaos and human physiology: the justification for non-linear neural computational analysis.

Background is presented to suggest that a great many biologic processes are chaotic. It is well known that chaotic processes can be accurately characterized by non-linear technologies. Evidence is presented that an artificial neural network, which is a known method for the application of non-linear statistics, is able to perform more accurately in identifying patients with and without myocardial infarction than either physicians or other computer paradigms. It is suggested that the improved performance may be due to the network's better ability to characterize what is a chaotic process imbedded in the problem of the clinical diagnosis of this entity.

Analysis of the clinical variables driving decision in an artificial neural network trained to identify the presence of myocardial infarction.

STUDY OBJECTIVE: To determine which clinical variables drive the output of an artificial neural network trained to identify the presence of myocardial infarction. DESIGN: Partial output analysis. SETTING: Tertiary university teaching center. PARTICIPANTS: Seven hundred six patients more than 18 years old presenting with anterior chest pain. MEASUREMENTS: Differential network output analysis. MAIN RESULTS: A methodology was developed as the first step in measuring the impact input clinical variables have on the output (diagnosis) of an artificial neural network trained to identify the presence of acute myocardial infarction. The methodology revealed that the network used the presence of ECG findings, as well as the presence of rales, syncope, jugular venous distension, response to trinitroglycerin, and nausea and vomiting, as major predictive sources. Although this first-step analysis studied individual variables, it must be stated that the network comes to clinical closure based on the settings of all variables in a pattern and that the impact of a single variable cannot be taken out of the context of a pattern. CONCLUSION: An artificial neural network trained to recognize the presence of myocardial infarction appears to place diagnostic importance on clinical variables that have not been shown previously to be highly predictive for infarction.

Use of an artificial neural network for the diagnosis of myocardial infarction.

OBJECTIVE: To validate prospectively the use of an artificial neural network to identify myocardial infarction in patients presenting to an emergency department with anterior chest pain. DESIGN: Prospective, blinded testing. SETTING: Tertiary university teaching center. PATIENTS: A total of 331 consecutive adult patients presenting with anterior chest pain. MEASUREMENTS: Diagnostic sensitivity and specificity with regard to the diagnosis of acute myocardial infarction. MAIN RESULTS: An artificial neural network was trained on clinical pattern sets retrospectively derived from the cases of 351 patients hospitalized with a high likelihood of having myocardial infarction. It was prospectively tested on 331 consecutive patients presenting to an emergency department with anterior chest pain. The ability of the network to distinguish patients with from those without acute myocardial infarction was compared with that of physicians caring for the same patients. The physicians had a diagnostic sensitivity of 77.7% (95% CI, 77.0% to 82.9%) and a diagnostic specificity of 84.7% (CI, 84.0% to 86.4%). The artificial neural network had a sensitivity of 97.2% (CI, 97.2% to 97.5%; P = 0.033) and a specificity of 96.2% (CI, 96.2% to 96.4%; P less than 0.001). CONCLUSION: An artificial neural network trained to identify myocardial infarction in adult patients presenting to an emergency department may be a valuable aid to the clinical diagnosis of myocardial infarction; however, this possibility must be confirmed through prospective testing on a larger patient sample.

The lack of full correlation between the Injury Severity Score and the resource needs of injured patients.

STUDY OBJECTIVE: To determine whether the Injury Severity Score (ISS) correlates with the resource requirements of severely injured patients by studying the association of the ISS with three major interventions (fluid resuscitation, invasive central nervous system monitoring, and acute operative repair) trauma centers routinely provide severely injured patients. DESIGN: Retrospective clinical review. SETTING: Level I trauma center. TYPE OF PARTICIPANTS: Eight hundred fourteen adult injured patients. MEASUREMENTS AND MAIN RESULTS: When an ISS of more than 9 was used as the definition of major trauma, the ISS undercorrelated 11% of the time with the need for any one procedure. When an ISS of more than 14 was used as the definition, it undercorrelated 20% of the time. CONCLUSION: The ISS may not be completely correlated with the resource requirements of injured patients and should not be used as the sole means by which to define major injury.

The trauma triage rule: a new, resource-based approach to the prehospital identification of major trauma victims.

STUDY OBJECTIVE: To develop a new trauma decision rule. DESIGN: Retrospective clinical review. SETTING: Level I trauma center. TYPE OF PARTICIPANTS: 1,004 injured adults. MEASUREMENTS AND MAIN RESULTS: A new trauma decision rule was derived from 1,004 injured adult patients using a new operational definition of major trauma. The rule, termed the Trauma Triage Rule, defines a major trauma victim as any injured adult patient whose systolic blood pressure is less than 85 mm Hg; whose motor component of the Glasgow Coma Score is less than 5; or who has sustained penetrating trauma of the head, neck, or trunk. Using the operational definition of major trauma, the rule had a sensitivity of 92% and a specificity of 92% when tested on the 1,004-patient cohort. CONCLUSION: The Trauma Triage Rule may significantly reduce overtriage while only minimally increasing undertriage. This approach must be validated prospectively before it can be used in the prehospital setting.

APACHE II scoring in the injured patient.

The purpose of this study was to measure the predictive power of Acute Physiology and Chronic Health Evaluation (APACHE II) with respect to mortality in a group of seriously injured patients and to compare this predictive power with that of the Trauma Score (TS) and the Injury Severity Score (ISS). Six hundred ninety-one helicopter-transported patients were studied. Individual logistic regressions demonstrated that all three scores had significant predictive power when considered individually (TS chi 2 = 136, p less than .0001; APACHE II chi 2 = 171, p less than .0001; ISS chi 2 = 109, p less than .0001). In addition, each severity score added significantly to the predictive power in a stepwise logistic regression (TS chi 2 = 15, p less than .0001; APACHE II chi 2 = 45, p less than .0001; ISS chi 2 = 15, p less than .0001). Areas under the receiver operating curves for the three scores were not significantly different (TS 0.8116, SD 0.0245; APACHE II 0.8515, SD 0.0204; ISS 0.7967; SD 0.0223). APACHE II is a good predictor of mortality, and its predictive power is complemented by TS and ISS.

Differences in air ambulance patient mix demonstrated by physiologic scoring.

Severity of illness or injury should be the primary justification for aeromedical transport. To determine whether differences in patient severity were detectable in air transport programs, helicopter-transported patients were examined by three established physiologic scores: the Trauma Score, the Acute Physiology and Chronic Health Evaluation Score, and the Rapid Acute Physiology Score. These scores were obtained prospectively on 1,868 consecutive patient transfer requests from six air medical services for periods ranging from two to six months. A patient meeting strict physiologic criteria was considered critically ill. Overall, 42.6% of the patients (range, 34.8% to 53.3%) were considered critically ill. Patients transported from inpatient hospital units and patients with cardiac disease were less likely to be critically ill than those transported emergently from scenes of accident or from emergency departments. There were also significant differences between programs with regard to the percentage of critically ill patients transported. This study suggests that physiologic scoring may be useful in comparing air ambulance programs and that a majority of patients transported by these services may not be critically ill.

The impact of a regionalized trauma system on trauma care in San Diego County.

A review of autopsy reports on traumatic deaths in 1986 was conducted to determine the impact on trauma mortality of the regionalized trauma system instituted in San Diego County in 1984. Determination of preventable death was made by a panel of experts and compared with an identical review of traumatic deaths in 1979, five years before the institution of regionalized trauma care. Of 211 traumatic deaths reviewed from 1986, two (1%) were classified as preventable, compared with 20 of 177 (11.4%) deaths in 1979 (P less than .001). A breakdown of trauma deaths into central nervous system and noncentral nervous system categories revealed the overall decline was in large part a consequence of the decline in non-central nervous system deaths from 16 of 83 in 1979 to one of 62 in 1986 (P less than .005). The decrease in central nervous system-related preventable deaths from four of 94 in 1979 to one of 149 in 1986 (P less than .10) was not statistically significant. Although it is likely the trauma system introduced in 1984 contributed to the decline in preventable death, it is not possible to isolate this variable from other changes that occurred during the interval between studies. A review of trauma deaths over the same time interval in a community with similar demographics but without a trauma system might help determine the relative contribution of the trauma system.

The failure of prehospital trauma prediction rules to classify trauma patients accurately.

Clinical prediction rules are used extensively by most regionalized trauma systems to identify which patients have sustained major injuries. Because of reported high misclassification rates of some of these rules and the known global difficulty of transporting prediction rules, four such rules (the Trauma Score, the CRAMS Scale, the Revised Trauma Score, and the Prehospital Index) and two newly derived rules were statistically analyzed using a cohort of 2,434 injured patients. All rules accurately predicted mortality with a minimum sensitivity and specificity of 85%. However, not one of the rules was able accurately to identify surviving patients who had sustained major injuries. In this instance, no rule was able to achieve a sensitivity of at least 70% while achieving a specificity of 70%. These results suggest that the problem with trauma prediction rules lies in the inherent limitations of the clinical data on which they are based. In view of this, the usefulness of existing prehospital trauma predictive rules must be questioned.

The impact of a physician as part of the aeromedical prehospital team in patients with blunt trauma.

To determine whether the presence of a physician in the prehospital setting influences patient outcome, the predicted mortality of 258 patients with blunt trauma treated and transported by a medical helicopter staffed by a flight nurse and flight paramedic was compared with that of 316 similar patients with blunt trauma treated and transported by a medical helicopter staffed by a flight nurse and flight physician. All patients were randomized between the two treatment teams. The mortality of the patients treated by the flight nurse/flight paramedic team was that predicted by the methodology. The mortality of the patients treated by the flight nurse/flight physician team was 35% lower than that predicted, and significantly lower than that of the flight nurse/flight paramedic-staffed helicopter.

The differential survival of trauma patients.

Advances in prehospital care and the development of regionalized trauma centers have reduced the mortality from major trauma. However, patients who have sustained severe brain injuries (Glasgow Coma Score less than or equal to 8) treated in such a setting still have a substantial mortality. In order to determine if mortality is a function of severe brain injury, 545 trauma patients with and without severe brain injury were prospectively studied. All patients were similarly treated by one medical center from the actual site of injury until discharge from the hospital. There were 104 patients who had sustained major blunt trauma with severe brain injury and 441 patients who had sustained major blunt trauma without severe brain injury. The mortality of the former group was 30.8%, whereas that of the latter group was 0.9% (p less than 0.0001).