Who reviews the reviewers? Feasibility of using a fictitious manuscript to evaluate peer reviewer performance.

STUDY OBJECTIVE: To determine whether a fictitious manuscript into which purposeful errors were placed could be used as an instrument to evaluate peer reviewer performance. METHODS: An instrument for reviewer evaluation was created in the form of a fictitious manuscript into which deliberate errors were placed in order to develop an approach for the analysis of peer reviewer performance. The manuscript described a double-blind, placebo control study purportedly demonstrating that intravenous propranolol reduced the pain of acute migraine headache. There were 10 major and 13 minor errors placed in the manuscript. The work was distributed to all reviewers of Annals of Emergency Medicine for review. RESULTS: The manuscript was sent to 262 reviewers; 203 (78%) reviews were returned. One-hundred ninety-nine reviewers recommended a disposition for the manuscript: 15 recommended acceptance, 117 rejection, and 67 revision. The 15 who recommended acceptance identified 17.3% (95% confidence interval [CI] 11.3% to 23.4%) of the major and 11.8% (CI 7.3% to 16.3%) of the minor errors. The 117 who recommended rejection identified 39.1 % (CI 36.3% to 41.9%) of the major and 25.2% (CI 23.0% to 27.4%) of the minor errors. The 67 who recommended revision identified 29.6% (CI 26.1% to 33.1%) of the major and 22.0% (CI 19.3% to 24.8%) of the minor errors. The number of errors identified differed significantly across recommended disposition. Sixty-eight percent of the reviewers did not realize that the conclusions of the work were not supported by the results. CONCLUSION: These data suggest that the use of a preconceived manuscript into which purposeful errors are placed may be a viable approach to evaluate reviewer performance. Peer reviewers in this study failed to identify two thirds of the major errors in such a manuscript.

Reliability of editors' subjective quality ratings of peer reviews of manuscripts.

CONTEXT: Quality of reviewers is crucial to journal quality, but there are usually too many for editors to know them all personally. A reliable method of rating them (for education and monitoring) is needed. OBJECTIVE: Whether editors' quality ratings of peer reviewers are reliable and how they compare with other performance measures. DESIGN: A 3.5-year prospective observational study. SETTING: Peer-reviewed journal. PARTICIPANTS: All editors and peer reviewers who reviewed at least 3 manuscripts. MAIN OUTCOME MEASURES: Reviewer quality ratings, individual reviewer rate of recommendation for acceptance, congruence between reviewer recommendation and editorial decision (decision congruence), and accuracy in reporting flaws in a masked test manuscript. INTERVENTIONS: Editors rated the quality of each review on a subjective 1 to 5 scale. RESULTS: A total of 4161 reviews of 973 manuscripts by 395 reviewers were studied. The within-reviewer intraclass correlation was 0.44 (P<.001), indicating that 20% of the variance seen in the review ratings was attributable to the reviewer. Intraclass correlations for editor and manuscript were only 0.24 and 0.12, respectively. Reviewer average quality ratings correlated poorly with the rate of recommendation for acceptance (R=-0.34) and congruence with editorial decision (R=0.26). Among 124 reviewers of the fictitious manuscript, the mean quality rating for each reviewer was modestly correlated with the number of flaws they reported (R=0.53). Highly rated reviewers reported twice as many flaws as poorly rated reviewers. CONCLUSIONS: Subjective editor ratings of individual reviewers were moderately reliable and correlated with reviewer ability to report manuscript flaws. Individual reviewer rate of recommendation for acceptance and decision congruence might be thought to be markers of a discriminating (ie, high-quality) reviewer, but these variables were poorly correlated with editors' ratings of review quality or the reviewer's ability to detect flaws in a fictitious manuscript. Therefore, they cannot be substituted for actual quality ratings by editors.

Prospective validation of artificial neural network trained to identify acute myocardial infarction.

BACKGROUND: Artificial neural networks apply non-linear statistics to pattern recognition problems. One such problem is acute myocardial infarction (AMI), a diagnosis which, in a patient presenting as an emergency, can be difficult to confirm. We report here a prospective comparison of the diagnostic accuracy of a network and that of physicians, on the same patients with suspected AMI. METHODS: Emergency department physicians who evaluated 1070 patients 18 years or older presenting to the emergency department of a teaching hospital in California, USA with anterior chest pain indicated whether they thought these patients had sustained a myocardial infarction. The network analysed the patient data collected by the physicians during their evaluations and also generated a diagnosis. FINDINGS: The physicians had a diagnostic sensitivity and specificity for myocardial infarction of 73.3% (95% confidence interval 63.3-83.3%) and 81.1% (78.7-83.5%), respectively, while the network had a diagnostic sensitivity and specificity of 96.0% (91.2-100%) and 96.0% (94.8-97.2%), respectively. Only 7% of patients had had an AMI, a low frequency but typical for anterior chest pain. INTERPRETATION: The application of non-linear neural computational analysis via an artificial neural network to the clinical diagnosis of myocardial infarction appears to have significant potential.

A neural network trained to identify the presence of myocardial infarction bases some decisions on clinical associations that differ from accepted clinical teaching.

An artificial neural network trained to identify the presence of myocardial infarction has been shown to function with a high degree of accuracy. The effects on network diagnosis of some of the clinical input variables used by this network have previously been shown to be distributed over two distinct maxima. Analysis of the basis for this distribution by studying the specific patterns in which these variables had significantly different impacts on network diagnosis revealed that the differential impacts were due to the contexts in which the variables whose effects were bimodally distributed were placed. These contexts were defined by the values of the other input data used by the network. In a number of instances, the clinical relationships implied by these associations were divergent from prior knowledge about factors predictive of myocardial infarction. One implication of these findings is that this network, which has been shown to perform with a high degree of diagnostic accuracy, may be doing so by identifying relationships between inputted information that are divergent from accepted teaching.

Complexity, chaos and human physiology: the justification for non-linear neural computational analysis.

Background is presented to suggest that a great many biologic processes are chaotic. It is well known that chaotic processes can be accurately characterized by non-linear technologies. Evidence is presented that an artificial neural network, which is a known method for the application of non-linear statistics, is able to perform more accurately in identifying patients with and without myocardial infarction than either physicians or other computer paradigms. It is suggested that the improved performance may be due to the network's better ability to characterize what is a chaotic process imbedded in the problem of the clinical diagnosis of this entity.

Analysis of the clinical variables driving decision in an artificial neural network trained to identify the presence of myocardial infarction.

STUDY OBJECTIVE: To determine which clinical variables drive the output of an artificial neural network trained to identify the presence of myocardial infarction. DESIGN: Partial output analysis. SETTING: Tertiary university teaching center. PARTICIPANTS: Seven hundred six patients more than 18 years old presenting with anterior chest pain. MEASUREMENTS: Differential network output analysis. MAIN RESULTS: A methodology was developed as the first step in measuring the impact input clinical variables have on the output (diagnosis) of an artificial neural network trained to identify the presence of acute myocardial infarction. The methodology revealed that the network used the presence of ECG findings, as well as the presence of rales, syncope, jugular venous distension, response to trinitroglycerin, and nausea and vomiting, as major predictive sources. Although this first-step analysis studied individual variables, it must be stated that the network comes to clinical closure based on the settings of all variables in a pattern and that the impact of a single variable cannot be taken out of the context of a pattern. CONCLUSION: An artificial neural network trained to recognize the presence of myocardial infarction appears to place diagnostic importance on clinical variables that have not been shown previously to be highly predictive for infarction.

Use of an artificial neural network for the diagnosis of myocardial infarction.

OBJECTIVE: To validate prospectively the use of an artificial neural network to identify myocardial infarction in patients presenting to an emergency department with anterior chest pain. DESIGN: Prospective, blinded testing. SETTING: Tertiary university teaching center. PATIENTS: A total of 331 consecutive adult patients presenting with anterior chest pain. MEASUREMENTS: Diagnostic sensitivity and specificity with regard to the diagnosis of acute myocardial infarction. MAIN RESULTS: An artificial neural network was trained on clinical pattern sets retrospectively derived from the cases of 351 patients hospitalized with a high likelihood of having myocardial infarction. It was prospectively tested on 331 consecutive patients presenting to an emergency department with anterior chest pain. The ability of the network to distinguish patients with from those without acute myocardial infarction was compared with that of physicians caring for the same patients. The physicians had a diagnostic sensitivity of 77.7% (95% CI, 77.0% to 82.9%) and a diagnostic specificity of 84.7% (CI, 84.0% to 86.4%). The artificial neural network had a sensitivity of 97.2% (CI, 97.2% to 97.5%; P = 0.033) and a specificity of 96.2% (CI, 96.2% to 96.4%; P less than 0.001). CONCLUSION: An artificial neural network trained to identify myocardial infarction in adult patients presenting to an emergency department may be a valuable aid to the clinical diagnosis of myocardial infarction; however, this possibility must be confirmed through prospective testing on a larger patient sample.