Screening for amblyopia in preverbal children with photoscreening photographs: IV. Interobserver variability in photograph grading: origin and method of reduction.

BACKGROUND: Earlier detection of childhood vision disorders is a prominent goal of vision screening. The Medical Technology and Innovation (MTI) PhotoScreener addresses this objective. Use of this camera does not require verbal feedback and may be administered early in a child's development. Decreasing the variability in photograph grading results will boost the utility of any photoscreening system. This report aims to understand and to decrease intra- and inter-observer variability in grading photoscreening photographs. METHODS: We dissected the photograph grading process and quantified the intra- and inter-observer agreement using intraclass plot correlation coefficients. We evaluated the outcome of a two grader verification system vs. adjudicated measurements with Receiver Operator Characteristic (ROC) curves. PARTICIPANTS: Data on 955 children under 5 years of age, normal except for refractive error, each with complete photoscreening and eye examination data, culled from two previous studies. MAIN OUTCOME MEASURES: Intra- and inter-observer agreement in measuring bright crescent dimensions and pupillary diameters. Sensitivity and specificity of detection of hyperopia. RESULTS: Measurements of bright crescents are associated with greater variability than are measurements of pupillary diameters. Recognition and omission of light "rim" measurements from photograph grading will result in superior inter-observer agreement. Photograph independent errors increase variability and may be corrected by remeasurement. A verification system in which the most discrepant 5% of measurements are redone results in ROC curves similar to adjudicated dimension. CONCLUSIONS: We conclude: 1) two novices grading photographs can do as well as one expert in most cases; 2) the proposed grading methodology has undergone statistical validation and can be used in other areas of ophthalmology and medicine; and 3) inter-observer variability, one of the limitations of photoscreening photograph grading, can be reduced. For 95% of the photographs, two novices achieve similar true positive and true negative values with or without adjudication.