RESUMEN
OBJECTIVES: The work reported here focuses on developing novel techniques which enable an expert to detect inconsistencies in 2 (or more) perspectives that the expert might have on the same (classification) task. The high level task which the experts (physicians) had set themselves was to classify, on a 5-point severity scale (A-E), the hourly reports produced by an intensive care unit's patient management system. METHOD: The INSIGHT system has been developed to support domain experts exploring, and removing inconsistencies in their conceptualization of a task. We report here a study of intensive care physicians reconciling 2 perspectives on their patients. The 2 perspectives provided to INSIGHT were an annotated set of patient records where the expert had selected the appropriate category to describe that snapshot of the patient, and a set of rules which are able to classify the various time points on the same 5-point scale. Inconsistencies between these 2 perspectives are displayed as a confusion matrix; moreover INSIGHT then allows the expert to revise both the annotated datasets (correcting data errors, or changing the assigned categories) and the actual rule-set. RESULTS: Each of the 3 experts achieved a very high degree of consensus (~97%) between his refined knowledge sources (i.e., annotated hourly patient records and the rule-set). We then had the experts produce a common rule-set and then refine their several sets of annotations against it; this again resulted in inter-expert agreements of ~97%. The resulting rule-set can then be used in applications with considerable confidence. CONCLUSION: This study has shown that under some circumstances, it is possible for domain experts to achieve a high degree of correlation between 2 perspectives of the same task. The experts agreed that the immediate feedback provided by INSIGHT was a significant contribution to this successful outcome.
