Development of a scoring system to quantify errors from semantic characteristics in incident reports.

OBJECTIVES: Incident reporting systems are widely used to identify risks and enable organisational learning. Free-text descriptions contain important information about factors associated with incidents. This study aimed to develop error scores by extracting information about the presence of error factors in incidents using an original decision-making model that partly relies on natural language processing techniques. METHODS: We retrospectively analysed free-text data from reports of incidents between January 2012 and December 2022 from Nagoya University Hospital, Japan. The sample data were randomly allocated to equal-sized training and validation datasets. We conducted morphological analysis on free text to segment terms from sentences in the training dataset. We calculated error scores for terms, individual reports and reports from staff groups according to report volume size and compared these with conventional classifications by patient safety experts. We also calculated accuracy, recall, precision and F-score values from the proposed 'report error score'. RESULTS: Overall, 114 013 reports were included. We calculated 36 131 'term error scores' from the 57 006 reports in the training dataset. There was a significant difference in error scores between reports of incidents categorised by experts as arising from errors (p<0.001, d=0.73 (large)) and other incidents. The accuracy, recall, precision and F-score values were 0.8, 0.82, 0.85 and 0.84, respectively. Group error scores were positively associated with expert ratings (correlation coefficient, 0.66; 95% CI 0.54 to 0.75, p<0.001) for all departments. CONCLUSION: Our error scoring system could provide insights to improve patient safety using aggregated incident report data.

Nationwide survey on Japanese residents' experience with and barriers to incident reporting.

The ability of any incident reporting system to improve patient care is dependent upon robust reporting practices. However, under-reporting is still a problem worldwide. We aimed to reveal the barriers experienced while reporting an incident through a nationwide survey in Japan. We conducted a cross-sectional survey. All first- and second-year residents who took the General Medicine In-Training Examination (GM-ITE) from February to March 2021 in Japan were selected for the study. The voluntary questionnaire asked participants regarding the number of safety incidents encountered and reported within the previous year and the barriers to reporting incidents. Demographics were obtained from the GM-ITE. The answers of respondents who indicated they had never previously reported an incident (non-reporting group) were compared to those of respondents who had reported at least one incident in the previous year (reporting group). Of 5810 respondents, the vast majority indicated they had encountered at least one safety incident in the past year (n = 4449, 76.5%). However, only 2724 (46.9%) had submitted an incident report. Under-reporting (more safety incidents compared to the number of reports) was evident in 1523 (26.2%) respondents. The most frequently mentioned barrier to reporting an incident was the time required to file the report (n = 2622, 45.1%). The barriers to incident reporting were significantly different between resident physicians who had previously reported and those who had never previously reported an incident. Our study revealed that resident physicians in Japan commonly encounter patient safety incidents but under-report them. Numerous perceived and experienced barriers to reporting remain, which should be addressed if incident reporting systems are to have an optimal impact on improving patient safety. Incident reporting is essential for improving patient safety in an institution, and this study recommends establishing appropriate interventions according to each learner's barriers for reporting.

Development of a Novel Scoring System to Quantify the Severity of Incident Reports: An Exploratory Research Study.

Incident reporting systems have been widely adopted to collect information about patient safety incidents. Much of the value of incident reports lies in the free-text section. Computer processing of semantic information may be helpful to analyze this. We developed a novel scoring system for decision making to assess the severity of incidents using the semantic characteristics of the text in incident reports, and compared its results with experts' opinions. We retrospectively analyzed free-text data from incident reports from January 2012 to September 2021 at Nagoya University Hospital, Aichi, Japan. The sample was allocated to training and validation datasets using the hold-out method. Morphological analysis was used to segment terms in the training dataset. We calculated a severity term score, a severity report score and severity group score, by report volume size, and compared these with conventional severity classifications by patient safety experts and reporters. We allocated 96,082 incident reports into two groups. We calculated 1,802 severity term scores from the 48,041 reports in the training dataset. There was a significant difference in severity report score between reports categorized as severe and not severe by experts (95% confidence interval [CI] -0.83 to -0.80, p < 0.001, d = 0.81). Severity group scores were positively associated with severity ratings from experts and reporters (correlation coefficients 0.73 [95% CI 0.63-0.80, p < 0.001] and 0.79 [95% CI 0.71-0.85, p < 0.001]) for all departments. Our severity scoring system could therefore contribute to better organizational patient safety.