Pediatric Gastroenterology, Hepatology and Nutrition Quality Assessment and Improvement Project: Reliability and Implementation of a Standardized Scoring Tool for Outpatient Assessment & Plan Documentation

ABSTRACT

Background: Clinical documentation is a means to document care and supports several important areas including inter-provider and patient communication, reimbursement, patient safety, and overall patient care. However, a significant knowledge gap exists with respect to the evaluation and potential improvement of the quality of outpatient clinical documentation. This may be related in part to a lack of standardized tools or metrics to assess clinical documentation. Aim(s) 1) To develop a reliable outpatient assessment & plan clinical documentation scoring tool;and 2) To assess its impact on improving clinical outpatient documentation quality amongst pediatric gastroenterologists. Method(s) Development of a clinical documentation scoring tool A tool was developed to score the assessment & plan sections of outpatient clinical notes for both clarity and soundness (Table). It was implemented in 2020, and after two rounds of division-wide scoring, a formal IRB-approved investigation of scale reliability was undertaken. An additional round of scoring took place in 2021. The implementation of telemedicine in response to the COVID-19 pandemic permitted subanalyses of telemedicine vs in-person clinical documentation. QI Project Assessment & plan documentation of new outpatient visits within a single center were extracted between 2020 and 2021 (18 months). Ten notes were compiled from each clinician, and the chief complaint was used as a guide to increase variability. All identifiers were removed. Using the tool, division clinicians then scored 10 clinic notes from other members of the division. Following each cycle, peer scores were provided back to each clinician. In addition, anonymized scores were shared amongst the division's clinical providers. Scales Reliability Assessment After the first two rounds of the QI project, IRB approval was secured for a separate study. Division clinicians used the developed tool to rate the clarity and soundness of assessment and plan documentation from a set of 10 examples (Set 1). The same providers were asked to complete a second set of 10 examples (Set 2) several months later. Both sets were comprised of 5 overlapping cases to calculate the inter-rater and intra-rater intra-class correlation coefficient (ICC) for both scales. Result(s) Scale Reliability Analyses The above protocol created 3 different scoring sets for calculating the inter-rater ICC of the clarity and soundness scales Set 1, Set 2, and Combined Set 1 & 2. The Clarity Scale inter-rater ICC for these sets were 0.71 (N=9), 0.51 (N=11), and 0.51 (N=8), respectively. The Soundness Scale inter-rater ICC for these scoring sets were 0.51 (N=9), 0.31 (N=10), and 0.32 (N=8), respectively. The Clarity Scale intra-rater ICC was 0.77 and 0.55 for the Soundness Scale (N=8 for both analyses). Quality Improvement We found a trend toward increasing clarity and soundness scores with each subsequent round of scoring (Figure). The number of evaluations which scored less than a 3 on the clarity scale decreased from (52/407) 12.8% to (34/394) 8.6% and finally (36/433) 8.3%. Of note, scoring of clarity and soundness in notes derived from telemedicine vs in-person visits were similar. Conclusion(s) There is great interest in QI in Pediatric Gastroenterology, although it is difficult to find universally applicable targets. The assessment & plan sections of the outpatient note is of central importance and has the potential to be a useful area for QI. Scoring the clarity of notes is relatively reproducible, and this program has the potential to yield substantial improvement amongst a clinical team. Soundness of the assessment & plan documentation is more subjective and will require additional revision in order to achieve favorable reliability.