Systematically assessing the quality of dental electronic health record data for an investigation into oral health care disparities.

OBJECTIVES: This work describes the process by which the quality of electronic health care data for a public health study was determined. The objectives were to adapt, develop, and implement data quality assessments (DQAs) based on the National Institutes of Health Pragmatic Trials Collaboratory (NIHPTC) data quality framework within the three domains of completeness, accuracy, and consistency, for an investigation into oral health care disparities of a preventive care program. METHODS: Electronic health record data for eligible children in a dental accountable care organization of 30 offices, in Oregon, were extracted iteratively from January 1, 2014, through March 31, 2022. Baseline eligibility criteria included: children ages 0-18 with a baseline examination, Oregon home address, and either Medicaid or commercial dental benefits at least once between 2014 and 2108. Using the NIHPTC framework as a guide, DQAs were conducted throughout data element identification, extraction, staging, profiling, review, and documentation. RESULTS: The data set included 91,487 subjects, 11 data tables comprising 75 data variables (columns), with a total of 6,861,525 data elements. Data completeness was 97.2%, the accuracy of EHR data elements in extracts was 100%, and consistency between offices was strong; 29 of 30 offices within 2 standard deviations of the mean (s = 94%). CONCLUSIONS: The NIHPTC framework proved to be a useful approach, to identify, document, and characterize the dataset. The concepts of completeness, accuracy, and consistency were adapted by the multidisciplinary research team and the overall quality of the data are demonstrated to be of high quality.

Improvements in appropriate placement of dental sealants after implementation of a clinical decision support system.

BACKGROUND: Dental sealants are effective for the prevention of caries in children at elevated risk levels, and increasing the proportion of children and adolescents who have dental sealants on 1 or more molars is a Healthy People 2030 objective. Electronic health record (EHR)-based clinical decision support systems (CDSSs) have the ability to improve patient care. A dental quality measure related to dental sealant placement for children at elevated risk of caries was targeted for improvement using a CDSS. METHODS: A validated dental quality measure was adapted to assess a patient's need for dental sealant placement. A CDSS was implemented to advise care team members whether a child was at elevated risk of developing caries and had sealant-eligible first or second molars. Data on dental sealant placement at examination visits during a 5-year period were analyzed, including 32 months before CDSS implementation and 28 months after CDSS implementation. RESULTS: From January 1, 2018, through December 31, 2022, the authors assessed 59,047 examination visits for children at elevated risk of developing caries and with sealant-eligible teeth. With the implementation of a CDSS and training to support the clinical care team members in September 2020, the appropriate placement of dental sealants at examination visits increased from 27% through 60% (P < .00001). CONCLUSIONS: Integration of a CDSS into the EHR as part of a quality improvement program was effective in increasing the delivery of sealants in eligible first and second molars of children aged 5 through 15 years and considered at high risk of developing caries. PRACTICAL IMPLICATIONS: An EHR-based CDSS can be implemented to improve standardization and provide timely and appropriate patient care in dental practices.

Development and validation of a rule-based algorithm to identify periodontal diagnosis using structured electronic health record data.

AIM: To develop and validate an automated electronic health record (EHR)-based algorithm to suggest a periodontal diagnosis based on the 2017 World Workshop on the Classification of Periodontal Diseases and Conditions. MATERIALS AND METHODS: Using material published from the 2017 World Workshop, a tool was iteratively developed to suggest a periodontal diagnosis based on clinical data within the EHR. Pertinent clinical data included clinical attachment level (CAL), gingival margin to cemento-enamel junction distance, probing depth, furcation involvement (if present) and mobility. Chart reviews were conducted to confirm the algorithm's ability to accurately extract clinical data from the EHR, and then to test its ability to suggest an accurate diagnosis. Subsequently, refinements were made to address limitations of the data and specific clinical situations. Each refinement was evaluated through chart reviews by expert periodontists at the study sites. RESULTS: Three-hundred and twenty-three charts were manually reviewed, and a periodontal diagnosis (healthy, gingivitis or periodontitis including stage and grade) was made by expert periodontists for each case. After developing the initial version of the algorithm using the unmodified 2017 World Workshop criteria, accuracy was 71.8% for stage alone and 64.7% for stage and grade. Subsequently, 16 modifications to the algorithm were proposed and 14 were accepted. This refined version of the algorithm had 79.6% accuracy for stage alone and 68.8% for stage and grade together. CONCLUSIONS: Our findings suggest that a rule-based algorithm for suggesting a periodontal diagnosis using EHR data can be implemented with moderate accuracy in support of chairside clinical diagnostic decision making, especially for inexperienced clinicians. Grey-zone cases still exist, where clinical judgement will be required. Future applications of similar algorithms with improved performance will depend upon the quality (completeness/accuracy) of EHR data.

Development of quality measures to assess tooth decay outcomes from electronic health record data.

OBJECTIVES: To develop outcomes of care quality measures derived from the dental electronic health record (EHR) to assess the occurrence and timely treatment of tooth decay. METHODS: Quality measures were developed to assess whether decay was treated within 6 months and if new decay occurred in patients seen. Using EHR-derived data of the state of each tooth surface, algorithms compared the patient's teeth at different dates to determine if decay was treated or new decay had occurred. Manual chart reviews were conducted at three sites to validate the measures. The measures were implemented and scores were calculated for three sites over four calendar years, 2016 through 2019. RESULTS: About 954 charts were manually reviewed for the timely treatment of tooth decay measure, with measure performance of sensitivity 97%, specificity 85%, positive predictive value (PPV) 91%, negative predictive value (NPV) 95%. About 739 charts were reviewed for new decay measure, with sensitivity 94%, specificity 99%, PPV 99%, and NPV 94%. Across all sites and years, 52.8% of patients with decay were fully treated within 6 months of diagnosis (n = 247,959). A total of 23.8% of patients experienced new decay, measured at an annual exam (n = 640,004). CONCLUSION: Methods were developed and validated for assessing timely treatment of decay and occurrence of new decay derived from EHR data, creating effective outcome measures. These EHR-based quality measures produce accurate and reliable results that support efforts and advancement in quality assessment, quality improvement, patient care and research.

Use GPT-J Prompt Generation with RoBERTa for NER Models on Diagnosis Extraction of Periodontal Diagnosis from Electronic Dental Records.

This study explored the usability of prompt generation on named entity recognition (NER) tasks and the performance in different settings of the prompt. The prompt generation by GPT-J models was utilized to directly test the gold standard as well as to generate the seed and further fed to the RoBERTa model with the spaCy package. In the direct test, a lower ratio of negative examples with higher numbers of examples in prompt achieved the best results with a F1 score of 0.72. The performance revealed consistency, 0.92-0.97 in the F1 score, in all settings after training with the RoBERTa model. The study highlighted the importance of seed quality rather than quantity in feeding NER models. This research reports on an efficient and accurate way to mine clinical notes for periodontal diagnoses, allowing researchers to easily and quickly build a NER model with the prompt generation approach.

Developing Automated Computer Algorithms to Phenotype Periodontal Disease Diagnoses in Electronic Dental Records.

OBJECTIVE: Our objective was to phenotype periodontal disease (PD) diagnoses from three different sections (diagnosis codes, clinical notes, and periodontal charting) of the electronic dental records (EDR) by developing two automated computer algorithms. METHODS: We conducted a retrospective study using EDR data of patients (n = 27,138) who received care at Temple University Maurice H. Kornberg School of Dentistry from January 1, 2017 to August 31, 2021. We determined the completeness of patient demographics, periodontal charting, and PD diagnoses information in the EDR. Next, we developed two automated computer algorithms to automatically diagnose patients' PD statuses from clinical notes and periodontal charting data. Last, we phenotyped PD diagnoses using automated computer algorithms and reported the improved completeness of diagnosis. RESULTS: The completeness of PD diagnosis from the EDR was as follows: periodontal diagnosis codes 36% (n = 9,834), diagnoses in clinical notes 18% (n = 4,867), and charting information 80% (n = 21,710). After phenotyping, the completeness of PD diagnoses improved to 100%. Eleven percent of patients had healthy periodontium, 43% were with gingivitis, 3% with stage I, 36% with stage II, and 7% with stage III/IV periodontitis. CONCLUSIONS: We successfully developed, tested, and deployed two automated algorithms on big EDR datasets to improve the completeness of PD diagnoses. After phenotyping, EDR provided 100% completeness of PD diagnoses of 27,138 unique patients for research purposes. This approach is recommended for use in other large databases for the evaluation of their EDR data quality and for phenotyping PD diagnoses and other relevant variables.

Translating periodontal data to knowledge in a learning health system.

BACKGROUND: A learning health system (LHS) is a health system in which patients and clinicians work together to choose care on the basis of best evidence and to drive discovery as a natural outgrowth of every clinical encounter to ensure the right care at the right time. An LHS for dentistry is now feasible, as an increased number of oral health care encounters are captured in electronic health records (EHRs). METHODS: The authors used EHRs data to track periodontal health outcomes at 3 large dental institutions. The 2 outcomes of interest were a new periodontitis case (for patients who had not received a diagnosis of periodontitis previously) and tooth loss due to progression of periodontal disease. RESULTS: The authors assessed a total of 494,272 examinations (new periodontitis outcome: n = 168,442; new tooth loss outcome: n = 325,830), representing a total of 194,984 patients. Dynamic dashboards displaying performance on both measures over time allow users to compare demographic and risk factors for patients. The incidence of new periodontitis and tooth loss was 4.3% and 1.2%, respectively. CONCLUSIONS: Periodontal disease, diagnosis, prevention, and treatment are particularly well suited for an LHS model. The results showed the feasibility of automated extraction and interpretation of critical data elements from the EHRs. The 2 outcome measures are being implemented as part of a dental LHS. The authors are using this knowledge to target the main drivers of poorer periodontal outcomes in a specific patient population, and they continue to use clinical health data for the purpose of learning and improvement. PRACTICAL IMPLICATIONS: Dental institutions of any size can conduct contemporaneous self-evaluation and immediately implement targeted strategies to improve oral health outcomes.

A generative adversarial inpainting network to enhance prediction of periodontal clinical attachment level.

OBJECTIVES: Bone level as measured by clinical attachment levels (CAL) are critical findings that determine the diagnosis of periodontal disease. Deep learning algorithms are being used to determine CAL which aid in the diagnosis of periodontal disease. However, the limited field-of-view of bitewing x-rays poses a challenge for convolutional neural networks (CNN) because out-of-view anatomy cannot be directly considered. This study presents an inpainting algorithm using generative adversarial networks (GANs) coupled with partial convolutions to predict out-of-view anatomy to enhance CAL prediction accuracy. METHODS: Retrospective purposive sampling of cases with healthy periodontium and diseased periodontium with bitewing and periapical radiographs and clinician recorded CAL were utilized. Data utilized was from July 1, 2016 through January 30, 2020. 80,326 images were used for training, 12,901 images were used for validation and 10,687 images were used to compare non-inpainted methods to inpainted methods for CAL predictions. Statistical analyses were mean bias error (MBE), mean absolute error (MAE) and Dunn's pairwise test comparing CAL at p=0.05. RESULTS: Comparator p-values demonstrated statistically significant improvement in CAL prediction accuracy between corresponding inpainted and non-inpainted methods with MAE of 1.04mm and 1.50mm respectively. The Dunn's pairwise test indicated statistically significant improvement in CAL prediction accuracy between inpainted methods compared to their non-inpainted counterparts, with the best performing methods achieving a Dunn's pairwise value of -63.89. CONCLUSIONS: This study demonstrates the superiority of using a generative adversarial inpainting network with partial convolutions to predict CAL from bitewing and periapical images. CLINICAL SIGNIFICANCE: Artificial intelligence was developed and utilized to predict clinical attachment level compared to clinical measurements. A generative adversarial inpainting network with partial convolutions was developed, tested and validated to predict clinical attachment level. The inpainting approach was found to be superior to non-inpainted methods and within the 1mm clinician-determined measurement standard.

Caries Risk Documentation And Prevention: eMeasures For Dental Electronic Health Records.

BACKGROUND: Longitudinal patient level data available in the electronic health record (EHR) allows for the development, implementation, and validations of dental quality measures (eMeasures). OBJECTIVE: We report the feasibility and validity of implementing two eMeasures. The eMeasures determined the proportion of patients receiving a caries risk assessment (eCRA) and corresponding appropriate risk-based preventative treatments for patients at elevated risk of caries (appropriateness of care [eAoC]) in two academic institutions and one accountable care organization, in the 2019 reporting year. METHODS: Both eMeasures define the numerator and denominator beginning at the patient level, populations' specifications, and validated the automated queries. For eCRA, patients who completed a comprehensive or periodic oral evaluation formed the denominator, and patients of any age who received a CRA formed the numerator. The eAoC evaluated the proportion of patients at elevated caries risk who received the corresponding appropriate risk-based preventative treatments. RESULTS: EHR automated queries identified in three sites 269,536 patients who met the inclusion criteria for receiving a CRA. The overall proportion of patients who received a CRA was 94.4% (eCRA). In eAoC, patients at elevated caries risk levels (moderate, high, or extreme) received fluoride preventive treatment ranging from 56 to 93.8%. For patients at high and extreme risk, antimicrobials were prescribed more frequently site 3 (80.6%) than sites 2 (16.7%) and 1 (2.9%). CONCLUSION: Patient-level data available in the EHRs can be used to implement process-of-care dental eCRA and AoC, eAoC measures identify gaps in clinical practice. EHR-based measures can be useful in improving delivery of evidence-based preventative treatments to reduce risk, prevent tooth decay, and improve oral health.

Assessing the completeness of periodontal disease documentation in the EHR: a first step in measuring the quality of care.

BACKGROUND: Our objective was to measure the proportion of patients for which comprehensive periodontal charting, periodontal disease risk factors (diabetes status, tobacco use, and oral home care compliance), and periodontal diagnoses were documented in the electronic health record (EHR). We developed an EHR-based quality measure to assess how well four dental institutions documented periodontal disease-related information. An automated database script was developed and implemented in the EHR at each institution. The measure was validated by comparing the findings from the measure with a manual review of charts. RESULTS: The overall measure scores varied significantly across the four institutions (institution 1 = 20.47%, institution 2 = 0.97%, institution 3 = 22.27% institution 4 = 99.49%, p-value < 0.0001). The largest gaps in documentation were related to periodontal diagnoses and capturing oral homecare compliance. A random sample of 1224 charts were manually reviewed and showed excellent validity when compared with the data generated from the EHR-based measure (Sensitivity, Specificity, PPV, and NPV > 80%). CONCLUSION: Our results demonstrate the feasibility of developing automated data extraction scripts using structured data from EHRs, and successfully implementing these to identify and measure the periodontal documentation completeness within and across different dental institutions.

An innovative approach to assess comprehensive care: Temple University's patient wellness report.

PURPOSE: The transition from a paper-based to an all-electronic patient health record took a major step forward in 2018, with the implementation of an electronic health record that supports the School's patient-centered comprehensive care model and facilitates outcomes assessment. The Patient Wellness Report (PWR) summarizes findings of the patient assessment, and it does so automatically by locating data already entered in axiUm forms. This study aimed to describe the PWR implementation procedures and to examine outcomes and characteristics among patients with completed treatment plans during an 18-month period. METHODS: Outcome data were extracted from axiUm for patients aged ≥16 years who completed comprehensive care treatment plans. Each PWR contained 14 metrics related to "dimensions" of wellness (quality of life, general health factors, oral hygiene, caries risk, and degree of periodontal inflammation and pocketing), each of which is rated on a 3-point scale based on best available scientific evidence. RESULTS: A total of 2074 patients completed planned procedures between July 2018 and January 2020, and met the study eligibility criteria. Improvement of several conditions was observed between baseline and follow-up in caries lesions (21%), blood pressure (9%), and periodontal pocket (3.2%). A majority of patients rated in good condition at baseline had their scores unchanged at follow-up in the following areas: dental anxiety (92%), speaking (88%), smoking (87%), and alcohol consumption (79%). CONCLUSION: Improvements in dental caries and blood pressure metrics were easily monitored using the PWR. In addition, disparities exist in improvement of patient outcomes by race/ethnicity.

Tracking oral health in a standardized, evidence-based, prevention-focused dental care system.

OBJECTIVES: Learning health-care systems are foundational for measuring and achieving value in oral health care. This article describes the components of a preventive dental care program and the quality of care in a large dental accountable care organization. METHODS: A retrospective study design describes and evaluates the cross-sectional measures of process of care (PoC), appropriateness of care (AoC), and outcomes of care (OoC) extracted from the electronic health record (EHR), between 2014 and 2019. Annual and composite measures are derived from EHR-based clinical decision support for risk determination, diagnostic and treatment terminology, and decayed-missing-filled-teeth (DMFT) measures. RESULTS: Annually, 253,515 ± 27,850 patients were cared for with 618,084 ± 80,559 visits, 209,366 ± 22,300 exams, and 2,072,844 ± 300,363 clinical procedures. PoC metrics included provider adherence (98.3 percent) in completing caries risk assessments and patient receipt (96.9 percent) of a proactive dental care plan. AoC metrics included patients receiving prevention according to the risk-based protocol. The percent of patients at risk for caries receiving fluoride varnish was 95.4 ± 0.4 percent. OoC metrics included untreated decay and new decay. The 6-year average prevalence of untreated decay was 11.3 ± 0.3 percent, and average incidence of new decay was 13.6 ± 0.5 percent, increasing with risk level: low = 7.5 percent, medium = 18.8 percent, high = 29.4 percent, and extreme = 28.1 percent. CONCLUSIONS: The preventive dental care system demonstrates excellent provider adherence to the evidence-based prevention protocol, with measurably better dental outcomes by patient risk compared to national estimates. These achievements are enabled by a value-centric, accountable model of care and incentivized by a compensation model aligned with performance measures.

Increasing value, reducing waste: tailoring the application of dental sealants according to individual caries risk.

OBJECTIVES: Despite a significant national investment in oral health, there is little understanding of the return in terms of quality. Value-based payments aim to refocus provider reimbursement based on the value created to the patient. Our objectives were to apply a set of dental quality measures to help determine the value of preventive dental care provided to children at two academic dental school clinics. METHODS: We queried the institutional electronic health records (EHRs) for patients between the ages of 6-14 years with sealable first or second permanent molars, determined caries risk status, identified if dental sealants were placed, and finally if the teeth showed evidence of new caries experience. In order to determine the cost-effectiveness of EHR-based triage of applying dental sealants, we calculated the incremental cost-effectiveness ratio (ICER) for the dental quality measures supported sealing program. RESULTS: Between the two academic sites, there were 6,155 unique children for a total of 12,302 eligible teeth without a sealant and 32,811 eligible teeth with a sealant. Teeth without a sealant were more likely to have decay (4.8 percent) than those with a sealant (1.7 percent). At both sites, patients with high caries risk were more likely to benefit from sealants compared to those patients with low risk. CONCLUSION: Implementation of caries risk stratified fissure sealant quality measures demonstrates the potential for extracting better value in oral health care.

Developing and Testing Electronic Health Record-Derived Caries Indices.

Caries indices, the basis of epidemiologic caries measures, are not easily obtained in clinical settings. This study's objective was to design, test, and validate an automated program (Valid Electronic Health Record Dental Caries Indices Calculator Tool [VERDICT]) to calculate caries indices from an electronic health record (EHR). Synthetic use case scenarios and actual patient cases of primary, mixed, and permanent dentition, including decayed, missing, and filled teeth (DMFT/dmft) and tooth surfaces (DMFS/dmfs) were entered into the EHR. VERDICT measures were compared to a previously validated clinical electronic data capture (EDC) system and statistical program to calculate caries indices. Four university clinician-researchers abstracted EHR caries exam data for 45 synthetic use cases into the EDC and post-processed with SAS software creating a gold standard to compare the -VERDICT-derived caries indices. Then, 2 senior researchers abstracted EHR caries exam data and calculated caries indices for 24 patients, allowing further comparisons to VERDICT indices. Agreement statistics were computed among abstractors, and discrepancies were resolved by consensus. Agreement statistics between the 2 final-phase abstractors and the VERDICT measures showed extremely high concordance: Lin's concordance coefficients (LCCs) >0.99 for dmfs, dmft, DS, ds, DT, dt, ms, mt, FS, fs, FT, and ft; LCCs >0.95 for DMFS and DMFT; and LCCs of 0.92-0.93 for MS and MT. Caries indices, essential to developing primary health outcome measures for research, can be reliably derived from an EHR using VERDICT. Using these indices will enable population oral health management approaches and inform quality improvement efforts.

Using a stakeholder-engaged approach to develop and validate electronic clinical quality measures.

OBJECTIVE: To describe the stakeholder-engaged processes used to develop, specify, and validate 2 oral health care electronic clinical quality measures. MATERIALS AND METHODS: A broad range of stakeholders were engaged from conception through testing to develop measures and test feasibility, reliability, and validity following National Quality Forum guidance. We assessed data element feasibility through semistructured interviews with key stakeholders using a National Quality Forum-recommended scorecard. We created test datasets of synthetic patients to test measure implementation feasibility and reliability within and across electronic health record (EHR) systems. We validated implementation with automated reporting of EHR clinical data against manual record reviews, using the kappa statistic. RESULTS: A stakeholder workgroup was formed and guided all development and testing processes. All critical data elements passed feasibility testing. Four test datasets, representing 577 synthetic patients, were developed and implemented within EHR vendors' software, demonstrating measure implementation feasibility. Measure reliability and validity were established through implementation at clinical practice sites, with kappa statistic values in the "almost perfect" agreement range of 0.80-0.99 for all but 1 measure component, which demonstrated "substantial" agreement. The 2 validated measures were published in the United States Health Information Knowledgebase. CONCLUSION: The stakeholder-engaged processes used in this study facilitated a successful measure development and testing cycle. Engaging stakeholders early and throughout development and testing promotes early identification of and attention to potential threats to feasibility, reliability, and validity, thereby averting significant resource investments that are unlikely to be fruitful.