Measuring the Coverage of the HL7® FHIR® Standard in Supporting Data Acquisition for 3 Public Health Registries.

With the increasing need for timely submission of data to state and national public health registries, current manual approaches to data acquisition and submission are insufficient. In clinical practice, federal regulations are now mandating the use of data messaging standards, i.e., the Health Level Seven (HL7®) Fast Healthcare Interoperability Resources (FHIR®) standard, to facilitate the electronic exchange of clinical (patient) data. In both research and public health practice, we can also leverage FHIR® ‒ and the infrastructure already in place for supporting exchange of clinical practice data ‒ to enable seamless exchange between the electronic medical record and public health registries. That said, in order to understand the current utility of FHIR® for supporting the public health use case, we must first measure the extent to which the standard resources map to the required registry data elements. Thus, using a systematic mapping approach, we evaluated the level of completeness of the FHIR® standard to support data collection for three public health registries (Trauma, Stroke, and National Surgical Quality Improvement Program). On average, approximately 80% of data elements were available in FHIR® (71%, 77%, and 92%, respectively; inter-annotator agreement rates: 82%, 78%, and 72%, respectively). This tells us that there is the potential for significant automation to support EHR-to-Registry data exchange, which will reduce the amount of manual, error-prone processes and ensure higher data quality. Further, identification of the remaining 20% of data elements that are "not mapped" will enable us to improve the standard and develop profiles that will better fit the registry data model.

Outcomes of the UAMS summer undergraduate research program to increase diversity in research and health professions.

The University of Arkansas for Medical Sciences (UAMS) Summer Undergraduate Research Program (SURP) aims to increase diversity in research and health-related careers. The SURP provides underrepresented minority (URM) and disadvantaged students with research, mentoring, and networking experiences; real-life surgical observations; and simulated cardiovascular demonstrations. A postprogram survey was developed to assess program outcomes and explore ways of improving the program to stimulate URM and disadvantaged students' interest in research and health-related careers. This is a report of those postprogram survey findings. Using a survey research design, an online survey was emailed to participants (n = 88). Data were collected for 6 weeks beginning March 2020. There were 37 multiple-choice and open-ended questions regarding education, career choices, and program experiences. Responses were downloaded to statistical software for analyses. Quantitative data were analyzed using descriptive statistics. Major themes were identified for qualitative data. Responses were received from 44.3% (n = 39) of former SURP participants. Overall, 59% stated that the SURP influenced their career goals. When asked about mentor-mentee relationships, 69.3% responded that their interactions were excellent or good; 61.5% maintained contact with their mentor after the SURP. Finally, 79% indicated their SURP experience was excellent or good, and 84.6% would recommend the SURP to others. The SURP has been successful at providing URM and disadvantaged students with positive research experiences and long-term mentor-mentee relationships and has influenced educational and/or career goals. Programs that expose URM and disadvantaged students to basic, clinical, and/or translational research are beneficial for stimulating interest in research and health-related careers.NEW & NOTEWORTHY Mentor-mentee relationships were extremely beneficial as many of the former participants maintained contact with their summer mentor after the program ended. This assessment also revealed that exposing underrepresented and minority students to research has a long-lasting effect on career and educational goals.

Measuring and controlling medical record abstraction (MRA) error rates in an observational study.

BACKGROUND: Studies have shown that data collection by medical record abstraction (MRA) is a significant source of error in clinical research studies relying on secondary use data. Yet, the quality of data collected using MRA is seldom assessed. We employed a novel, theory-based framework for data quality assurance and quality control of MRA. The objective of this work is to determine the potential impact of formalized MRA training and continuous quality control (QC) processes on data quality over time. METHODS: We conducted a retrospective analysis of QC data collected during a cross-sectional medical record review of mother-infant dyads with Neonatal Opioid Withdrawal Syndrome. A confidence interval approach was used to calculate crude (Wald's method) and adjusted (generalized estimating equation) error rates over time. We calculated error rates using the number of errors divided by total fields ("all-field" error rate) and populated fields ("populated-field" error rate) as the denominators, to provide both an optimistic and a conservative measurement, respectively. RESULTS: On average, the ACT NOW CE Study maintained an error rate between 1% (optimistic) and 3% (conservative). Additionally, we observed a decrease of 0.51 percentage points with each additional QC Event conducted. CONCLUSIONS: Formalized MRA training and continuous QC resulted in lower error rates than have been found in previous literature and a decrease in error rates over time. This study newly demonstrates the importance of continuous process controls for MRA within the context of a multi-site clinical research study.

Summer undergraduate biomedical research program for underrepresented minority students in a rural, low-income state.

Introduction: Diversity can enhance the agenda and quality of biomedical research, but a dearth of underrepresented minorities and women serve as biomedical researchers. The study purpose was to examine the impact of the a summer undergraduate research program on self-efficacy in research, scientific communication, and leadership as well as scientific identity, valuing objectives of the scientific community, and intent to pursue a biomedical research career. Methods: Underrepresented minority and female undergraduate students participated in a mentored research experience in a rural, low-income state. Results: Students' self-efficacy in research, scientific communication, and leadership as well as scientific identity, valuing objectives of the scientific community, and intent to pursue a biomedical research career increased post-program compared to pre-program. Conclusion: This study supports implementation of a biomedical summer undergraduate research program for URM and women in a poor, rural, settings.

Experiences with developing and implementing a courageous conversations pilot classroom through synchronous meetings via zoom.

Background: The murder of George Floyd created national outcry that echoed down to national institutions, including universities and academic systems to take a hard look at systematic and systemic racism in higher education. This motivated the creation of a fear and tension-minimizing, curricular offering, "Courageous Conversations," collaboratively engaging students, staff, and faculty in matters of diversity, equity, and inclusion (DEI) in the Department of Health Outcomes and Biomedical Informatics at the University of Florida. Methods: A qualitative design was employed assessing narrative feedback from participants during the Fall semester of 2020. Additionally, the ten-factor model implementation framework was applied and assessed. Data collection included two focus groups and document analysis with member-checking. Thematic analysis (i.e., organizing, coding, synthesizing) was used to analyze a priori themes based on the four agreements of the courageous conversations framework, stay engaged, expect to experience discomfort, speak your truth, and expect and accept non-closure. Results: A total of 41 participants of which 20 (48.78%) were department staff members, 11 (26.83%) were department faculty members, and 10 (24.30%) were graduate students. The thematic analysis revealed 1) that many participants credited their learning experiences to what their peers had said about their own personal lived experiences during group sessions, and 2) several participants said they would either retake the course or recommend it to a colleague. Conclusion: With structured implementation, courageous conversations can be an effective approach to create more diverse, equitable, and inclusive spaces in training programs with similar DEI ecosystems.

Quantifying care delivery team influences on the hospitalization outcomes of patients with multimorbidity: Implications for clinical informatics.

The primary objective was to quantify the influences of care delivery teams on the outcomes of patients with multimorbidity. Electronic medical record data on 68,883 patient care encounters (i.e., 54,664 patients) were extracted from the Arkansas Clinical Data Repository. Social network analysis assessed the minimum care team size associated with improved care outcomes (i.e., hospitalizations, days between hospitalizations, and cost) of patients with multimorbidity. Binomial logistic regression further assessed the influence of the presence of seven specific clinical roles. When compared to patients without multimorbidity, patients with multimorbidity had a higher mean age (i.e., 47.49 v. 40.61), a higher mean dollar amount of cost per encounter (i.e., $3,068 v. $2,449), a higher number of hospitalizations (i.e., 25 v. 4), and a higher number of clinicians engaged in their care (i.e., 139,391 v. 7,514). Greater network density in care teams (i.e., any combination of two or more Physicians, Residents, Nurse Practitioners, Registered Nurses, or Care Managers) was associated with a 46-98% decreased odds of having a high number of hospitalizations. Greater network density (i.e., any combination of two or more Residents or Registered Nurses) was associated with 11-13% increased odds of having a high cost encounter. Greater network density was not significantly associated with having a high number of days between hospitalizations. Analyzing the social networks of care teams may fuel computational tools that better monitor and visualize real-time hospitalization risk and care cost that are germane to care delivery.

Care delivery team composition effect on hospitalization risk in African Americans with congestive heart failure.

The care delivery team (CDT) is critical to providing care access and equity to patients who are disproportionately impacted by congestive heart failure (CHF). However, the specific clinical roles that are associated with care outcomes are unknown. The objective of this study was to examine the extent to which specific clinical roles within CDTs were associated with care outcomes in African Americans (AA) with CHF. Deidentified electronic medical record data were collected on 5,962 patients, representing 80,921 care encounters with 3,284 clinicians between January 1, 2014 and December 31, 2021. Binomial logistic regression assessed associations of specific clinical roles and the Mann Whitney-U assessed racial differences in outcomes. AAs accounted for only 26% of the study population but generated 48% of total care encounters, the same percentage of care encounters generated by the largest racial group (i.e., Caucasian Americans; 69% of the study population). AAs had a significantly higher number of hospitalizations and readmissions than Caucasian Americans. However, AAs had a significantly higher number of days at home and significantly lower care charges than Caucasian Americans. Among all CHF patients, patients with a Registered Nurse on their CDT were less likely to have a hospitalization (i.e. 30%) and a high number of readmissions (i.e., 31%) during the 7-year study period. When stratified by heart failure phenotype, the most severe patients who had a Registered Nurse on their CDT were 88% less likely to have a hospitalization and 50% less likely to have a high number of readmissions. Similar decreases in the likelihood of hospitalization and readmission were also found in less severe cases of heart failure. Specific clinical roles are associated with CHF care outcomes. Consideration must be given to developing and testing the efficacy of more specialized, empirical models of CDT composition to reduce the disproportionate impact of CHF.

Comparing Medical Record Abstraction (MRA) Error Rates in an Observational Study to Pooled Rates Identified in the Data Quality Literature.

Background: Medical record abstraction (MRA) is a commonly used method for data collection in clinical research, but is prone to error, and the influence of quality control (QC) measures is seldom and inconsistently assessed during the course of a study. We employed a novel, standardized MRA-QC framework as part of an ongoing observational study in an effort to control MRA error rates. In order to assess the effectiveness of our framework, we compared our error rates against traditional MRA studies that had not reported using formalized MRA-QC methods. Thus, the objective of this study was to compare the MRA error rates derived from the literature with the error rates found in a study using MRA as the sole method of data collection that employed an MRA-QC framework. Methods: Using a moderator meta-analysis employed with Q-test, the MRA error rates from the meta-analysis of the literature were compared with the error rate from a recent study that implemented formalized MRA training and continuous QC processes. Results: The MRA process for data acquisition in clinical research was associated with both high and highly variable error rates (70 - 2,784 errors per 10,000 fields). Error rates for the study using our MRA-QC framework were between 1.04% (optimistic, all-field rate) and 2.57% (conservative, populated-field rate) (or 104 - 257 errors per 10,000 fields), 4.00 - 5.53 percentage points less than the observed rate from the literature (p<0.0001). Conclusions: Review of the literature indicated that the accuracy associated with MRA varied widely across studies. However, our results demonstrate that, with appropriate training and continuous QC, MRA error rates can be significantly controlled during the course of a clinical research study.

Error Rates of Data Processing Methods in Clinical Research: A Systematic Review and Meta-Analysis of Manuscripts Identified Through PubMed.

Background: In clinical research, prevention of systematic and random errors of data collected is paramount to ensuring reproducibility of trial results and the safety and efficacy of the resulting interventions. Over the last 40 years, empirical assessments of data accuracy in clinical research have been reported in the literature. Although there have been reports of data error and discrepancy rates in clinical studies, there has been little systematic synthesis of these results. Further, although notable exceptions exist, little evidence exists regarding the relative accuracy of different data processing methods. We aim to address this gap by evaluating error rates for 4 data processing methods. Methods: A systematic review of the literature identified through PubMed was performed to identify studies that evaluated the quality of data obtained through data processing methods typically used in clinical trials: medical record abstraction (MRA), optical scanning, single-data entry, and double-data entry. Quantitative information on data accuracy was abstracted from the manuscripts and pooled. Meta-analysis of single proportions based on the Freeman-Tukey transformation method and the generalized linear mixed model approach were used to derive an overall estimate of error rates across data processing methods used in each study for comparison. Results: A total of 93 papers (published from 1978 to 2008) meeting our inclusion criteria were categorized according to their data processing methods. The accuracy associated with data processing methods varied widely, with error rates ranging from 2 errors per 10,000 fields to 2,784 errors per 10,000 fields. MRA was associated with both high and highly variable error rates, having a pooled error rate of 6.57% (95% CI: 5.51, 7.72). In comparison, the pooled error rates for optical scanning, single-data entry, and double-data entry methods were 0.74% (0.21, 1.60), 0.29% (0.24, 0.35) and 0.14% (0.08, 0.20), respectively. Conclusions: Data processing and cleaning methods may explain a significant amount of the variability in data accuracy. MRA error rates, for example, were high enough to impact decisions made using the data and could necessitate increases in sample sizes to preserve statistical power. Thus, the choice of data processing methods can likely impact process capability and, ultimately, the validity of trial results.

Cultivating informatics capacity for multimorbidity: A learning health systems use case.

Background: The aim of this study was to characterize patterns of multimorbidity across patients and identify opportunities to strengthen the informatics capacity of learning health systems that are used to characterize multimorbidity across patients. Methods: Electronic health record (EHR) data on 225,710 multimorbidity patients were extracted from the Arkansas Clinical Data Repository as a use case. Hierarchical cluster analysis identified the most frequently occurring combinations of chronic conditions within the learning health system's captured data. Results: Results revealed multimorbidity was highest among patients ages 60 to 74, Caucasians, females, and Medicare payors. The largest numbers of chronic conditions occurred in the smallest numbers of patients (i.e., 70,262 (31%) patients with two conditions, two (<1%) patients with 22 chronic conditions). The results revealed urgent needs to improve EHR systems and processes that collect and manage multimorbidity data (e.g., creating new, multimorbidity-centric data elements in EHR systems, detailed longitudinal tracking of compounding disease diagnoses). Conclusions: Without additional capacity to collect and aggregate large-scale data, multimorbidity patients cannot benefit from the recent advancements in informatics (i.e., clinical data registries, emerging data standards) that are abundantly working to improve the outcomes of patients with single chronic conditions. Additionally, robust socio-technical system studies of clinical workflows are needed to assess the feasibility of integrating the collection of risk factor data elements (i.e., psycho-social, cultural, ethnic, and socioeconomic attributes of populations) into primary care encounters. These approaches to advancing learning health systems for multimorbidity could substantially reduce the constraints of current technologies, data, and data-capturing processes.

Informing training needs for the revised certified clinical data manager (CCDMTM) exam: analysis of results from the previous exam.

Objective: To inform training needs for the revised Certified Clinical Data Manager (CCDMTM) Exam. Introduction: Clinical data managers hold the responsibility for processing the data on which research conclusions and regulatory decisions are based, highlighting the importance of applying effective data management practices. The use of practice standards such as the Good Clinical Data Management Practices increases confidence in data, emphasizing that the study conclusions likely hold much more weight when utilizing standard practices. Methods: A quantitative, descriptive study, and application of classic test theory was undertaken to analyze past data from the CCDMTM Exam to identify potential training needs. Data across 952 sequential exam attempts were pooled for analysis. Results: Competency domain-level analysis identified training needs in 4 areas: design tasks; data processing tasks; programming tasks; and coordination and management tasks. Conclusions: Analysis of past CCDMTM Exam results using classic test theory identified training needs reflective of exam takers. Training in the identified areas could benefit CCDMTM Exam takers and improve their ability to apply effective data management practices. While this may not be reflective of individual or organizational needs, recommendations for assessing individual and organizational training needs are provided.

Performance measures of racially underrepresented Ph.D. students in biomedical sciences: The UAMS IMSD Program Outcomes.

The purpose of this study was to identify performance measures of racially underrepresented minority (RUM) Ph.D. trainees who needed additional training initiatives to assist with completing the UAMS biomedical science degree. A sample of 37 trainees in the 10-year NIH-NIGMS funded Initiative for Maximizing Student Development (IMSD) program at the University of Arkansas for Medical Sciences (UAMS) were examined. Descriptive statistics and correlations examined process measures (GRE scores, GPAs, etc.) and outcome measures (time-to-degree, publications, post-doctoral fellowship, etc.) While differences were found, there were no statistically significant differences between how these two groups (Historically Black Colleges and Universities (HBCUs) and Predominately White Institutions (PWIs)) of students performed over time as Ph.D. students. Graduates who scored lower on the verbal section of the GRE also had a higher final graduate school grade point average in graduates who received their undergraduate training from HBCUs. Of the graduates who received their undergraduate training from PWIs, graduates who scored lower on the quantitative section of the GRE had higher numbers of publications. These findings stimulate the need to 1) reduce reliance on the use of the GRE in admission committee decisions, 2) identify psychometrically valid indicators that tailored to assess outcome variables that are relevant to the careers of biomedical scientists, and 3) ensure the effective use of the tools in making admission decisions.

Analysis of Professional Competencies for the Clinical Research Data Management Profession.

OBJECTIVE: This job analysis was conducted to compare, assess and refine the competencies of the clinical research data management profession. MATERIALS AND METHODS: Two questionnaires were administered in 2015 and 2018 to collect information from data managers on professional competencies, types of data managed, types of studies supported, and necessary foundational knowledge. RESULTS: In 2018 survey, 67 professional competencies were identified. Job tasks differed between early- to mid-career and mid- to late-career practitioners. A large variation in the types of studies conducted and variation in the data managed by the participants was observed. DISCUSSION: Clinical research data managers managed different types of data with variety of research settings, which indicated a need for training in methods and concepts that could be applied across therapeutic areas and types of data. CONCLUSION: The competency survey reported here serves as the foundation for the upcoming revision of the Certified Clinical Data Manager (CCDMTM) exam.

Assessing the Alignment of Objectives, Instructional Activities, and Assessments in a Biomedical Informatics Curriculum.

The objective of this research was to assess the alignment of course learning objectives, instructional activities, and course assessments in a Biomedical Informatics curriculum. Each syllabi in the curriculum was reviewed and scored according to a validated rubric. Disagreements among reviewers adjudicated by consensus. Only low and moderate levels of alignment were identified. The results indicated the needs and goals of courses could be more effectively met with faculty investment in syllabi redesign and clarification to achieve course objectives. Root causes included word choice in learning objective statement as well as lack of consideration of instructional scaffolding by the course developer.

Analysis of professional competencies for the clinical research data management profession: implications for training and professional certification.

OBJECTIVE: To assess and refine competencies for the clinical research data management profession. MATERIALS AND METHODS: Based on prior work developing and maintaining a practice standard and professional certification exam, a survey was administered to a captive group of clinical research data managers to assess professional competencies, types of data managed, types of studies supported, and necessary foundational knowledge. RESULTS: Respondents confirmed a set of 91 professional competencies. As expected, differences were seen in job tasks between early- to mid-career and mid- to late-career practitioners. Respondents indicated growing variability in types of studies for which they managed data and types of data managed. DISCUSSION: Respondents adapted favorably to the separate articulation of professional competencies vs foundational knowledge. The increases in the types of data managed and variety of research settings in which data are managed indicate a need for formal education in principles and methods that can be applied to different research contexts (ie, formal degree programs supporting the profession), and stronger links with the informatics scientific discipline, clinical research informatics in particular. CONCLUSION: The results document the scope of the profession and will serve as a foundation for the next revision of the Certified Clinical Data Manager TM exam. A clear articulation of professional competencies and necessary foundational knowledge could inform the content of graduate degree programs or tracks in areas such as clinical research informatics that will develop the current and future clinical research data management workforce.