Assessing the Ability of a Large Language Model to Score Free-Text Medical Student Clinical Notes: Quantitative Study.

Background: Teaching medical students the skills required to acquire, interpret, apply, and communicate clinical information is an integral part of medical education. A crucial aspect of this process involves providing students with feedback regarding the quality of their free-text clinical notes. Objective: The goal of this study was to assess the ability of ChatGPT 3.5, a large language model, to score medical students' free-text history and physical notes. Methods: This is a single-institution, retrospective study. Standardized patients learned a prespecified clinical case and, acting as the patient, interacted with medical students. Each student wrote a free-text history and physical note of their interaction. The students' notes were scored independently by the standardized patients and ChatGPT using a prespecified scoring rubric that consisted of 85 case elements. The measure of accuracy was percent correct. Results: The study population consisted of 168 first-year medical students. There was a total of 14,280 scores. The ChatGPT incorrect scoring rate was 1.0%, and the standardized patient incorrect scoring rate was 7.2%. The ChatGPT error rate was 86%, lower than the standardized patient error rate. The ChatGPT mean incorrect scoring rate of 12 (SD 11) was significantly lower than the standardized patient mean incorrect scoring rate of 85 (SD 74; P=.002). Conclusions: ChatGPT demonstrated a significantly lower error rate compared to standardized patients. This is the first study to assess the ability of a generative pretrained transformer (GPT) program to score medical students' standardized patient-based free-text clinical notes. It is expected that, in the near future, large language models will provide real-time feedback to practicing physicians regarding their free-text notes. GPT artificial intelligence programs represent an important advance in medical education and medical practice.

Assessing the usability by clinicians of VISION: A hierarchical display of patient-collected physiological information to clinicians.

BACKGROUND: The inability of patients to accurately and completely recount their clinical status between clinic visits reduces the clinician's ability to properly manage their patients. One way to improve this situation is to collect objective patient information while the patients are at home and display the collected multi-day clinical information in parallel on a single screen, highlighting threshold violations for each channel, and allowing the viewer to drill down to any analog signal on the same screen, while maintaining the overall physiological context of the patient. All this would be accomplished in a way that was easy for the clinician to view and use. METHODS: Patients used five mobile devices to collect six heart failure-related clinical variables: body weight, systolic and diastolic blood pressure, pulse rate, blood oxygen saturation, physical activity, and subjective input. Fourteen clinicians practicing in a heart failure clinic rated the display using the System Usability Scale that, for acceptability, had an expected mean of 68 (SD, 12.5). In addition, we calculated the Intraclass Correlation Coefficient of the clinician responses using a two-way, mixed effects model, ICC (3,1). RESULTS: We developed a single-screen temporal hierarchical display (VISION) that summarizes the patient's home monitoring activities between clinic visits. The overall System Usability Scale score was 92 (95% CI, 87-97), p < 0.0001; the ICC was 0.89 (CI, 0.79-0.97), p < 0.0001. CONCLUSION: Clinicians consistently found VISION to be highly usable. To our knowledge, this is the first single-screen, parallel variable, temporal hierarchical display of both continuous and discrete information acquired by patients at home between clinic visits that presents clinically significant information at the point of care in a manner that is usable by clinicians.

Electronic physiologic and subjective data acquisition in home-dwelling heart failure patients: An assessment of patient use and perception of usability.

BACKGROUND: The current approach to the outpatient management of heart failure involves patients recollecting what has happened to them since their last clinic visit. But patients' recollection of their symptoms may not be sufficiently accurate to optimally manage their disease. Most of what is known about heart failure is related to patients' diurnal symptoms and activities. Some mobile electronic technologies can operate continuously to collect data from the time patients go to bed until they get up in the morning. We were therefore interested to evaluate if patients would use a system of selected patient-facing devices to collect physiologic and subjective state data in and around the patients' period of sleep, and if there were differences in device use and perceptions of usability at the device level METHODS: This descriptive observational study of home-dwelling patients with heart failure, between 21 and 90 years of age, enrolled in an outpatient heart failure clinic was conducted between December 2014 and June 2015. Patients received five devices, namely, body weight scale, blood pressure device, an iPad-based subjective states assessment, pulse oximeter, and actigraph, to collect their physiologic (body weight, blood pressure, heart rate, blood oxygen saturation, and physical activity) and subjective state data (symptoms and subjective states) at home for the next six consecutive nights. Use was defined as the ratio of observed use over expected use, where 1.0 is observed equals expected. Usability was determined by the overall System Usability Scale score. RESULTS: Participants were 39 clinical heart failure patients, mean age 68.1 (SD, 12.3), 72% male, 62% African American. The ratio of observed over expected use for the body weight scale, blood pressure device, iPad application, pulse oximeter and actigraph was 0.8, 1.0, 1.1, 0.9, and 1.9, respectively. The mean overall System Usability Scale score for each device were 84.5, 89.7, 85.7, 87.6, and 85.2, respectively. CONCLUSIONS: Patients were able to use all of the devices and they rated the usability of all the devices higher than expected. Our study provides support for at-home patient-collected physiologic and subjective state data. To our knowledge, this is the first study to assess the use and usability of electronic objective and subjective data collection devices in heart failure patients' homes overnight.

The adoption of an electronic health record did not improve A1c values in Type 2 diabetes.

BACKGROUND: A major justification for the clinical adoption of electronic health records (EHRs) was the expectation that it would improve the quality of medical care. No longitudinal study has tested this assumption. OBJECTIVE: We used hemoglobin A1c, a recognized clinical quality measure directly related to diabetes outcomes, to assess the effect of EHR use on clinical quality. METHODS: We performed a five-and-one-half-year multicentre longitudinal retrospective study of the A1c values of 537 type 2 diabetic patients. The same patients had to have been seen on at least three occasions: once approximately six months prior to EHR adoption (before-EHR), once approximately six monthsafter EHR adoption (after-EHR) and once approximately five years after EHR adoption (five-years), for a total of 1,611 notes. RESULTS: The overall mean confidence interval (CI) A1c values for the before- EHR, after-EHR and five-years were 7.07 (6.91 - 7.23), 7.33 (7.14 - 7.52) and 7.19 (7.06 - 7.32), respectively. There was a small but significant increase in A1c values between before-EHR and after-EHR, p = .04; there were no other significant differences. There was a significant decrease in notes missing at least one A1c value, from 42% before-EHR to 16% five-years (p < .001). CONCLUSION: We found that based on patient's A1c values, EHRs did not improve the clinical quality of diabetic care in six months and five years after EHR adoption. To our knowledge, this is the first longitudinal study to directly assess the relationshipbetween the use of an EHR and clinical quality.

Point-of-Care Estimated Radiation Exposure and Imaging Guidelines Can Reduce Pediatric Radiation Burden.

INTRODUCTION: The steady increase in the use of computed tomography (CT) has particular concerns for children. Family physicians must often select pediatric imaging without any decision support. We hypothesized that point-of-care decision support would lead to the selection of imaging that lowered radiation exposure and improved guideline congruence. METHODS: Our double-blind, randomized simulation included family physicians in the Military Health System. Participants initially reviewed a pediatric hematuria scenario and selected imaging without decision support. Participants were subsequently randomized to either receive imaging-appropriateness guidelines and then estimated radiation exposure information or receive estimated radiation information then guidelines; imaging selections were required after each step. The primary outcome was the selected imaging modality with point-of-care decision support. RESULTS: The first arm increased CT ordering after viewing the guidelines (P = .008) but then decreased it after reviewing radiation exposure information (P = .007). In the second arm radiation information decreased CT and plain film use (P = not significant), with a subsequent increase in ultrasound and CT after the guideline presentation (P = .05). CONCLUSIONS: Decision support during a simulated pediatric scenario helped family physicians select imaging that lowered radiation exposure and was aligned with current guidelines, especially when presented with radiation information after guideline review. This information could help inform electronic medical record design.

Electronic health records improve clinical note quality.

BACKGROUND AND OBJECTIVE: The clinical note documents the clinician's information collection, problem assessment, clinical management, and its used for administrative purposes. Electronic health records (EHRs) are being implemented in clinical practices throughout the USA yet it is not known whether they improve the quality of clinical notes. The goal in this study was to determine if EHRs improve the quality of outpatient clinical notes. MATERIALS AND METHODS: A five and a half year longitudinal retrospective multicenter quantitative study comparing the quality of handwritten and electronic outpatient clinical visit notes for 100 patients with type 2 diabetes at three time points: 6 months prior to the introduction of the EHR (before-EHR), 6 months after the introduction of the EHR (after-EHR), and 5 years after the introduction of the EHR (5-year-EHR). QNOTE, a validated quantitative instrument, was used to assess the quality of outpatient clinical notes. Its scores can range from a low of 0 to a high of 100. Sixteen primary care physicians with active practices used QNOTE to determine the quality of the 300 patient notes. RESULTS: The before-EHR, after-EHR, and 5-year-EHR grand mean scores (SD) were 52.0 (18.4), 61.2 (16.3), and 80.4 (8.9), respectively, and the change in scores for before-EHR to after-EHR and before-EHR to 5-year-EHR were 18% (p<0.0001) and 55% (p<0.0001), respectively. All the element and grand mean quality scores significantly improved over the 5-year time interval. CONCLUSIONS: The EHR significantly improved the overall quality of the outpatient clinical note and the quality of all its elements, including the core and non-core elements. To our knowledge, this is the first study to demonstrate that the EHR significantly improves the quality of clinical notes.

QNOTE: an instrument for measuring the quality of EHR clinical notes.

BACKGROUND AND OBJECTIVE: The outpatient clinical note documents the clinician's information collection, problem assessment, and patient management, yet there is currently no validated instrument to measure the quality of the electronic clinical note. This study evaluated the validity of the QNOTE instrument, which assesses 12 elements in the clinical note, for measuring the quality of clinical notes. It also compared its performance with a global instrument that assesses the clinical note as a whole. MATERIALS AND METHODS: Retrospective multicenter blinded study of the clinical notes of 100 outpatients with type 2 diabetes mellitus who had been seen in clinic on at least three occasions. The 300 notes were rated by eight general internal medicine and eight family medicine practicing physicians. The QNOTE instrument scored the quality of the note as the sum of a set of 12 note element scores, and its inter-rater agreement was measured by the intraclass correlation coefficient. The Global instrument scored the note in its entirety, and its inter-rater agreement was measured by the Fleiss κ. RESULTS: The overall QNOTE inter-rater agreement was 0.82 (CI 0.80 to 0.84), and its note quality score was 65 (CI 64 to 66). The Global inter-rater agreement was 0.24 (CI 0.19 to 0.29), and its note quality score was 52 (CI 49 to 55). The QNOTE quality scores were consistent, and the overall QNOTE score was significantly higher than the overall Global score (p=0.04). CONCLUSIONS: We found the QNOTE to be a valid instrument for evaluating the quality of electronic clinical notes, and its performance was superior to that of the Global instrument.

rs660 polymorphism in Ro52 (SSA1; TRIM21) is a marker for age-dependent tolerance induction and efficiency of alloimmunization in sickle cell disease.

Patients with sickle cell disease (SCD) who receive red blood cell (RBC) transfusions have a higher rate of anti-RBC (allo and auto) antibody development than other transfused subjects. We hypothesized that an incidence and/or kinetics of RBC-specific antibody formation in SCD patients is influenced by a linked inheritance of the hemoglobin beta S (HbbetaS) allele and a polymorphism rs660C/T in the neighboring Ro52 gene. We found that 75% of C/T heterozygous and only 30.8% of T/T homozygous patients that developed antibodies were first transfused before the age of five. In addition, there was a significant inverse correlation between time of exposure to antigen or number of transfusions received and the age when T/T patients received first transfusion, indicating progressive development of competence of their immune system. In contrast, this correlation was not observed in patients with C/T genotype. Finally, increased expression of Ro52 was associated with the presence of the T/T genotype. These results suggest that rs660 polymorphism is a marker of efficiency of tolerance induction in early childhood and immune competence development to RBC antigens in SCD patients of pre-teen/teen age.

High-dose chemotherapy with autologous stem cell rescue in stage IIIB inflammatory breast cancer.

BACKGROUND: Despite the advances in breast cancer care, inflammatory breast cancer (IBC) has a poor prognosis. The purpose of this study was to determine the efficacy of high-dose chemotherapy (HDCT) with thiotepa, mitoxantrone and carboplatin (TMJ regimen) in women with TNM stage IIIB IBC. PATIENTS AND METHODS: Between 1991 and 1998, twenty-eight patients with stage IIIB IBC underwent an autologous stem cell transplant after undergoing chemotherapy, surgery and/or radiation. Stem cells were collected from the bone marrow and periphery after mobilization with growth factors. Patients received thiotepa 250 mg/m2 once daily i.v. for 3 days, mitoxantrone 40 mg/m2 for 1 day and carboplatin 333 mg/m2 once daily i.v. for 3 days as the conditioning regimen for the HDCT. Radiation therapy and tamoxifen was offered to patients post HDCT if appropriate. Progression-free survival and overall survival was assessed over a 15-year period. RESULTS: At the time of last follow-up in May, 2007, sixteen patients had relapsed. The median overall survival was 49.5 months. The median progression free survival was 40 months. There were no transplant-related deaths. Mucositis and infections were the major side-effects. These results show that HDCT with the TMJ regimen is safe and effective in patients with stage IIIB IBC.

Combination therapy with thalidomide, temozolomide and tamoxifen improves quality of life in patients with malignant astrocytomas.

BACKGROUND: Patients with malignant astrocytomas (MA) have a poor survival rate despite surgery, radiation therapy (RT), and chemotherapy (CT). Patients deteriorate rapidly with decreasing quality of life (QoL). The purpose of the current study was to determine the safety and efficacy, including QoL evaluation, of oral therapy with temozolomide, thalidomide, and tamoxifen (TTT) in patients with MA in an Institutional Review Board (IRB)-approved, prospective trial. PATIENTS AND METHODS: Twenty-three patients met the eligibility requirements and were enrolled after informed consent was signed. After baseline testing, patients received temozolomide 75 mg/m2 orally (p.o.) for the first 21 days, thalidomide 100 mg p.o. daily, and tamoxifen 100 mg p.o. daily for each 28-day cycle. Treatment continued until disease progression. Primary outcome measurements were survival (Kaplan-Meier analysis), response to treatment, toxicity (National Cancer Institute's Common Toxicity Criterion) and QoL evaluation. RESULTS: The Kaplan-Meier analysis showed that survival time from diagnosis was 78.4+/-15 weeks with a median survival of 54.6 weeks and from date of enrollment was 46.1+/-10 weeks with median survival of 33.3 weeks. Toxicity was limited to 5 patients with deep venous thrombosis (DVT), 2 of whom had pulmonary emboli (PE). All recovered with anticoagulation therapy and none suffered long term sequelae. Several QoL measures, including the global health status scores (p=0.003), were significantly improved after 2 cycles of treatment compared to the baseline assessment. CONCLUSION: The combination of temozolomide, thalidomide and tamoxifen administered as outpatient oral therapy resulted in significantly improved QoL for patients with MA without significant toxicity.

Long-term correction of ammonia metabolism and prolonged survival in ornithine transcarbamylase-deficient mice following liver-directed treatment with adeno-associated viral vectors.

The purpose of this study was to determine the efficacy of novel recombinant adeno-associated viral (AAV) vector constructs in correcting metabolic defects in the liver in two strains of ornithine transcarbamylase (OTC)-deficient mice (spf and spf-ash). AAV vectors expressing mouse OTC were produced with capsids from AAV2 and the novel serotypes AAV7, 8, and 9. OTC-deficient mice were infused with these vectors as well as a control AAV2/8 vector expressing LacZ. In vivo activity of OTC was assessed by measuring a surrogate marker, urine orotate. The novel vectors restored orotate levels to virtually normal 15 days after infusion, and each persisted to 1 year posttreatment. Liver OTC enzyme activity in spf mice was substantially higher in animals receiving novel vectors compared to those receiving AAV2 vectors. Animals receiving novel OTC-expressing vectors lived longer than those treated with AAV2 OTC or untreated controls, and they were tolerant to a challenge with NH3 at 21 days and beyond, which caused severe morbidity in control OTC-deficient animals. Numerous mice, representative of all treatment groups followed for +250 days, were observed to have either nodules or discrete tumors in the liver, the etiology of which is the subject of a companion paper.

Analysis of tumors arising in male B6C3F1 mice with and without AAV vector delivery to liver.

The present study reports on the frequency of liver tumors observed in a gene therapy study with AAV vectors in male mice of the B6C3F1 hybrid background, which are known to have a high frequency of spontaneous liver tumors. Male mice with mutations in their Otc gene and their wild-type siblings received AAV vectors expressing either the murine Otc or the LacZ gene. Untreated control animals were included in the study. All experimental groups, including wild-type and OTC-deficient animals not treated with vector, developed liver nodules, which in some cases were due to hepatocellular carcinoma. Vector DNA was lower in tumors than in adjacent normal liver. A statistical analysis of the data did not show an association between treatment with Otc vectors and formation of tumors in OTC-deficient mice. However, mice treated with LacZ vectors showed increased risks of tumor formation and hepatocellular carcinoma relative to untreated animals or animals that had received vectors with Otc as the transgene. It appears that AAV vectors alone do not contribute to the formation of tumors in these strains of mice although the expression of LacZ alone or in combination with vector may be problematic.