Use of ecological momentary assessment to guide curricular change in graduate medical education.

PURPOSE: To assess whether a novel evaluation tool could guide curricular change in an internal medicine residency program. METHOD: The authors developed an 8-item Ecological Momentary Assessment tool and collected daily evaluations from residents of the relative educational value of 3 differing ambulatory morning report formats (scale: 8  =  best, 0  =  worst). From the evaluations, they made a targeted curricular change and used the tool to assess its impact. RESULTS: Residents completed 1388 evaluation cards for 223 sessions over 32 months, with a response rate of 75.3%. At baseline, there was a decline in perceived educational value with advancing postgraduate (PGY) year for the overall mean score (PGY-1, 7.4; PGY-2, 7.2; PGY-3, 7.0; P < .01) and for percentage reporting greater than 2 new things learned (PGY-1, 77%; PGY-2, 66%; PGY-3, 50%; P < .001). The authors replaced the format of a lower scoring session with one of higher cognitive content to target upper-level residents. The new session's mean score improved (7.1 to 7.4; P  =  .03); the adjusted odds ratios before and after the change for percentage answering, "Yes, definitely" to "Area I need to improve" was 2.53 (95% confidence interval [CI], 1.45-4.42; P  =  .001) and to "Would recommend to others," it was 2.08 (95% CI, 1.12-3.89; P  =  .05). CONCLUSIONS: The Ecological Momentary Assessment tool successfully guided ambulatory morning report curricular changes and confirmed successful curricular impact. Ecological Momentary Assessment concepts of multiple, frequent, timely evaluations can be successfully applied in residency curriculum redesign.

Quantitative pediatric vision screening in primary care settings in Alabama.

PURPOSE: Alabama Medicaid reimburses "objective" vision screening (VS), i.e., by acuity or similar quantitative method, and well child checks (WCCs) separately. We analyzed the frequency of each service obtained. METHODS: Claims for WCC and VS provided between October 1, 2002 and September 30, 2003 for children aged 3 to 18 years, and summary data for all enrolled children, were obtained from Alabama Medicaid. We used univariate analysis followed by logistic regression to explore the potential influence of factors (patient age, provider type, and provider's volume of WCCs) on the receipt of VS at pre-school ages. RESULTS: Children receiving WCCs were 55% black, 40% white, and 5% other. Percentages of children with WCC claims were highest at 4 years (57%) and thereafter declined to 30% at 6 to 14 years and to <10% at 18 years. Nearly all VS (>98% at each age) occurred the same day as the WCC. Pediatricians provided 68% of all WCCs. Multivariate analysis, after adjusting for nesting of pre-school patients within provider, showed the odds ratios (ORs) of VS were increased by patient age (5 years vs. 3 years, OR = 3.57, p < 0.0001), nonphysician provider type (nonphysician vs. pediatrician, OR = 1.80, p = 0.0004) and high WCC volume (at or above vs. below the median number (n = 8) of WCC per provider per year (OR = 7.11, p < 0.0001)). Because VS rates were high when attendance to WCC visits was low, few enrolled children received VS at any age (6% at the age of 3, 13% at the age of 4, and a maximum of 20% at the age of 5). CONCLUSIONS: National efforts to reduce preventable vision loss from amblyopia are hampered because children are not available for screening and because providers miss many opportunities to screen vision at pre-school age. Efforts to improve VS should target pediatrician-led practices, because these serve greater numbers of children.

Identifying physician-recognized depression from administrative data: consequences for quality measurement.

BACKGROUND: Multiple factors limit identification of patients with depression from administrative data. However, administrative data drives many quality measurement systems, including the Health Plan Employer Data and Information Set (HEDIS). METHODS: We investigated two algorithms for identification of physician-recognized depression. The study sample was drawn from primary care physician member panels of a large managed care organization. All members were continuously enrolled between January 1 and December 31, 1997. Algorithm 1 required at least two criteria in any combination: (1) an outpatient diagnosis of depression or (2) a pharmacy claim for an antidepressant Algorithm 2 included the same criteria as algorithm 1, but required a diagnosis of depression for all patients. With algorithm 1, we identified the medical records of a stratified, random subset of patients with and without depression (n = 465). We also identified patients of primary care physicians with a minimum of 10 depressed members by algorithm 1 (n = 32,819) and algorithm 2 (n = 6,837). RESULTS: The sensitivity, specificity, and positive predictive values were: Algorithm 1: 95 percent, 65 percent, 49 percent; Algorithm 2: 52 percent, 88 percent, 60 percent. Compared to algorithm 1, profiles from algorithm 2 revealed higher rates of follow-up visits (43 percent, 55 percent) and appropriate antidepressant dosage acutely (82 percent, 90 percent) and chronically (83 percent, 91 percent) (p < 0.05 for all). CONCLUSIONS: Both algorithms had high false positive rates. Denominator construction (algorithm 1 versus 2) contributed significantly to variability in measured quality. Our findings raise concern about interpreting depression quality reports based upon administrative data.

Does early discharge with nurse home visits affect adequacy of newborn metabolic screening?

OBJECTIVE: To examine the impact of early discharge on newborn metabolic screening. STUDY DESIGN: Metabolic screening results were obtained from the Alabama State Lab for all infants born at our hospital between 8/1/97, and 1/31/99, and were matched with an existing database of early discharge infants. An early newborn discharge was defined as a discharge between 24 and 47 hours of age. Metabolic screening tests included phenylketonuria (PKU), hypothyroidism, and congenital adrenal hyperplasia (CAH). Early discharge and traditional stay infants were compared to determine the percentage of newborns screened and the timing of the first adequate specimen. RESULTS: The state laboratory received specimens from 3860 infants; 1324 were on early discharge newborns and 2536 infants in the traditional stay group. At least one filter paper test (PKU, hypothyroidism, and CAH) was collected on 99.2% of early discharge infants and 96.0% of traditional stay infants (P<.0001). Early discharge infants had a higher rate of initial filter paper specimens being inadequate (22.9%) compared with traditional stay infants (14.3%, P<.0001) but had a higher rate of repeat specimens when the initial specimen was inadequate (85.0% early discharge vs 75.3% traditional stay, P=.002). The early discharge group was more likely to have an adequate specimen within the first 9 days of life (1001, 98.8% early discharge vs 2016, 96.7% traditional stay, P=.0005). CONCLUSIONS: In this well established early discharge program with nurse home visits, newborn metabolic screening is not compromised by early discharge.