Validity Evidence for a Residency Admissions Standardized Assessment Letter for Pediatrics.

Construct: This study aims to provide validity evidence for the standardized Assessment Letter for Pediatrics as a measure of competencies expected of a 1st-year pediatrics resident as part of a pediatric residency admissions process. BACKGROUND: The Narrative Letter of Recommendation is a frequently used tool in the residency admissions process even though it has poor interrater reliability, lacks pertinent content, and does not correlate with residency performance. A newer tool, the Standardized Letter, has shown validity evidence for content and interrater reliability in other specialties. We sought to develop and provide validity evidence for the standardized Assessment Letter for Pediatrics. APPROACH: All 2012 and 2013 applicants invited to interview at the University of Geneva Pediatrics Residency Program provided 2 standardized Assessment Letters. Content for the letter was based on CanMEDS roles and ratings of 6 desired competencies and an overall assessment. Validity evidence was gathered for internal structure (Cronbach's alpha and generalizability), response process (interrater reliability with intraclass correlation), relations to other variables (Pearson's correlation coefficient), and consequences (logistic regression to predict admission). RESULTS: One hundred fourteen faculty completed 142 standardized Assessment Letters for 71 applicants. Average overall assessment was 3.0 of 4 (SD = 0.59). Cronbach's alpha was 0.93. The G-coefficient was 0.59. The decision study projected that four Assessment Letters are needed to attain a G-coefficient of 0.73. Applicant variance (28.5%) indicated high applicant differentiation. The Assessment Letter intraclass coefficient was 0.51, 95% confidence interval (CI) [0.43, 0.59]. Assessment Letter scores were correlated with the structured interview (r = .28), 95% CI [0.05, 0.51]; global rating (r = .36), 95% CI [0.13, 0.58]; and admissions decision (r = .25), 95% CI [0.02, 0.46]. Assessment Letter scores did not predict the admissions decision (odds ratio = 1.67, p = .37) after controlling for the unique contribution of the structured interview and global rating scores. CONCLUSION: Validity evidence supports use of the Assessment Letter for Pediatrics; future studies should refine items to improve predictive validity and explore how to best integrate the Assessment Letter into the residency admissions process.

Improving the residency admissions process by integrating a professionalism assessment: a validity and feasibility study.

The purpose of this study was to provide validity and feasibility evidence in measuring professionalism using the Professionalism Mini-Evaluation Exercise (P-MEX) scores as part of a residency admissions process. In 2012 and 2013, three standardized-patient-based P-MEX encounters were administered to applicants invited for an interview at the University of Geneva Pediatrics Residency Program. Validity evidence was gathered for P-MEX content (item analysis); response process (qualitative feedback); internal structure (inter-rater reliability with intraclass correlation and Generalizability); relations to other variables (correlations); and consequences (logistic regression to predict admission). To improve reliability, Kane's formula was used to create an applicant composite score using P-MEX, structured letter of recommendation (SLR), and structured interview (SI) scores. Applicant rank lists using composite scores versus faculty global ratings were compared using the Wilcoxon signed-rank test. Seventy applicants were assessed. Moderate associations were found between pairwise correlations of P-MEX scores and SLR (r = 0.25, P = .036), SI (r = 0.34, P = .004), and global ratings (r = 0.48, P < .001). Generalizability of the P-MEX using three cases was moderate (G-coefficient = 0.45). P-MEX scores had the greatest correlation with acceptance (r = 0.56, P < .001), were the strongest predictor of acceptance (OR 4.37, P < .001), and increased pseudo R-squared by 0.20 points. Including P-MEX scores increased composite score reliability from 0.51 to 0.74. Rank lists of applicants using composite score versus global rating differed significantly (z = 5.41, P < .001). Validity evidence supports the use of P-MEX scores to improve the reliability of the residency admissions process by improving applicant composite score reliability.

Economic evaluation of a 100% whey-based partially hydrolyzed infant formula in the prevention of atopic dermatitis among Swiss children.

OBJECTIVE: A pharmacoeconomic analysis was undertaken to determine costs, consequences, and cost-effectiveness of a partially hydrolyzed 100% whey-based infant formula, NAN-HA®, manufactured by Nestlé S.A, Switzerland (PHF-W), branded under BEBA HA® in Switzerland, in the prevention of atopic dermatitis (AD) in 'at risk' Swiss children when compared to standard cow's milk formula (SF). METHODS: Based on a 12-month time horizon including 6 months of formula consumption, an economic model was developed synthesizing treatment pathways, resource utilization, and costs associated with the treatment of AD in healthy 'at risk' Swiss newborns who could not be exclusively breastfed. Model inputs were retrieved from the literature, official formularies, and expert opinion. The treatment pathways considered a medical treatment approach, supplemented in some instances by a change of formula. The final outcome was the expected cost per avoided case of AD, yielding an incremental cost effectiveness ratio (ICER) for PHF-W vs SF. Outcomes were presented from three perspectives: the Swiss public healthcare system (MOH), the subject's family, and society (SOC). A secondary analysis compared PHF-W to whey-based extensively hydrolyzed formula (EHF) in prevention. RESULTS: The model yielded 1653 avoided AD cases by selecting PHF-W over SF in a birth cohort of 22,933 'at risk' infants. The base case analyses generated an expected ICER of CHF 982 from the MOH perspective as well as savings of CHF 2202 and CHF 1220 from the family and SOC perspectives, respectively. PHF-W yielded CHF 11.4M savings against EHF when the latter was assumed to be used in prevention. One-way and probabilistic sensitivity analyses confirmed the robustness of the model. CONCLUSION: Under a range of assumptions, this analysis has established the dominance from the family and societal perspectives and cost-effectiveness from the MOH perspective of PHF-W vs SF in the prevention of AD among 'at risk' Swiss infants.

Partially hydrolysed 100% whey-based infant formula and the prevention of atopic dermatitis: comparative pharmacoeconomic analyses.

Clinical trials have demonstrated that the risk of developing atopic dermatitis is reduced when using hydrolysed formulas to feed infants with a documented risk of atopy (i.e. an affected parent and/or sibling)when breastfeeding is not practised. However, little is known about the cost-effectiveness of using hydrolysed formulas. Consequently, economic analyses in 5 European countries (Denmark, France, Germany, Spain and Switzerland) have evaluated the costs and cost-effectiveness of a specific brand of 100% whey-based partially hydrolysed infant formula, NAN-HA® (PHF-W) compared with a cow's milk standard formula (SF) in the prevention of atopic dermatitis in at-risk children. This review synthesises the findings of these studies. Cost-effectiveness analyses (CEA) used a decision-analytic model to determine treatment pathways, resource utilisation and costs associated with the management of atopic dermatitis in healthy at-risk newborns who were not exclusively breastfed. The model had a 12-month horizon and applied reimbursement rates of 60-100% depending on the country. Outcomes were considered from the perspective of the public healthcare system (e.g. the Ministry of Health; MOH), family and society. The final outcome was the incremental cost-effectiveness ratio per avoided case of atopic dermatitis (ICER) for PHF-W versus SF. A cost-minimisation analysis was also performed to compare PHF-W with extensively hydrolysed formulas (EHF). The base-case CEA produced ICERs per avoided case for PHF-W versus SF of EUR 982-1,343 (MOH perspective), EUR -2,202 to -624 (family perspective) indicating savings, and EUR -1,220 to 719 from the societal perspective. The main costs related to formula (MOH and society) and time loss (family). In the cost-minimisation analysis, PHF-W yielded savings of between EUR 4.3 and 120 million compared with EHF-whey when the latter was used in prevention. In conclusion, PHF-W was cost-effective versus SF in the prevention of atopic dermatitis and cost saving compared with EHF when used in prevention.