The Optimal Item Pool Design in Multistage Computerized Adaptive Tests With the p-Optimality Method.

The present study extended the p-optimality method to the multistage computerized adaptive test (MST) context in developing optimal item pools to support different MST panel designs under different test configurations. Using the Rasch model, simulated optimal item pools were generated with and without practical constraints of exposure control. A total number of 72 simulated optimal item pools were generated and evaluated by an overall sample and conditional sample using various statistical measures. Results showed that the optimal item pools built with the p-optimality method provide sufficient measurement accuracy under all simulated MST panel designs. Exposure control affected the item pool size, but not the item distributions and item pool characteristics. This study demonstrated that the p-optimality method can adapt to MST item pool design, facilitate the MST assembly process, and improve its scoring accuracy.

The Evidence for a Subscore Structure in a Test of English Language Competency for English Language Learners.

How to compute and report subscores for a test that was originally designed for reporting scores on a unidimensional scale has been a topic of interest in recent years. In the research reported here, we describe an application of multidimensional item response theory to identify a subscore structure in a test designed for reporting results using a unidimensional scale. This research also dealt with the problem of planned missing data due to low levels of item overlap among multiple test forms. Furthermore, we provided evidence for the generalizability of the multidimensional structure using multiple forms of the same test. We also compared the subscores from multiple groups to show the usefulness of the subscores. The research provides evidence that subscores can be identified and produced to provide useful information about different constructs for multiple examinee groups even though the test data were well fit by a unidimensional model.

The role of parenting in drug use among black, Latino and white adolescents.

This study investigates the role of parenting in adolescent drug use for black, white and Latino adolescents. Parenting has been consistently identified as a crucial factor in drug use by adolescents. This study uses data from the National Longitudinal Study of Adolescent Health. Results show that parenting has a significant effect on drug use for these adolescents. The relationship between parenting and drug use is more strongly negative for the Latino adolescents, than for black and white adolescents. This indicates that greater parental warmth and family acceptance exert a stronger impact in reducing drug use for Latino adolescents than is the case for the black and white adolescents.

Athletes' evaluations of their head coach's coaching competency.

This study provided initial validity evidence for multidimensional measures of coaching competency derived from the Coaching Competency Scale (CCS). Data were collected from intercollegiate men's (n = 8) and women's (n = 13) soccer and women's ice hockey teams (n = 11). The total number of athletes was 585. Within teams, a multidimensional internal model was retained in which motivation, game strategy, technique, and character building comprised the dimensions of coaching competency. Some redundancy among the dimensions was observed. Internal reliabilities ranged from very good to excellent. Practical recommendations for the CCS are given in the Discussion section.

Extending validity evidence for multidimensional measures of coaching competency.

This study extended validity evidence for multidimensional measures of coaching competency derived from the Coaching Competency Scale (CCS; Myers, Feltz, Maier, Wolfe, & Reckase, 2006) by examining use of the original rating scale structure and testing how measures related to satisfaction with the head coach within teams and between teams. Motivation, game strategy, technique, and character building comprised the dimensions of coaching competency. Data were collected from athletes (N = 585) nested within intercollegiate men's (g = 8) and women's (g = 13) soccer and women's ice hockey (g = 11) teams (G = 32). Validity concerns were observed for the original rating scale structure and the predicted positive relationship between motivation competency and satisfaction with the coach between teams. Validity evidence was offered for a condensed post hoc rating scale and the predicted relationship between motivation competency and satisfaction with the coach within teams.

Portfolios as a novel approach for residency evaluation.

The authors describe the development process for a portfolio that programs may use for evaluating psychiatry residents. A committee carefully developed the portfolio requirements for 13 necessary skills in psychiatry. Psychiatrists were trained to rate portfolios. Review of the development and training process suggested that portfolios provided information both on individual residents and on the residency program itself. The process requires orientation of both residents and faculty, attention to time demands, and availability of examples and resources.

Demonstration of portfolios to assess competency of residents.

Residency educators are identifying approaches to measure resident competence. Portfolios are well suited since they require work already completed as part of patient care where competency must be demonstrated. This paper describes assessment of the reliability and validity of portfolios in a psychiatry residency program. This was a cross-sectional study across 4years of residency education. Using guidelines, 18 residents assembled portfolios containing five entries chosen from 13 skills. Trained raters scored the portfolios. Residents and faculty were interviewed about their perceptions. Generalizability results indicated five entries and two raters were sufficient for relative decisions. Six entries or a third rater would be sufficient for absolute decisions. Portfolio scores tended to improve with years of training and correlated with psychiatric knowledge but not clinical performance. Residents and faculty identified benefits to assembling a portfolio. Portfolios incorporate tasks embedded in the residency to provide evidence of resident competency. The results support that the score is reliable and valid.