Measuring the continuous quality improvement orientation of medical education programs.

PURPOSE: There is a growing interest in applying continuous quality improvement (CQI) methodologies and tools to medical education contexts. One such tool, the "Are We Making Progress" questionnaire from the Malcolm Baldrige National Quality Award framework, adequately captures the dimensions critical for performance excellence and allows organizations to assess their performance and identify areas for improvement. Its results have been widely validated in business, education, and health care and might be applicable in medical education contexts. The measurement properties of the questionnaire data were analyzed using Rasch modeling to determine if validity evidence, based on Messick's framework, supports the interpretation of results in medical education contexts. Rasch modeling was performed since the questionnaire uses Likert-type scales whose estimates might not be amenable to parametric statistical analyses. DESIGN/METHODOLOGY/APPROACH: Leaders and teachers at 16 of the 17 Canadian medical schools were invited in 2015-2016 to complete the 40-item questionnaire. Data were analyzed using the ConQuest Rasch calibration program, rating scale model. FINDINGS: 491 faculty members from 11 (69 percent) schools participated. A seven-dimensional, four-point response scale model better fit the data. Overall data fit to model requirements supported the use of person measures with parametric statistics. The structural, content, generalizability, and substantive validity evidence supported the interpretation of results in medical education contexts. ORIGINALITY/VALUE: For the first time, the Baldrige questionnaire results were validated in medical education contexts. Medical education leaders are encouraged to serially use this questionnaire to measure progress on their school's CQI focus.

Psychometric Properties of the General Movement Optimality Score using Rasch Measurement.

AIM: To explore the psychometric properties of the general movements optimality score (GMOS) by examining its dimensionality, rating scale functioning, and item hierarchies using Rasch measurement. METHODS: Secondary data analysis of the GMOS data for video-recording of 383 infants with uni-, multidimensional, and mixed Rasch partial credit models. Videos were scored based on the global General Movement Assessment categories, and on the amplitude, speed, spatial range, proximal and distal rotations, onset and offset, tremulous and cramped components of the upper and lower extremities (21 items), resulting in the GMOS. RESULTS: The GMOS data fits best to a unidimensional mixed Rasch model with three different classes of infants, with all but two items contributing to the infants' separation. Rating scales functioned well for 19 items. Item difficulty hierarchies varied depending on infants' class. No floor effect and no substantive gaps between item difficulty estimates were found. CONCLUSION: The GMOS has strong psychometric properties to distinguish infants with different functional motor performance and provides a quantitative measure of quality of movement. INTERPRETATION: The GMOS can be confidently used to assist with early diagnosis, grade motor performance, and provide a solid base to study individual general movement developmental trajectories.

Psychometric Properties and Differential Item Functioning of a Web-Based Assessment of Children's Social Perspective-Taking.

Social perspective-taking (SPT), which involves the ability infer others' intentions, is a consequential social cognitive process. The purpose of this study is to evaluate the psychometric properties of a web-based social perspective-taking (SELweb SPT) assessment designed for children in kindergarten through third grade. Data were collected from two separate samples of children. The first sample included 3224 children and the second sample included 4419 children. Data were calibrated using Rasch dichotomous model (Rasch, 1960). Differential item and test functioning were also evaluated across gender and ethnicity groups. Across both samples, we found: evidence of consistent item fit; unidimensional item structure; and adequate item targeting. Poor item targeting at high and low ability levels suggests that more items are needed to distinguish low and high ability respondents. Analyses of DIF found some significant item-level DIF across gender, but no DIF across ethnicity. The analyses of person measure calibrations with and without DIF items evidenced negligible differential test functioning (DTF) across gender and ethnicity groups in both samples.

Beliefs about language development: construct validity evidence.

Understanding language development is incomplete without recognizing children's sociocultural environments, including adult beliefs about language development. Yet there is a need for data supporting valid inferences to assess these beliefs. The current study investigated the psychometric properties of data from a survey (MODeL) designed to explore beliefs in the popular culture, and their alignment with more formal theories. Support for the content, substantive, structural, generalizability, and external aspects of construct validity of the data were investigated. Subscales representing Behaviorist, Cognitive, Nativist, and Sociolinguistic models were identified as dimensions of beliefs. More than half of the items showed a high degree of consensus, suggesting culturally-transmitted beliefs. Behaviorist ideas were most popular. Bilingualism and ethnicity were related to Cognitive and Sociolinguistic beliefs. Identifying these beliefs may clarify the nature of child-directed speech, and enable the design of language intervention programs that are congruent with family and cultural expectations.

Using the Mixed Rasch Model to analyze data from the beliefs and attitudes about memory survey.

In this study, we used the Mixed Rasch Model (MRM) to analyze data from the Beliefs and Attitudes About Memory Survey (BAMS; Brown, Garry, Silver, and Loftus, 1997). We used the original 5-point BAMS data to investigate the functioning of the "Neutral" category via threshold analysis under a 2-class MRM solution. The "Neutral" category was identified as not eliciting the model expected responses and observations in the "Neutral" category were subsequently treated as missing data. For the BAMS data without the "Neutral" category, exploratory MRM analyses specifying up to 5 latent classes were conducted to evaluate data-model fit using the consistent Akaike information criterion (CAIC). For each of three BAMS subscales, a two latent class solution was identified as fitting the mixed Rasch rating scale model the best. Results regarding threshold analysis, person parameters, and item fit based on the final models are presented and discussed as well as the implications of this study.

Validity evidence for the Child Occupational Self Assessment.

To ensure that researchers and practitioners make valid interpretations from children's self-reports, evidence must demonstrate that the self-report has appropriate psychometric properties. The Child Occupational Self Assessment (COSA) is a self-report of occupational competence and value for everyday activities designed to involve children in identifying goals and assessing outcomes. Five hundred two children with disabilities, ages 6-17, completed the COSA. We used a Rasch Partial Credit model and parametric and nonparametric statistics to obtain validity evidence. Evidence suggests that the COSA has good content, structural, and substantive validity as given by item and child fit statistics and unidimensionality evaluation. Evidence for external validity was mixed because child fit status and measures varied with some demographic and assessment administration variables. Evidence suggests that most children's responses to the COSA can be validly interpreted as indicators of occupational competence and value for everyday activities.

Measuring sexual risk for HIV: a Rasch scaling approach.

In this study, we developed an HIV transmission risk scale and examined its psychometric properties using data on sexual behavior obtained from a probability sample of adult men who have sex with men living in Chicago. We used Messick's (Am Psychol 50:741-749, 1995) conceptualization of unified validity theory to organize the psychometric properties of data. Evidence related to scale content was investigated via Rasch item fit statistics, point-measure correlations, and expert evaluation. The substantive aspect of validity was addressed by interpreting the meaningfulness of the item difficulty hierarchy (continuum of risky behaviors) and assessment of person fit. The structural aspect of validity was assessed using Rasch item fit statistics, principal component analysis of standardized residuals, and other residual analyses. The generalizability aspect of validity was investigated via internal consistency reliability estimates for both items and persons, and aspects of external validity were addressed by examining between-group differences with respect to levels of risky behavior. Applications and suggested future studies are discussed.

Rating scale use by children with disabilities on a self-report of everyday activities.

UNLABELLED: Kramer JM, Smith EV Jr, Kielhofner G. Rating scale use by children with disabilities on a self-report of everyday activities. OBJECTIVE: To examine whether children with disabilities interpret a self-report of perceived competence and importance of everyday activities in a consistent manner and use the rating scales as intended. If not, are differences in how children interpret the scale associated with personal or contextual variables? DESIGN: Assessment and rating scale development using the Mixed Rasch Model. SETTING: Rehabilitation clinics, schools, and research sites in the United States and Europe. PARTICIPANTS: Children (N=407) aged 6 to 17 years; all had a diagnosed disability or received occupational therapy services. INTERVENTIONS: Not applicable. MAIN OUTCOME MEASURE: The Child Occupational Self Assessment; 25 items representing everyday activities and two 4-point scales: competence and importance. RESULTS: For each scale, 2 groups of children were identified. Approximately 50% of the children used the rating scales as intended. The remainder used the scales as reversed 2-point scales; these children were younger and more likely to have an intellectual disability. Country and practice setting were also associated with rating scale use. All items but 1 had acceptable fit to the Rasch model, and groups of children differed in the relative competence and importance reported. CONCLUSIONS: Personal and contextual variables are associated with children with disabilities' use of self-report rating scales. Younger children and children with intellectual disabilities use a modified response pattern.

Caffeine expectancy: instrument development in the Rasch measurement framework.

Although caffeine is the most widely consumed psychoactive drug in the world, the mechanisms associated with consumption are not well understood. Nonetheless, outcome expectancies for caffeine use are thought to underlie caffeine's reinforcing properties. To date, however, there is no available, sufficient measure by which to assess caffeine expectancy. Therefore, the current study sought to develop such a measure employing Rasch measurement models. Unlike traditional measurement development techniques, Rasch analyses afford dynamic and interactive control of the analysis process and generate helpful information to guide instrument construction. A 5-stage developmental process is described, ultimately yielding a 37-item Caffeine Expectancy Questionnaire (CEQ) comprised of 4 factors representing "withdrawal symptoms," "positive effects," "acute negative effects," and "mood effects." Initial evaluation of the CEQ yielded sufficient evidence for various aspects of validity. Although additional research with more heterogeneous samples is required to further assess the measure's reliability and validity, the CEQ demonstrates potential with regard to its utility in experimental laboratory research and clinical application.

Selecting a measurement model for the analysis of the National Institutes of Health Stroke Scale.

To select the most appropriate model for the analysis of data from the National Institutes of Health Stroke Scale (NIHSS), the graded-response, Rasch partial credit, and generalized partial credit models were used to analyze NIH stroke data of 1,191 acute ischemic stroke patients. Based on Akaike's Information Criterion (AIC) and Bayesian Information Criterion (BIC), the generalized partial credit model has the most generalizable parameters. Items on the NIHSS have different discriminating powers. The generalized partial credit model, which allows varying slopes of item response functions, is the most appropriate model for the analysis of the NIHSS.

The study skills self-efficacy scale for use with Chinese students.

Silver, Smith and Greene (2001) examined the dimensionality of responses to the Study Skills Self-Efficacy Scale (SSSES) using exploratory principal factor analysis (PFA) and Rasch measurement techniques based on a sample of social science students from a community college in the United States. They found that responses defined three related dimensions. In the present study, Messick's (1995) conceptualization of validity was used to organize the exploration of the psychometric properties of data from a Chinese version of the SSSES. Evidence related to the content aspect of validity was obtained via item fit evaluation; the substantive aspect of validity was addressed by examining the functioning of the rating scales; the structural aspect of validity was explored with exploratory PFA and Rasch item fit statistics; and support for the generalizability aspect of validity was investigate via differential item functioning and internal consistency reliability estimates for both items and persons. The exploratory PFA and Rasch analysis of responses to the Chinese version of the SSSES were conducted with a sample of 494 Hong Kong high school students. Four factors emerged including Study Routines, Resource Use, Text-Based Critical Thinking, and Self-Modification. The fit of the data to the Rasch rating scale model for each dimension generally supported the unidimensionality of the four constructs. The ordered average measures and thresholds from the four Rasch analyses supported the continued use of the six-point response format. Item and person reliability were found to be adequate. Differential item functioning across gender and language taught in was minimal.

Psychometric Assessment of the Job Embeddedness Instrument: A Rasch Perspective.

The aim of this study was to examine the psychometric properties of the job embeddedness instrument (JEI) using a Rasch perspective in a sample of Registered Nurses (RNs). A secondary analysis of data was conducted from a previous study examining the job embeddedness of rural and urban RNs. A Rasch analysis supported the six underlying dimensions: organizational fit, community fit, organizational links, community links, organizational sacrifice, and community sacrifice. The results of this study also demonstrate additional evidence of the validity, reliability, and generalizability of the JEI inferences with a sample of RNs. In total, 38 of 39 items of the original JEI were retained in the model. The psychometric evaluation attained through this multidimensional Rasch analysis provided support for using the JEI to assess the level of job embeddedness for RNs.

Evaluating the Psychometric Properties of a Measure of Ethnic Identity Among Black South African Youth.

The authors examined the dimensionality and psychometric properties of the Multigroup Ethnic Identity Measure (MEIM) among Black South African adolescents (ages = 13-14; 52% female) representing several ethnic groups (Ndebele, Pedi, Sotho, Swati, Tsonga, Tswana, Venda, Xhosa, Zulu) and evaluated the measure for differential item functioning primarily among four ethnic groups (Sotho, Tswana, Xhosa, and Zulu). Results indicated that a two-dimensional model best represented the data, reflecting ethnic search/clarity and ethnic affirmations. Subsequently, we evaluated the equivalence of the MEIM among the four South African ethnic groups and a sample of African American adolescents (Mage  = 15.57 years; SD = 1.22; 51% female). Further analyses revealed that configural and metric models were excellent across the four South African ethnic groups and the African American group. However, scalar invariance (i.e., intercept) was not found; the item intercepts were different for the South African ethnic groups and African Americans. Findings are discussed with consideration for conducting research on ethnic identity among youth in South Africa.

Relationship of candidate communication and organization skills to oral certification examination scores.

This study investigated the relationship between candidate's proficiency in communication/organization and ability measures derived from judges' oral examination ratings on a medical specialty certification examination. Judges who rated candidates on the oral examination also provided a separate rating for candidates' communication/organization skills. ANOVA was used to examine differences among levels of communication/organizational skills with respect to candidates' ability measures on the oral examination. There was a statistically significant increase in oral examination measures for candidates with higher levels of communication/organization skills. This supports the hypothesis that candidates who were more proficient in organizing and presenting their responses were more likely to have better performance on the oral examination. Given communication and oral examination ratings were provided by the same judges, future research should investigate whether these preliminary findings generalize to situations in which communication ratings are obtained from independent ratings during and outside the oral examination.

Instrument development tools and activities for measure validation using Rasch models: part I - instrument development tools.

Instrument development is an arduous task that, if undertaken with care and consideration, can lay the foundation for the development of validity arguments relating to the inferences and decisions that are based on test measures. This article, Part I of a two-part series, provides an overview of validity concepts and describes how instrument development efforts can be conducted to facilitate the development of validity arguments. Our discussion focuses on documentation of the purpose of measurement, creation of test specifications, item development, expert review, and planning of pilot studies. Through these instrument development activities and tools, essential information is documented that will feed into the analysis, summary, and reporting of data relevant to validity arguments discussed in Part II of this series.

Instrument development tools and activities for measure validation using Rasch models: part II--validation activities.

Accumulation of validity evidence is an important part of the instrument development process. In Part I of a two-part series, we provided an overview of validity concepts and described how instrument development efforts can be conducted to facilitate the development of validity arguments. In this, Part II of the series, we identify how analyses, especially those conducted within a Rasch measurement framework, can be used to provide evidence to support validity arguments that are founded during the instrument development process.

The assessment of diabetes knowledge and self-efficacy in a diverse population using Rasch measurement.

The purpose of this research was to develop survey instruments to evaluate diabetes knowledge and self-efficacy in a diverse population, and investigate the psychometric properties of data obtained with these instruments using Rasch measurement. Two-hundred and fifty-five urban-dwelling participants with diabetes were recruited to complete surveys through independent interviews. To evaluate the association of health literacy on metabolic control, formal literacy and hemoglobin A1c fingerstick testing were performed. Rasch analysis of the data yielded item and person calibrations for self-efficacy and knowledge, with variable maps created to provide both norm and criterion-referenced interpretations. Knowledge scale person separation reliability was 0.50 and item separation reliability was 0.98; while self-efficacy scale person separation reliability was 0.72 with item separation reliability of 0.92. Statistically significant partial correlations were observed between knowledge and health literacy (r = 0.41, p<.001), and self-efficacy and hemoglobin A1c (r = -0.33, p<.001). However, there was no correlation between diabetes knowledge and hemoglobin A1c (r = 0.035, p = 0.29), or health literacy and A1c (r = 0.022, p = 0.36). Diabetes knowledge varied, with non-English speaking individuals having lower measures than English speakers (t(252) = -4.86, p<.001). Non-English speaking individuals also had lower self-efficacy measures than English speakers (t(251) = -2.68, p = .008). Current knowledge deficits and perceptions of self-management may be estimated visually through variable mapping, which may help in individualizing informational needs for people with diabetes.

Implementation and evaluation of a low-literacy diabetes education computer multimedia application.

OBJECTIVE: To evaluate a clinic-based multimedia intervention for diabetes education targeting individuals with low health literacy levels in a diverse population. RESEARCH DESIGN AND METHODS: Five public clinics in Chicago, Illinois, participated in the study with computer kiosks installed in waiting room areas. Two hundred forty-four subjects with diabetes were randomized to receive either supplemental computer multimedia use (intervention) or standard of care only (control). The intervention includes audio/video sequences to communicate information, provide psychological support, and promote diabetes self-management skills without extensive text or complex navigation. HbA(1c) (A1C), BMI, blood pressure, diabetes knowledge, self-efficacy, self-reported medical care, and perceived susceptibility of complications were evaluated at baseline and 1 year. Computer usage patterns and implementation barriers were also examined. RESULTS: Complete 1-year data were available for 183 subjects (75%). Overall, there were no significant differences in change in A1C, weight, blood pressure, knowledge, self-efficacy, or self-reported medical care between intervention and control groups. However, there was an increase in perceived susceptibility to diabetes complications in the intervention group. This effect was greatest among subjects with lower health literacy. Within the intervention group, time spent on the computer was greater for subjects with higher health literacy. CONCLUSIONS: Access to multimedia lessons resulted in an increase in perceived susceptibility to diabetes complications, particularly in subjects with lower health literacy. Despite measures to improve informational access for individuals with lower health literacy, there was relatively less use of the computer among these participants.

Preliminary Development of a Brief Cannabis Use Disorder Screening Tool: The Cannabis Use Disorder Identification Test Short-Form.

Introduction: Rates of cannabis use disorder (CUD) among vulnerable populations have increased in recent years, highlighting a need to equip providers with an efficient screening tool. Materials and Methods: A short form of the Cannabis Use Disorder Identification Test-Revised (CUDIT-R) was developed by using item response theory and traditional statistical methods, with data from two community samples of cannabis users representing two countries. Four item selection methods (Rasch regression, test characteristic curve, logistic regression, discriminant function analysis) were employed to identify the optimal three-item shortened version. The diagnostic ability of the short form was evaluated by using receiver operating characteristic curves. Results: Using a cut score of 2, the 3-item CUDIT-Short Form (CUDIT-SF; reliability alpha=0.66, Sample 1; 0.80, Sample 2) identified 78.26% of participants in Sample 1 and 78.31% of participants in Sample 2 who met DSM-5 criteria for CUD, with 98% agreement in Sample 1 and 93% agreement in Sample 2 with the full CUDIT-R on CUD classifications using a cut score of 13. Specificity was 76.70 and 78.00 in Samples 1 and 2, respectively. Conclusions: The CUDIT-SF may be useful in busy clinical settings for a stepwise screening. Further validation of this shortened version with larger samples and in different settings is warranted.

Effect of item redundancy on Rasch item and person estimates.

One of the assumptions of many latent trait models is local independence. This assumption specifies that, after controlling for the underlying trait, item responses are independent. Given the lack of studies of model robustness against such violations, it appears that this assumption is frequently taken for granted. Therefore, this study investigated the robustness of Rasch item and person estimates with simulated data under varying number of items, sample sizes, and levels of item redundancy. Item and person reliabilities, the standard deviations of the person and item estimates, the root mean squared differences and mean signed differences among person and item estimates, the correlations between person estimates, and the percentage of person estimates shifting by more than .50 logits were used to evaluate the impact of item redundancy. Both norm and criterion-reference interpretations may be influenced by the imputation of redundancy into the data. However, it appears that the amount of redundancy needs to be considerable before such interpretations would be adversely impacted. Suggestions for further simulation research are provided.