Does Instruction Affect the Underlying Dimensionality of a Kinesiology Test?

Does effective instruction, which changes students' knowledge and possibly alters their cognitive functions, also affect the dimensionality of an achievement test? This question was examined by the parameterization of kinesiology test items (n = 42) with a Rasch dichotomous model, followed by an investigation of dimensionality in a pre- and post-test quasi-experimental study design. College students (n = 108) provided responses to kinesiology achievement test items. Then the stability of item difficulties, gender differences, and the interaction of item content categories with dimensionality were examined. In addition, a PCA/t-test protocol was implemented to examine dimensionality threats from the item residuals. Internal construct validity was investigated by regressing item content components on calibrated item difficulties. Measurement model item residuals were also investigated with statistical decomposition methods. In general, the results showed significant student achievement between pre and post testing, and dimensionality disturbances were relatively minor. The amount of unexpected item "shift" in an un-equated measurement dimension between pre and post testing was less than ten percent of the total items and largely concentrated among several unrelated items. An unexpected finding was a residual cluster consisting of several items testing related technical content. Complicating interpretation, these items tended to appear near the end of the test, which implicates test position as a threat to measurement equivalence. In general, the results across several methods did not tend to identify common threats and instead pointed to multiple sources of threats with varying degree of prominence. These results suggest conventional approaches to measurement equivalence that emphasize expedient overall procedures such as DIF, IRT, and factor analysis are probably capturing isolated sources of variability. Their implementation probably improves measurement equivalence but with substantial residual sources undetected.

Automatic item generation implemented for measuring artistic judgment aptitude.

Automatic item generation (AIG) is a broad class of methods that are being developed to address psychometric issues arising from internet and computer-based testing. In general, issues emphasize efficiency, validity, and diagnostic usefulness of large scale mental testing. Rapid prominence of AIG methods and their implicit perspective on mental testing is bringing painful scrutiny to many sacred psychometric assumptions. This report reviews basic AIG ideas, then presents conceptual foundations, image model development, and operational application to artistic judgment aptitude testing.

Multi-factor scale consolidation when theory is weak.

As a practical matter, Spirituality and Quality of Life in the health sciences are usually measured separately. Theoretical foundations for this distinction, however, are not strong. In this research, an empirical investigation was conducted into their joint calibration with a Rasch model. Functional Assessment of Cancer Therapy-General (28 items), a cancer health-related quality of life measure (HRQOL), and Functional Assessment of Chronic Illness - Spiritual Well-Being (12 items), a measure of religious and existential well-being (Spirituality), were co-calibrated with a Rasch model implemented with WINSTEPS software for ratings from 545 breast cancer patients. The results show a hierarchical integration of QOL and Spirituality items on a common variable, and both patient separation (2.66) and reliability (.88) improve after co-calibration. Principal Component Analysis of co-calibrated item residuals did not show major threats to dimensionality, and joint calibration explains item variance comparable to separate calibrations (51.9%). Although patient measures (logits) based on separate and co-calibration are within two standard errors, ethnic and racial group values shift after co-calibration.

An external validation study of a classification of mixed connective tissue disease and systemic lupus erythematosus patients.

Mixed Connective Tissue Disease (MCTD) and Systemic Lupus Erythematosus (SLE) are autoimmune rheumatic diseases that are difficult for physicians to diagnose and to distinguish for a variety of reasons. The correct classification of these two diseases is a crucial issue for clinicians who treat autoimmune rheumatic diseases. In prior research, medical risk factors represented by instrument or laboratory measures and physician judgments (12 key features for MCTD and 12 key features for SLE) were parameterized with a one parameter logistic function in a Rasch model. Those results identified separate diagnostic dimensions for MCTD and SLE. This procedure was replicated in the present research with a sample of largely African American and Hispanic patients. Results verified separate dimensions for MCTD and SLE, which suggests MCTD is a separate disease from SLE.

An ADL measure for spinal cord injury.

Occupational therapists do not have a comprehensive, objective method for measuring how persons with tetraplegia perform activities of daily living (ADL) in their homes and communities, because SCI ADL performance is usually determined in rehabilitation. The ADL Habits Survey (ADLHS) is designed specifically to address this knowledge gap by surveying performance on relevant and meaningful activities in homes and communities. After a comprehensive task analysis and pilot development, 30 activities were selected that emphasize a broad range of hand and wrist, reaching, and grasping movements in compound activities. A sample of 49 persons with cervical spinal cord injuries responded to items. The sample was predominantly male, median age was 41 years, and ASIA motor classification levels ranged from C2 through C8/T1 with majority concentration in C4, C5, or C6 (68%). Each participant report was rated by an occupational therapist using a seven category rating scale, and the item by participant response matrix (30 X 49) was analyzed with a Rasch model for rating scales. Results showed excellent participant separation (>4) and very high reliability (>.95), and both item and participant fit values were adequate (STANDARDIZED INFIT less than absolute value of 3). With only two exceptions, all participants fit the Rasch rating scale model, and only one item "Light housekeeping" presented significant fit issues. Principal Components Analysis an analysis of item residuals did not reveal serious threats to unidimensionality. A between group fit comparison of participants with more versus less movement found invariant item calibrations, and ANOVA of participant measures found statistically significant differences across ASIA motor classification levels. These ADLHS results offer occupational therapists a new method for measuring ADL that is potentially more sensitive to functional changes in tetraplegia than most instruments in common use. Accommodation of step disorder with a three category rating scale did not diminish measurement properties.

Measurement of mothers' confidence to care for children assisted with tracheostomy technology in family homes.

The purpose of this research was to develop an objective, linear measure of mothers' confidence to care for children assisted with tracheostomy medical technology in their homes. Caregiver confidence is addressed in this research for three technologies, namely, a) trachesotomy, b) tracheostomy and ventilator, and c) BiPAP/CPAP although detailed measurement results are only reported for tracheostomy, and its co-calibration with tracheostomy and ventilator caregiving items. The sample consisted of 53 mothers responding to several caregiver questionnaires based on a caregiving task matrix after content and clinical validation. A major challenge was integrating this construct with overarching principles already established by Functional Caregiving, a multi-level humanistic caregiving model for children with intellectual disabilities. Empirical analyses included principal components analysis, and then linear transformation of Tracheostomy item ratings to an objective, equal-interval scale with a Rasch model. Results show caregiver separation on the Tracheostomy caregiving scale was 2.66 and reliability, .88. In general, co-calibration improved measurement properties without affecting mothers' caregiving confidence measures. Although sample size was small, measuring mothers' confidence to care for a child supported by complex medical technologies appears very promising.

Foreword: emergence of efficiency in health outcome measurement.

Psychosocial measurement in the 21st Century is a dynamic field that is addressing challenges unthinkable even a generation ago. Sophisticated methods and modern technology has brought psychometrics to the cusp of scientific objectivity. This Foreword provides historical context and intellectual foundations for appreciating contemporary psychometric advancements, as well as a perspective on issues that are determining future advances. Efficiency in outcome measurement is one of these forces driving future advances. Efficiency, however, can easily become conflated with expediency, and neither can substitute for effectiveness. Blind efficiency runs risk of degrading measurement properties. Likewise, measurement advancement without accommodation to ordinary needs leads to practical rejection. Bouchard presents a biographical link between scientific physics and Rasch models that opened the door for fundamental psychosocial measurement. Symposium papers presented in this issue present a broad range of ideas about contemporary psychosocial measurement. Granger summarizes key ideas underlying achievement of objective, fundamental measurement. Massof, then, Stenner and Stone present alternative perspectives on scientific knowledge systems, which are prominent landmarks on the psychometric horizon. Fisher and Burton describe fundamental measurement methodology in diagnosis and implementation of technology, which will consolidate isolated and redundant constructs in PROMIS. Hart presents an overview on computer adaptive testing, which is the vanguard in health outcome measurement. Kisala and Tulsky present a qualitative strategy that is improving sensitivity and validity of new outcome measures. Their diversity reflects an intense competition of ideas about solving measurement problems. Their collection together in this special issue is a milestone and tribute to scientific ingenuity.

A Rasch analysis for classification of systemic lupus erythematosus and mixed connective tissue disease.

The classification of rheumatic diseases is challenging because these diseases have protean and frequently overlapping clinical and laboratory manifestations. This problem is typified by the difficulty of classification and differentiation of two prototypic multi-system autoimmune diseases, Systemic Lupus Erythematosus (SLE) and Mixed Connective Tissue Disease (MCTD). The researchers submitted medical risk factor data represented by instrument or laboratory measures and physician judgments (12 key features for SLE) from 43 patients diagnosed with SLE and 12 key features for MCTD from 51 patients diagnosed with MCTD to the WINSTEPS Rasch analysis program. Using Rasch model parameterization, and fit and residuals analyses, the researchers identified separate dimensions for MCTD and SLE, thereby lending support to the position that MCTD is its own separate disease, distinct from SLE.

Nonequivalent survey consolidation: an example from functional caregiving.

Functional Caregiving (FC) is a construct about mothers caring for children (both old and young) with intellectual disabilities, which is operationally defined by two nonequivalent survey forms, urban and suburban, respectively. The purposes of this research are, first, to generalize school-based achievement test principles to survey methods by equating two nonequivalent survey forms. A second purpose is to expand FC foundations by a) establishing linear measurement properties for new caregiving items, b) replicate a hierarchical item structure across an urban, school-based population, c) consolidate survey forms to establish a calibrated item bank, and d) collect more external construct validity data. Results supported invariant item parameters of a fixed item form (96 items) for two urban samples (N = 186). FC measures also showed expected construct relationships with age, mental depression, and health status. However, only five common items between urban and suburban forms were statistically stable because suburban mothers' age and child's age appear to interact with medical information and social activities.

Relative precision, efficiency and construct validity of different starting and stopping rules for a computerized adaptive test: the GAIN substance problem scale.

Substance abuse treatment programs are being pressed to measure and make clinical decisions more efficiently about an increasing array of problems. This computerized adaptive testing (CAT) simulation examined the relative efficiency, precision and construct validity of different starting and stopping rules used to shorten the Global Appraisal of Individual Needs' (GAIN) Substance Problem Scale (SPS) and facilitate diagnosis based on it. Data came from 1,048 adolescents and adults referred to substance abuse treatment centers in 5 sites. CAT performance was evaluated using: (1) average standard errors, (2) average number of items, (3) bias in person measures, (4) root mean squared error of person measures, (5) Cohen's kappa to evaluate CAT classification compared to clinical classification, (6) correlation between CAT and full-scale measures, and (7) construct validity of CAT classification vs. clinical classification using correlations with five theoretically associated instruments. Results supported both CAT efficiency and validity.

Substance use disorder symptoms: evidence of differential item functioning by age.

This study examined the applicability of substance abuse diagnostic criteria for adolescents, young adults, and adults using the Global Appraisal of Individual Need's Substance Problems Scale (SPS) from 7,408 clients. Rasch analysis was used to: 1) evaluate whether the SPS operationalized a single reliable dimension, and 2) examine the extent to which the severity of each symptom and the overall test functioned the same or differently by age. Rasch analysis indicated that the SPS was unidimensional with a person reliability of .84. Eight symptoms were significantly different between adolescents and adults. Young adult calibrations tended to fall between adolescents and adults. Differential test functioning was clinically negligible for adolescents but resulted in about 7% more adults being classified as high need. These findings have theoretical implications for screening and treatment of adolescents vs. adults. SPS can be used across age groups though age-specific calibrations enable greater precision of measurement.

Rasch analysis of a new construct: functional caregiving for adult children with intellectual disabilities.

This research examined empirical evidence for a new construct, Functional Caregiving, which is a theory about mothers' caregiving of their adult children with intellectual disabilities. A sample of 108 biological mothers and primary caregivers rated survey items about their confidence to perform caregiving tasks. Rasch rating scale analysis found 61 items defined an empirical construct with three caregiving levels: Advocacy, Personal Caregiving, and Community. Results show item separation was 3.11 with high reliability, .91, and mother separation was 2.93 and reliability, .90. Both items and mothers showed adequate INFIT and OUTFIT values. Item invariance was confirmed between older and younger mothers, and principle components analysis of item residuals did not reveal any major dimensionality threats. Item decomposition analysis showed FC content theory to account for 58 percent of item calibration variance (R2 = .58, F = 42.3, p < .001). These results have important practical implications for health and social services, as well as family caregiving, interdisciplinary practices, and health policy development.

Breakthrough measuring neighborhoods.

An empirical strategy is presented for transforming ordinal counts and percentages to interval scale measures by recoding them as ordered categories and estimating Rasch model rating scale parameters. This strategy is demonstrated for a neighborhood construct socioeconomic disadvantage operationally defined by eight characteristics of Chicago neighborhoods (N = 77). Results show surprisingly sound model fit and satisfactory scale invariance between 1980 and 1990 census. A striking finding obscured by traditional methods is many Chicago neighborhoods are four times more disadvantaged than official U.S. poverty threshold. Intramodel construct validation confirms this scale structure is consistent with sociological expectations about property values, income, and race. A general benefit of this approach over conventional categorical socioeconomic indices is neighborhood measurement on a linear scale.

A multi-factor Rasch scale for artistic judgment.

Measurement properties are reported for a combined scale of abstract and figurative artistic judgment aptitude items. Abstract items are synthetic, rule-based images from Visual Designs Test which implements a statistical algorithm to control design complexity and redundancy, and figurative items are canvas paintings in five styles, Fauvism, Post-Impressionism, Surrealism, Renaissance, and Baroque especially created for this research. The paintings integrate syntactic structure from VDT Abstract designs with thematic content for each style at four levels of complexity while controlling redundancy. Trained test administrators collected preference for synthetic abstract designs and authentic figurative art from 462 examinees in Johnson O'Connor Research Foundation testing offices in Boston, New York, Chicago, and Dallas. The Rasch model replicated measurement properties for VDT Abstract items and identified an item hierarchy that was statistically invariant between genders and generally stable across age for new, authentic figurative items. Further examination of the figurative item hierarchy revealed that complexity interacts with style and meaning. Sound measurement properties for a combined VDT Abstract and Figurative scale shows promise for a comprehensive artistic judgment construct.

Screening for atypical suicide risk with person fit statistics among people presenting to alcohol and other drug treatment.

UNLABELLED: Symptoms of internalizing disorders (depression, anxiety, somatic, trauma) are the major risk factors for suicide. Atypical suicide risk is characterized by people with few or no symptoms of internalizing disorders. OBJECTIVE: In persons screened at intake to alcohol or other drug (AOD) treatment, this research examined whether person fit statistics would support an atypical subtype at high risk for suicide that did not present with typical depression and other internalizing disorders. METHODS: Symptom profiles of the prototypical, typical, and atypical persons, as defined using fit statistics, were tested on 7408 persons entering AOD treatment using the Global Appraisal of Individual Needs (GAIN; Dennis et al., 2003a,b). RESULTS: Of those with suicide symptoms, the findings were as expected with the atypical group being higher on suicide and lower on symptoms of internalizing disorders. In addition, the atypical group was similar or lower on substance problems, symptoms of externalizing disorders, and crime and violence. CONCLUSIONS: Person fit statistics were useful in identifying persons with atypical suicide profiles and in enlightening aspects of existing theory concerning atypical suicidal ideation.