An exploratory study of the electronic medical record and outpatient vascular neurology consultations.

We carried out a quality improvement project utilizing the electronic medical record (EMR) to determine (1) the quality of vascular neurologists' recommendations for recurrent stroke prevention and (2) primary care provider (PCP) acknowledgement of the vascular neurologists' recurrent stroke prevention recommendations and their frequency of meeting the recommended metrics for risk factor control and lifestyle modification. We conducted a retrospective EMR chart review on a convenience sample of ischemic stroke patients during two epochs. Data collected included risk factors, stroke subtype, and process and outcome guidance metrics for recurrent ischemic stroke prevention according to American Heart Association/American Stroke Association (AHA/ASA) recommendations. Overall, vascular neurologists commonly recommended appropriate AHA/ASA risk factor management standards, but were less likely to do so for lifestyle management. Improvements in the EMR system over time, including the establishment of guideline-driven importable recurrent stroke prevention templates, led to a high frequency of proper risk factor and lifestyle recommendations made by vascular neurologists. Statistical analysis provided further evidence that the EMR positively influenced the delivery of proper recurrent stroke prevention guidance. Although PCPs infrequently acknowledged receipt of vascular neurology consultations, there was a relatively high frequency of achieved risk factor control. The latter may be attributed at least in part to pre-existent quality improvement programs implemented at primary care offices. Our exploratory findings suggest that proper use of the EMR may heighten efforts to provide appropriate and consistent recurrent stroke prevention recommendations in a primary care setting.

A Note on Comparing the Bifactor and Second-Order Factor Models: Is the Bayesian Information Criterion a Routinely Dependable Index for Model Selection?

This note demonstrates that the widely used Bayesian Information Criterion (BIC) need not be generally viewed as a routinely dependable index for model selection when the bifactor and second-order factor models are examined as rival means for data description and explanation. To this end, we use an empirically relevant setting with multidimensional measuring instrument components, where the bifactor model is found consistently inferior to the second-order model in terms of the BIC even though the data on a large number of replications at different sample sizes were generated following the bifactor model. We therefore caution researchers that routine reliance on the BIC for the purpose of discriminating between these two widely used models may not always lead to correct decisions with respect to model choice.

Factor structure of neurocognition and functional capacity in schizophrenia: a multidimensional examination of temporal stability.

Although neurocognition is commonly described in terms of different functional domains, some factor analytic studies have suggested a simpler dimensional structure for neuropsychological (NP) tests in patients with schizophrenia. Standardized tasks of everyday functioning, or tests of "functional capacity" (FC), are viewed differently from traditional NP tests, and are hence used as a co-primary measure in treatment studies. However, FC and NP tests have been found to be highly correlated. In fact, a recent study of ours suggested that performances on these different types of tasks constituted a single latent trait in a cross-sectional analysis. The current study examined the longitudinal factor structure of a combined set of NP and FC tests. Patients with schizophrenia (n = 195) were examined at two assessment occasions separated by periods ranging from 6 weeks to 6 months. Participants were assessed with the MATRICS Consensus Cognitive Battery (MCCB) and two performance-based assessments of FC. A single latent trait was extracted using full information maximum likelihood procedures, and its temporal stability was examined in terms of: stability of the latent trait scores, the inter-correlations of the three indicators of the latent trait, and the stability of loadings for the FC and NP items underlying the latent trait at the two measurement occasions. All indices of temporal stability were confirmed, with stability not related to follow-up duration. Variation in clinical symptoms and treatments across the measurement occasions was negligible. These findings raise the question of whether cognitive abilities measured by NP tests and FC instruments are tapping a single ability construct, which might have shared causal influences as well.

Evaluation of Polytomous Item Locations in Multicomponent Measuring Instruments: A Note on a Latent Variable Modeling Procedure.

This note is concerned with evaluation of location parameters for polytomous items in multiple-component measuring instruments. A point and interval estimation procedure for these parameters is outlined that is developed within the framework of latent variable modeling. The method permits educational, behavioral, biomedical, and marketing researchers to quantify important aspects of the functioning of items with ordered multiple response options, which follow the popular graded response model. The procedure is routinely and readily applicable in empirical studies using widely circulated software and is illustrated with empirical data.

On the Importance of Coefficient Alpha for Measurement Research: Loading Equality Is Not Necessary for Alpha's Utility as a Scale Reliability Index.

The population relationship between coefficient alpha and scale reliability is studied in the widely used setting of unidimensional multicomponent measuring instruments. It is demonstrated that for any set of component loadings on the common factor, regardless of the extent of their inequality, the discrepancy between alpha and reliability can be arbitrarily small in any considered population and hence practically ignorable. In addition, the set of parameter values where this discrepancy is negligible is shown to possess the same dimensionality as that of the underlying model parameter space. The article contributes to the measurement and related literature by pointing out that (a) approximate or strict loading identity is not a necessary condition for the utility of alpha as a trustworthy index of scale reliability, and (b) coefficient alpha can be a dependable reliability measure with any extent of inequality in the component loadings.

Estimating the Irreducible Uncertainty in Visual Diagnosis: Statistical Modeling of Skill Using Response Models.

BACKGROUND: For the representative problem of prostate cancer grading, we sought to simultaneously model both the continuous nature of the case spectrum and the decision thresholds of individual pathologists, allowing quantitative comparison of how they handle cases at the borderline between diagnostic categories. METHODS: Experts and pathology residents each rated a standardized set of prostate cancer histopathological images on the International Society of Urological Pathologists (ISUP) scale used in clinical practice. They diagnosed 50 histologic cases with a range of malignancy, including intermediate cases in which clear distinction was difficult. We report a statistical model showing the degree to which each individual participant can separate the cases along the latent decision spectrum. RESULTS: The slides were rated by 36 physicians in total: 23 ISUP pathologists and 13 residents. As anticipated, the cases showed a full continuous range of diagnostic severity. Cases ranged along a logit scale consistent with the consensus rating (Consensus ISUP 1: mean -0.93 [95% confidence interval {CI} -1.10 to -0.78], ISUP 2: -0.19 logits [-0.27 to -0.12]; ISUP 3: 0.56 logits [0.06-1.06]; ISUP 4 1.24 logits [1.10-1.38]; ISUP 5: 1.92 [1.80-2.04]). The best raters were able to meaningfully discriminate between all 5 ISUP categories, showing intercategory thresholds that were quantifiably precise and meaningful. CONCLUSIONS: We present a method that allows simultaneous quantification of both the confusability of a particular case and the skill with which raters can distinguish the cases. IMPLICATIONS: The technique generalizes beyond the current example to other clinical situations in which a diagnostician must impose an ordinal rating on a biological spectrum. HIGHLIGHTS: Question: How can we quantify skill in visual diagnosis for cases that sit at the border between 2 ordinal categories-cases that are inherently difficult to diagnose?Findings: In this analysis of pathologists and residents rating prostate biopsy specimens, decision-aligned response models are calculated that show how pathologists would be likely to classify any given case on the diagnostic spectrum. Decision thresholds are shown to vary in their location and precision.Significance: Improving on traditional measures such as kappa and receiver-operating characteristic curves, this specialization of item response models allows better individual feedback to both trainees and pathologists, including better quantification of acceptable decision variation.

On the Relationship Between Item Stem Formulation and Criterion Validity of Multiple-Component Measuring Instruments.

The possible dependency of criterion validity on item formulation in a multicomponent measuring instrument is examined. The discussion is concerned with evaluation of the differences in criterion validity between two or more groups (populations/subpopulations) that have been administered instruments with items having differently formulated item stems. The case of complex item stems involving two stimuli description sentences (double-barreled questions) is thereby compared with the setting where items contained a single sentence. Using empirical data, the latent criterion validity differences are evaluated across three groups that are randomly assigned to conditions characterized by item stems with differing number of stimuli. The results indicate that validity of an instrument can be influenced by the specific way item stem is formulated. Implications for empirical educational, behavioral, and social science research are discussed.

Design Effect in Multilevel Settings: A Commentary on a Latent Variable Modeling Procedure for Its Evaluation.

A latent variable modeling-based procedure is discussed that permits to readily point and interval estimate the design effect index in multilevel settings using widely circulated software. The method provides useful information about the relationship of important parameter standard errors when accounting for clustering effects relative to conducting single-level analyses. The approach can also be employed as an addendum to point and interval estimation of the intraclass correlation coefficient in empirical research. The discussed procedure makes it easily possible to evaluate the design effect in two-level studies by utilizing the popular latent variable modeling methodology and is illustrated with an example.

Evaluation of Second- and Third-Level Variance Proportions in Multilevel Designs With Completely Observed Populations: A Note on a Latent Variable Modeling Procedure.

Two- and three-level designs in educational and psychological research can involve entire populations of Level-3 and possibly Level-2 units, such as schools and educational districts nested within a given state, or neighborhoods and counties in a state. Such a design is of increasing relevance in empirical research owing to the growing popularity of large-scale studies in these and cognate disciplines. The present note discusses a readily applicable procedure for point-and-interval estimation of the proportions of second- and third-level variances in such multilevel settings, which may also be employed in model choice considerations regarding ensuing analyses for response variables of interest. The method is developed within the framework of the latent variable modeling methodology, is readily utilized with widely used software, and is illustrated with an example.

On Effect Size Measures for Nested Measurement Models.

A class of effect size indices are discussed that evaluate the degree to which two nested confirmatory factor analysis models differ from each other in terms of fit to a set of observed variables. These descriptive effect measures can be used to quantify the impact of parameter restrictions imposed in an initially considered model and are free from an explicit relationship to sample size. The described indices represent the extent to which respective linear combinations of the proportions of explained variance in the manifest variables are changed as a result of introducing the constraints. The indices reflect corresponding aspects of the impact of the restrictions and are independent of their statistical significance or lack thereof. The discussed effect size measures are readily point and interval estimated, using popular software, and their application is illustrated with numerical examples.

Behavior problems at ages 6 and 11 and high school academic achievement: longitudinal latent variable modeling.

Previous studies documented long-run effects of behavior problems at the start of school on academic achievement. However, these studies did not examine whether the observed effects of early behavior problems are explained by more proximate behavior problems, given the tendency of children's behavior problems to persist. Latent variable modeling was applied to estimate the effects of behavior problems at ages 6 and 11 on academic achievement at age 17, using data from a longitudinal study (n=823). Behavior problems at ages 6 and 11, each stage independently of the other, predicted lower math and reading test scores at age 17, controlling for intelligence quotient (IQ), birth weight, maternal characteristics, family and community environment, and taking into account behavior problems at age 17. Behavior problems at the start of school, independent of later behavior problems, exert lingering effects on achievement by impeding the acquisition of cognitive skills that are the foundation for later academic progress.

Evaluation of convergent and discriminant validity with multitrait-multimethod correlations.

A procedure for evaluation of convergent and discriminant validity coefficients is outlined. The method yields interval estimates of these coefficients in a construct validation study conducted via the multitrait-multimethod approach, as well as permitting examination of their population relationships. The procedure is readily employed in behavioural research using the increasingly popular latent variable modelling methodology. The method is illustrated with a numerical example.

Proportion of general factor variance in a hierarchical multiple-component measuring instrument: a note on a confidence interval estimation procedure.

A confidence interval construction procedure for the proportion of explained variance by a hierarchical, general factor in a multi-component measuring instrument is outlined. The method provides point and interval estimates for the proportion of total scale score variance that is accounted for by the general factor, which could be viewed as common to all components. The approach may also be used for testing composite (one-tailed) or simple hypotheses about this proportion, and is illustrated with a pair of examples.

On the Pitfalls of Estimating and Using Standardized Reliability Coefficients.

The population discrepancy between unstandardized and standardized reliability of homogeneous multicomponent measuring instruments is examined. Within a latent variable modeling framework, it is shown that the standardized reliability coefficient for unidimensional scales can be markedly higher than the corresponding unstandardized reliability coefficient, or alternatively substantially lower than the latter. Based on these findings, it is recommended that scholars avoid estimating, reporting, interpreting, or using standardized scale reliability coefficients in empirical research, unless they have strong reasons to consider standardizing the original components of utilized scales.

Evaluating Restrictive Models in Educational and Behavioral Research: Local Misfit Overrides Model Tenability.

The frequent practice of overall fit evaluation for latent variable models in educational and behavioral research is reconsidered. It is argued that since overall plausibility does not imply local plausibility and is only necessary for the latter, local misfit should be considered a sufficient condition for model rejection, even in the case of omnibus model tenability. The argument is exemplified with a comparison of the widely used one-parameter and two-parameter logistic models. A theoretically and practically relevant setting illustrates how discounting local fit and concentrating instead on overall model fit may lead to incorrect model selection, even if a popular information criterion is also employed. The article concludes with the recommendation for routine examination of particular parameter constraints within latent variable models as part of their fit evaluation.

Model Selection and Average Proportion Explained Variance in Exploratory Factor Analysis.

A procedure for evaluating the average R-squared index for a given set of observed variables in an exploratory factor analysis model is discussed. The method can be used as an effective aid in the process of model choice with respect to the number of factors underlying the interrelationships among studied measures. The approach is developed within the framework of exploratory structural equation modeling and is readily applicable with popular statistical software. The outlined procedure is illustrated using a numerical example.

Examining Multidimensional Measuring Instruments for Proximity to Unidimensional Structure Using Latent Variable Modeling.

A widely applicable procedure of examining proximity to unidimensionality for multicomponent measuring instruments with multidimensional structure is discussed. The method is developed within the framework of latent variable modeling and allows one to point and interval estimate an explained variance proportion-based index that may be considered a measure of proximity to unidimensional structure. The approach is readily utilized in educational, behavioral, and social research when it is of interest to evaluate whether a more general structure scale, test, or measuring instrument could be treated as being associated with an approximately unidimensional latent structure for some empirical purposes.

Proportion of third-level variation in multi-level studies: A note on an interval estimation procedure.

An interval estimation procedure is outlined that can be used for evaluating the proportion of observed variance in a response variable, which is due to the third level of nesting in a hierarchical design. The approach is also useful when it is of concern to address the necessity of including a third level in analyses of data from a multi-level study, relative to an alternative of proceeding with two-level modelling. The proposed method is illustrated with an empirical example.

Testing multivariate mean collinearity via latent variable modelling.

A procedure for testing mean collinearity in multidimensional spaces is outlined, which is applicable in settings with missing data and can be used when examining group mean differences. The approach is based on non-linear parameter restrictions and is developed within the framework of latent variable modelling. The method provides useful information about the constellation of multiple response centroids in the populations studied, and is illustrated with an example.

A method of bias correction for maximal reliability with dichotomous measures.

This paper is concerned with the reliability of weighted combinations of a given set of dichotomous measures. Maximal reliability for such measures has been discussed in the past, but the pertinent estimator exhibits a considerable bias and mean squared error for moderate sample sizes. We examine this bias, propose a procedure for bias correction, and develop a more accurate asymptotic confidence interval for the resulting estimator. In most empirically relevant cases, the bias correction and mean squared error correction can be performed simultaneously. We propose an approximate (asymptotic) confidence interval for the maximal reliability coefficient, discuss the implementation of this estimator, and investigate the mean squared error of the associated asymptotic approximation. We illustrate the proposed methods using a numerical example.