Assessing the quality of tests: revision of the EFPA review model / Evaluación de la calidad de los tests: revisión del modelo de evaluación de la EFPA

Background: Diverse national and international organizations have been developing projects for many years to improve testing practices. The main goal of this paper is to present the revised model of the European Federation of Psychologists’ Associations (EFPA) for the evaluation of the quality of tests. This model aims to provide test users with rigorous information about the theoretical, practical and psychometric characteristics of tests, in order to enhance their use. Method: For the revision of the test review model, an EFPA task force was established, consisting of six European experts from different countries, who worked on the update of the previous European model, adapting it to the recent developments in the field of psychological and educational measurement. Results: The updated EFPA model provides for the comprehensive evaluation of tests. The first part describes test characteristics exhaustively, and in the second part, a quantitative and narrative evaluation of the most relevant psychometric characteristics of tests is presented. Conclusions: A revision of the European model for the description and evaluation of psychological and educational tests is presented. The revised model is analyzed in light of recent developments in the field (AU)

Antecedentes: el objetivo de este trabajo es presentar una revisión del modelo de la Federación Europea de Asociaciones de Psicólogos (EFPA) para la evaluación de los tests. El modelo trata de poner a disposición de los usuarios información contrastada sobre las características teóricas, prácticas y psicométricas de los tests, facilitando con ello un mejor uso de las pruebas. Método: para llevar a cabo la revisión del modelo de evaluación de los tests se formó una comisión de trabajo en el seno de la EFPA formada por seis expertos de diferentes países que trabajaron en la actualización del modelo europeo previo, adaptándolo a los nuevos desarrollos en el ámbito de la evaluación psicológica y educativa. Resultados: la versión actualizada del modelo de la EFPA permite evaluar los tests de forma integral. En una primera parte se describe la prueba de forma exhaustiva, y en la segunda se lleva a cabo la evaluación cuantitativa y cualitativa de las características psicométricas de la prueba. Conclusiones: se presenta la revisión del modelo europeo para la descripción y evaluación de los tests y se comentan los resultados a la luz de los desarrollos recientes en el ámbito de la evaluación psicológica y educativa (AU)

Assessing the quality of tests: revision of the EFPA review model.

BACKGROUND: Diverse national and international organizations have been developing projects for many years to improve testing practices. The main goal of this paper is to present the revised model of the European Federation of Psychologists' Associations (EFPA) for the evaluation of the quality of tests. This model aims to provide test users with rigorous information about the theoretical, practical and psychometric characteristics of tests, in order to enhance their use. METHOD: For the revision of the test review model, an EFPA task force was established, consisting of six European experts from different countries, who worked on the update of the previous European model, adapting it to the recent developments in the field of psychological and educational measurement. RESULTS: The updated EFPA model provides for the comprehensive evaluation of tests. The first part describes test characteristics exhaustively, and in the second part, a quantitative and narrative evaluation of the most relevant psychometric characteristics of tests is presented. CONCLUSIONS: A revision of the European model for the description and evaluation of psychological and educational tests is presented. The revised model is analyzed in light of recent developments in the field.

Lack of consensus among competency ratings of the same occupation: noise or substance?

Although rating differences among incumbents of the same occupation have traditionally been viewed as error variance in the work analysis domain, such differences might often capture substantive discrepancies in how incumbents approach their work. This study draws from job crafting, creativity, and role theories to uncover situational factors (i.e., occupational activities, context, and complexity) related to differences among competency ratings of the same occupation. The sample consisted of 192 incumbents from 64 occupations. Results showed that 25% of the variance associated with differences in competency ratings of the same occupation was related to the complexity, the context, and primarily the nature of the occupation's work activities. Consensus was highest for occupations involving equipment-related activities and direct contact with the public.

Choosing the best method for local validity estimation: relative accuracy of meta-analysis versus a local study versus Bayes-analysis.

This study assessed the relative accuracy of 3 techniques--local validity studies, meta-analysis, and Bayesian analysis--for estimating test validity, incremental validity, and adverse impact in the local selection context. Bayes-analysis involves combining a local study with nonlocal (meta-analytic) validity data. Using tests of cognitive ability and personality (conscientiousness) as predictors, an empirically driven selection scenario illustrates conditions in which each of the 3 estimation techniques performs best. General recommendations are offered for how to estimate local parameters, based on true population variability and the number of studies in the meta-analytic prior. Benefits of empirical Bayesian analysis for personnel selection are demonstrated, and equations are derived to help guide the choice of a local validity technique (i.e., meta-analysis vs. local study vs. Bayes-analysis).

The Great Eight competencies: a criterion-centric approach to validation.

The author presents results of a meta-analysis of 29 validation studies (N=4,861) that uses the Great Eight competency factors (Kurz & Bartram, 2002) as the criterion measurement framework. Predictors of the Great Eight competencies based only on personality scales show moderate to good correlations with line-manager ratings for all 8 of the competencies. On their own, ability tests correlate with 4 of the 8 competencies, and together ability and personality data yield operational validities ranging from 0.20 to 0.44 for the 8 competencies. Operational validities for aggregated predictors with aggregated criteria were estimated to be 0.53. The value of differentiating the criterion space and of relating predictor variables to criterion variables in a one-to-one fashion is discussed.