Is the performance at the implicit association test sensitive to feedback presentation? A Rasch-based analysis.

The Implicit Association Test (IAT) is commonly used for the indirect assessment of psychological constructs. While the features of the IAT that might influence the performance of the respondents have been extensively investigated, the effect of informing the respondents about the correctness of their responses (i.e., feedback presentation) has been poorly addressed so far. The study addresses this issue by presenting an across-domain (implicit prejudice and food preference) Rasch-based analysis of IAT data obtained with and without feedback presentation. Results showed that speed was influenced by the interaction between feedback presentation and associative condition, whereas accuracy was influenced by the associative condition. This result varied across-domain. Results suggested that IATs administered with feedback presentation provide more accurate information on the construct of interest.

Does discrimination beat association in the IAT? The discrimination-association model reconceived.

The discrimination-association model (DAM; Stefanutti et al. 2013) disentangles two components underlying the responses to the implicit association test (IAT), which pertain to stimuli discrimination (the strength of the association of the stimuli with their own category) and automatic association (the strength of the association between targets and attributes). The assumption of the DAM that these two components sum into a single process generates critical drawbacks. The present work provides a new formulation of the model, called DAM-4C, in which stimuli discrimination and automatic association are separate, independent, and competing processes. Results of theoretical and simulation studies suggest that the DAM-4C outperforms the DAM. The IAT effect is found to vary with the association rates of the DAM-4C and not with those of the DAM. The parameters of the DAM-4C fitted on data from a Coca-Pepsi IAT are found to account for variance in brand attractiveness, taste preference, and cola choice that is not accounted for by the D score and the diffusion model. In addition, the association rates estimated on data from a Black-White IAT are in line with expectations.

Testing the actual equivalence of automatically generated items.

If the automatic item generation is used for generating test items, the question of how the equivalence among different instances may be tested is fundamental to assure an accurate assessment. In the present research, the question was dealt by using the knowledge space theory framework. Two different ways of considering the equivalence among instances are proposed: The former is at a deterministic level and it requires that all the instances of an item template must belong to exactly the same knowledge states; the latter adds a probabilistic level to the deterministic one. The former type of equivalence can be modeled by using the BLIM with a knowledge structure assuming equally informative instances; the latter can be modeled by a constrained BLIM. This model assumes equality constraints among the error parameters of the equivalent instances. An approach is proposed for testing the equivalence among instances, which is based on a series of model comparisons. A simulation study and an empirical application show the viability of the approach.

An evaluation system for postgraduate pediatric residency programs: report of a 3-year experience.

The way a postgraduate medical training program is organized and the capacity of faculty members to function as tutors and to organize effective professional experiences are among the elements that affect the quality of training. An evaluation system designed to target these elements has been implemented within the framework of the Pediatric Residency Program of the University of Padua (Italy). The aim of this report is to describe some aspects of the experience gained in the first 3 years of implementation of the system (2013-2015). Data were collected using four validated questionnaires: the "Resident Assessment Questionnaire", the "Tutor-Assessment Questionnaire", the "Rotation-Assessment Questionnaire", and the "Resident Affairs Committee-Assessment Questionnaire". The response rate was 72% for the "Resident Assessment Questionnaires"; 78% for the "Tutor-/Rotation-Assessment Questionnaires" and 84% for the "Resident Affair Committee-Assessment Questionnaires". The scores collected were validated by psychometric tests. CONCLUSION: The high rates of completed questionnaires returned and the psychometric validation of the results collected indicate that the evaluation system reported herein can be effectively implemented. Efforts should be made to refine this system and, more importantly, to document its impact in improving the Pediatric Residency Program. What is known: • The elements that influence the quality of postgraduate training programs and the knowledge, performance, and competences of residents must be regularly assessed. • Comprehensive evaluation systems for postgraduate residency programs are not universally implemented also because quite often common guidelines and rules, well-equipped infrastructures, and financial resources are missing. What is new: • We show the feasibility of implementing an evaluation system that targets some of the key elements of a postgraduate medical training program in Italy, a European country in which the regulations governing training programs and, notably, the evaluation of residents are still being developed.

Constructing tests for skill assessment with competence-based test development.

Competence-based test development is a recent and innovative method for the construction of tests that are as informative as possible about the competence state (the set of skills an individual has available) underlying the observed item responses. It finds application in different contexts, including the development of tests from scratch, and the improvement or shortening of existing tests. Given a fixed collection of competence states existing in a population of individuals and a fixed collection of competencies (each of which being the subset of skills that allow for solving an item), the competency deletion procedure results in tests that differ from each other in the competencies but are all equally informative about individuals' competence states. This work introduces a streamlined version of the competency deletion procedure that considers information necessary for test construction only, illustrates a straightforward way to incorporate test developer preferences about competencies into the test construction process, and evaluates the performance of the resulting tests in uncovering the competence states from the observed item responses.

PsycAssist: A Web-Based Artificial Intelligence System Designed for Adaptive Neuropsychological Assessment and Training.

Assessing executive functions in individuals with disorders or clinical conditions can be challenging, as they may lack the abilities needed for conventional test formats. The use of more personalized test versions, such as adaptive assessments, might be helpful in evaluating individuals with specific needs. This paper introduces PsycAssist, a web-based artificial intelligence system designed for neuropsychological adaptive assessment and training. PsycAssist is a highly flexible and scalable system based on procedural knowledge space theory and may be used potentially with many types of tests. We present the architecture and adaptive assessment engine of PsycAssist and the two currently available tests: Adap-ToL, an adaptive version of the Tower of London-like test to assess planning skills, and MatriKS, a Raven-like test to evaluate fluid intelligence. Finally, we describe the results of an investigation of the usability of Adap-ToL and MatriKS: the evaluators perceived these tools as appropriate and well-suited for their intended purposes, and the test-takers perceived the assessment as a positive experience. To sum up, PsycAssist represents an innovative and promising tool to tailor evaluation and training to the specific characteristics of the individual, useful for clinical practice.

A discrimination-association model for decomposing component processes of the implicit association test.

A formal model is proposed that decomposes the implicit association test (IAT) effect into three process components: stimuli discrimination, automatic association, and termination criterion. Both response accuracy and reaction time are considered. Four independent and parallel Poisson processes, one for each of the four label categories of the IAT, are assumed. The model parameters are the rate at which information accrues on the counter of each process and the amount of information that is needed before a response is given. The aim of this study is to present the model and an illustrative application in which the process components of a Coca-Pepsi IAT are decomposed.

Enhancing Computerized Adaptive Testing with Batteries of Unidimensional Tests.

The article presents a new computerized adaptive testing (CAT) procedure for use with batteries of unidimensional tests. At each step of testing, the estimate of a certain ability is updated on the basis of the response to the latest administered item and the current estimates of all other abilities measured by the battery. The information deriving from these abilities is incorporated into an empirical prior that is updated each time that new estimates of the abilities are computed. In two simulation studies, the performance of the proposed procedure is compared with that of a standard procedure for CAT with batteries of unidimensional tests. The proposed procedure yields more accurate ability estimates in fixed-length CATs, and a reduction of test length in variable-length CATs. These gains in accuracy and efficiency increase with the correlation between the abilities measured by the batteries.

Machine learning-decision tree classifiers in psychiatric assessment: An application to the diagnosis of major depressive disorder.

This work illustrates the advantages of using machine learning classifiers in psychiatric assessment. Machine learning-decision trees (ML-DTs) represent a new approach to scoring and interpreting psychodiagnostic test data that allows for increasing assessment accuracy and efficiency. The approach is outlined in an easy yet detailed way, and its application is illustrated on real psychodiagnostic test data. Specifically, cross-sectional data concerning nonclinical and clinical Japanese populations were taken from a panel registered with an internet survey company. Responses to the Patient Health Questionnaire-9 (PHQ-9) underwent receiver operating characteristic (ROC) curve, DSM algorithm, and ML-DT analyses. The results showed greater diagnostic accuracy for ML-DT (0.71-0.75) compared with the DSM algorithm (0.69) and ROC curves (0.70-0.71). Moreover, ML-DT enabled classifying participants as having or not having a diagnosis of depression using, on average, the information from 2.99 out of 9 items (SD = 1.35). The application showed that ML-DTs can provide information of high clinical value to integrate traditional psychometric methods. The resulting assessments are informative, accurate, and efficient.

Development of a scale for capturing psychological aspects of physical-digital integration: relationships with psychosocial functioning and facial emotion recognition.

The present work aims at developing a scale for the assessment of a construct that we called "physical-digital integration", which refers to the tendency of some individuals not to perceive a clear differentiation between feelings and perceptions that pertain to the physical or digital environment. The construct is articulated in four facets: identity, social relationships, time-space perception, and sensory perception. Data from a sample of 369 participants were collected to evaluate factor structure (unidimensional model, bifactor model, correlated four-factor model), internal consistency (Cronbach's α, McDonald's ω), and correlations of the physical-digital integration scale with other measures. Results showed that the scale is valid and internally consistent, and that both the total score and the scores at its four subscales are worthy of consideration. The physical-digital integration scores were found to be differently associated with digital and non-digital behaviors, individuals' ability to read emotions in the facial expressions of others, and indicators of psychosocial functioning (anxiety, depression, and satisfaction with social relationships). The paper proposes a new measure whose scores are associated with several variables that may have relevant consequences at both individual and social levels.

Assessing the local identifiability of probabilistic knowledge structures.

Given a collection Q of problems, in knowledge space theory Doignon & Falmagne, (International Journal of Man-Machine Studies 23:175-196, 1985) the knowledge state of a student is the collection K ⊆ Q of all problems that this student is capable of solving. A knowledge structure is a pair (Q, ), where is a collection of knowledge states that contains at least the empty set and Q. A probabilistic knowledge structure (PKS) is a knowledge structure (Q, , π), where π is a probability distribution on the knowledge states. The PKS that has received the most attention is the basic local independence model BLIM; Falmagne & Doignon, (British Journal of Mathematical and Statistical Psychology 41:1-23, 1988a, Journal of Mathematical Psychology 32:232-258, 1988b). To the best of our knowledge, systematic investigations in the literature concerning the identifiability of the BLIM are totally missing. Based on the theoretical work of Bamber and van Santen (Journal of Mathematical Psychology 29:443-473, 1985), the present article is aimed to present a method and a corresponding computerized procedure for assessing the local identifiability of the BLIM, which is applicable to any finite knowledge structure of moderate size.

COVID-19 emergency: the influence of implicit attitudes, information sources, and individual characteristics on psychological distress, intentions to get vaccinated, and compliance with restrictive rules.

BACKGROUND: To limit the spread of the COVID-19 emergency, a massive vaccination program was implemented and restrictive measures were imposed on the population. However, the propensity to adhere to the vaccination program has struggled to take off. Moreover, complying with the restrictive rules and maintaining social distancing have been highly distressing for many individuals. PARTICIPANTS AND PROCEDURE: Italian participants (N = 140, females = 65%, mean age = 29.50, SD = 10.80) were presented with an online survey consisting of multiple-choice questions and two single-category implicit association tests (SC-IATs). One SC-IAT evaluated the tendency of participants to automatically associate personal protective equipment (PPE) and vaccines with safety or danger; the other evaluated their tendency to automatically associate social situations with good or bad. Multiple-choice questions explored individual, social, and environmental factors that were expected to contribute to vaccine propensity, compliance with restrictive rules, and feelings of distress. RESULTS: Using scientific information sources was related to implicitly associating PPE and vaccines with safety, which in turn was associated with the propensity to get the vaccine. Moreover, being female, young, unsatisfied with social relationships, having suffered health and economic consequences due to the pandemic, and having negative implicit attitudes toward social situations contributed to increasing feelings of distress. CONCLUSIONS: Communication may contribute to individuals' behavior and preferences and it can also be associated with implicit attitudes, becoming consequently one of the main leverages to reduce vaccine hesitancy. Recovery programs should prioritize the development of interventions aimed at fostering psychological well-being through the enhancement of social contacts.

An Item Response Theory-Based Scoring of the South Oaks Gambling Screen-Revised Adolescents.

The South Oaks Gambling Screen-Revised Adolescent (SOGS-RA) is one of the most widely used screening tools for problem gambling among adolescents. In this study, item response theory was used for computing measures of problem gambling severity that took into account how much information the endorsed items provided about the presence of problem gambling. A zero-inflated mixture two-parameter logistic model was estimated on the responses of 4,404 adolescents to the South Oaks Gambling Screen-Revised Adolescent to compute the difficulty and discrimination of each item, and the problem gambling severity level (θ score) of each respondent. Receiver operating characteristic curve analysis was used to identify the cutoff on the θ scores that best distinguished daily and nondaily gamblers. This cutoff outperformed the common cutoff defined on the sum scores in identifying daily gamblers but fell behind it in identifying nondaily gamblers. When screening adolescents to be subjected to further investigations, the cutoff on the θ scores must be preferred to that on the sum scores.

Italian Diagnostic Adaptive Behavior Scale: Reliability and diagnostic accuracy compared with the Vineland-II.

BACKGROUND: The Diagnostic Adaptive Behavior Scale (DABS) is a short scale with excellent properties to assess the conceptual, social, and practical adaptive behavior domains for the diagnosis of intellectual disability (ID) in individuals aged 4-21 years. AIMS: Investigate the test-retest and inter-respondent reliability of the Italian adaptation of the DABS, verify its diagnostic accuracy in identifying individuals with ID and excluding individuals with typical development (TD), and compare its psychometric properties to those of the Vineland-II. METHODS: Test-retest reliability: The same respondent completed the Italian DABS for the same assessed person at two separate times (n = 71). Inter-respondent reliability: Two respondents for the same assessed person completed the Italian DABS independently (n = 57). Diagnostic accuracy: The same respondent completed the Italian DABS and Vineland-II for the same assessed person (n = 378; 50 % ID, 50 % TD). RESULTS: Italian DABS test-retest and inter-respondent correlation coefficients were excellent. Italian DABS sensitivity was 86 % and specificity was 99 %, Italian DABS Areas Under the ROC Curves were excellent (or good, practical skill domain), and comparable to the results reported for the Vineland-II. CONCLUSIONS: The Italian DABS is an excellent measure to evaluate the adaptive behavior for ID diagnosis; it is comparable to the Vineland-II but being shorter, the Italian DABS requires less time to administer.

Assessing learning processes with the gain-loss model.

Within the framework of knowledge space theory, a probabilistic skill multimap model for assessing learning processes is proposed. The learning process of a student is modeled as a function of the interaction between his competence state and the effect of a learning object on specific skills. Model parameters are initial probabilities of the skills, effects of learning objects on gaining and losing the skills, careless error, and lucky guess probabilities of the problems. A simulation study assessed model identifiability and goodness-of-recovery under several conditions. Practical implications of using the model are discussed, and the MATLAB code for simulating, estimating and testing it is available in the Psychonomic Society supplemental archive.

The many-facet Rasch model in the analysis of the go/no-go association task.

This article provides a many-facet Rasch measurement (MFRM) analysis of go/no-go association task (GNAT)-based measures of implicit attitudes toward sweet and salty food. We describe the statistical model and the strategy we adopted to score the GNAT, and we emphasize that, when analyzing implicit measures, MFRM indexes have to be interpreted in a peculiar way. In comparison with traditional scoring algorithms, an MFRM analysis of implicit measures provides some additional information and suffers from fewer limitations and assumptions. MFRM might help to overcome some limitations of current implicit measures, since it directly addresses some known issues and potential confounds, such as those related to a rational zero point, to the arbitrariness of the metric, and to participants' task-set switching ability.

New Perspectives in Computing the Point of Subjective Equality Using Rasch Models.

In psychophysics, the point of subject equality (PSE) is any of the points along a stimulus dimension at which a variable stimulus (visual, tactile, auditory, and so on) is judged by an observer to be equal to a standard stimulus. Rasch models have been found to offer a valid solution for computing the PSE when the method of constant stimuli is applied in the version of the method of transitions. The present work provides an overview of the procedures for computing the PSE using Rasch models and proposes some new developments. An adaptive procedure is described that allows for estimating the PSE of an observer without presenting him/her with all stimuli pairs. This procedure can be particularly useful in those situations in which psychophysical conditions of the individuals require that the number of trials is limited. Moreover, it allows for saving time that can be used to scrutinize the results of the experiment or to run other experiments. Also, the possibility of using Rasch-based fit statistics for identifying observers who gave unexpected judgments is explored. They could be individuals who, instead of carefully evaluating the presented stimuli pairs, gave random, inattentive, or careless responses, or gave the same response to many consecutive stimuli pairs. Otherwise, they could be atypical and clinically relevant individuals who deserve further investigation. The aforementioned developments are implemented using procedures and statistics that are well established in the framework of Rasch models. In particular, computerized adaptive testing procedures are used for efficiently estimating the PSE of the observers, whereas infit and outfit mean-squares statistics are used for detecting observers who gave unexpected judgments. Results of the analyses carried out on simulated data sets suggest that the proposed developments can be used in psychophysical experiments.

DscoreApp: A Shiny Web Application for the Computation of the Implicit Association Test D-Score.

Several options are available for computing the most common score for the Implicit Association Test, the so-called D-score. However, all these options come with some drawbacks, related to either the need for a license, for being tailored on a specific administration procedure, or for requiring a degree of familiarity with programming. By using the R shiny package, a user-friendly, interactive, and open source web application (DscoreApp) has been created for the D-score computation. This app provides different options for computing the D-score algorithms and for applying different cleaning criteria. Beyond making the D-score computation easier, DscoreApp offers the chance to have an immediate glimpse on the results and to see how they change according to different settings configurations. The resulting D-scores are immediately available and can be seen in easy-readable and interactive graphs, along with meaningful descriptive statistics. Graphical representations, data sets containing the D-scores, and other information on participants' performance are downloadable. In this work, the use of DscoreApp is illustrated on an empirical data set.

A Comparison of Classical and Modern Measures of Internal Consistency.

Three measures of internal consistency - Kuder-Richardson Formula 20 (KR20), Cronbach's alpha (α), and person separation reliability (R) - are considered. KR20 and α are common measures in classical test theory, whereas R is developed in modern test theory and, more precisely, in Rasch measurement. These three measures specify the observed variance as the sum of true variance and error variance. However, they differ for the way in which these quantities are obtained. KR20 uses the error variance of an "average" respondent from the sample, which overestimates the error variance of respondents with high or low scores. Conversely, R uses the actual average error variance of the sample. KR20 and α use respondents' test scores in calculating the observed variance. This is potentially misleading because test scores are not linear representations of the underlying variable, whereas calculation of variance requires linearity. Contrariwise, if the data fit the Rasch model, the measures estimated for each respondent are on a linear scale, thus being numerically suitable for calculating the observed variance. Given these differences, R is expected to be a better index of internal consistency than KR20 and α. The present work compares the three measures on simulated data sets with dichotomous and polytomous items. It is shown that all the estimates of internal consistency decrease with the increasing of the skewness of the score distribution, with R decreasing to a larger extent. Thus, R is more conservative than KR20 and α, and prevents test users from believing a test has better measurement characteristics than it actually has. In addition, it is shown that Rasch-based infit and outfit person statistics can be used for handling data sets with random responses. Two options are described. The first one implies computing a more conservative estimate of internal consistency. The second one implies detecting individuals with random responses. When there are a few individuals with a consistent number of random responses, infit and outfit allow for correctly detecting almost all of them. Once these individuals are removed, a "cleaned" data set is obtained that can be used for computing a less biased estimate of internal consistency.

Using Item Response Theory for the Development of a New Short Form of the Eysenck Personality Questionnaire-Revised.

The present work aims at developing a new version of the short form of the Eysenck Personality Questionnaire-Revised, which includes Psychoticism, Extraversion, Neuroticism, and Lie scales (48 items, 12 per scale). The work consists of two studies. In the first one, an item response theory model was estimated on the responses of 590 individuals to the full-length version of the questionnaire (100 items). The analyses allowed the selection of 48 items well discriminating and distributed along the latent continuum of each trait, and without misfit and differential item functioning. In the second study, the functioning of the new form of the questionnaire was evaluated in a different sample of 300 individuals. Results of the two studies show that reliability of the four scales is better than, or equal to that of the original forms. The new version outperforms the original one in approximating scores of the full-length questionnaire. Moreover, convergent validity coefficients and relations with clinical constructs were consistent with literature.