The influence of spatial frequency information on temporal synchrony perception on audiovisual stimuli.

Previous studies have shown that the spatial frequency (SF) of visual stimuli alters the perceived timing of subjective simultaneity. However, these studies have been limited to the effects of a single SF component. In this study, I measured and compared the points of subjective simultaneity (PSS) for audiovisual stimuli among low, high, and composited SF components. This experiment comprised a dual-presentation timing task and a ternary response format to eliminate response bias. The results indicated that the PSS value of the composition-SF stimuli was more toward visual-lead timing than the low-SF stimuli and did not differ significantly from that of the high-SF stimuli. The correlation coefficients showed that the PSS in composition-SF stimuli marginally approximated that of high-SF stimuli higher than that of low-SF stimuli. Future studies are needed to confirm these findings using visual stimuli with a wider range of SF components and with a modulated contrast.

Measuring Unipolar Traits With Continuous Response Items: Some Methodological and Substantive Developments.

In recent years, some models for binary and graded format responses have been proposed to assess unipolar variables or "quasi-traits." These studies have mainly focused on clinical variables that have traditionally been treated as bipolar traits. In the present study, we have made a proposal for unipolar traits measured with continuous response items. The proposed log-logistic continuous unipolar model (LL-C) is remarkably simple and is more similar to the original binary formulation than the graded extensions, which is an advantage. Furthermore, considering that irrational, extreme, or polarizing beliefs could be another domain of unipolar variables, we have applied this proposal to an empirical example of superstitious beliefs. The results suggest that, in certain cases, the standard linear model can be a good approximation to the LL-C model in terms of parameter estimation and goodness of fit, but not trait estimates and their accuracy. The results also show the importance of considering the unipolar nature of this kind of trait when predicting criterion variables, since the validity results were clearly different.

A 2PLM-RANK multidimensional forced-choice model and its fast estimation algorithm.

High-stakes non-cognitive tests frequently employ forced-choice (FC) scales to deter faking. To mitigate the issue of score ipsativity derived, many scoring models have been devised. Among them, the multi-unidimensional pairwise preference (MUPP) framework is a highly flexible and commonly used framework. However, the original MUPP model was developed for unfolding response process and can only handle paired comparisons. The present study proposes the 2PLM-RANK as a generalization of the MUPP model to accommodate dominance RANK format response. In addition, an improved stochastic EM (iStEM) algorithm is devised for more stable and efficient parameter estimation. Simulation results generally supported the efficiency and utility of the new algorithm in estimating the 2PLM-RANK when applied to both triplets and tetrads across various conditions. An empirical illustration with responses to a 24-dimensional personality test further supported the practicality of the proposed model. To further aid in the application of the new model, a user-friendly R package is also provided.

Response format changes the reading the mind in the eyes test performance of autistic and non-autistic adults.

LAY ABSTRACT: Recognizing and understanding the perspectives of others-also called theory of mind-is important for effective communication. Studies have found that some autistic individuals have greater difficulty with theory of mind compared to non-autistic individuals. One purported theory of mind measure is the Reading the Mind in the Eyes Test (RMET). This test presents participants with photographs of pairs of eyes and asks them to identify the emotion displayed by each pair of eyes from four choices. Some researchers have argued that the multiple-choice format of the RMET may not be an accurate measure of theory of mind, as participants could simply be guessing or using a process of elimination to select the correct answer. Participants may also be disadvantaged if they are not familiar with the specific emotion words used in the multiple-choice answers. We examined whether a free-report (open-ended) format RMET would be a more valid measure of theory of mind than the multiple-choice RMET. Autistic and non-autistic adults performed better on the multiple-choice RMET than the free-report RMET. However, both versions successfully differentiated autistic and non-autistic adults, irrespective of their level of verbal ability. Performance on both versions was also correlated with another well-validated adult measure of theory of mind. Thus, the RMET's multiple-choice format does not, of itself, appear to underpin its ability to differentiate autistic and non-autistic adults.

Effects of response format on achievement and aptitude assessment results: multi-level random effects meta-analyses.

Psychological achievement and aptitude tests are fundamental elements of the everyday school, academic and professional lives of students, instructors, job applicants, researchers and policymakers. In line with growing demands for fair psychological assessment tools, we aimed to identify psychometric features of tests, test situations and test-taker characteristics that may contribute to the emergence of test bias. Multi-level random effects meta-analyses were conducted to estimate mean effect sizes for differences and relations between scores from achievement or aptitude measures with open-ended (OE) versus closed-ended (CE) response formats. Results from 102 primary studies with 392 effect sizes revealed positive relations between CE and OE assessments (mean r = 0.67, 95% CI [0.57; 0.76]), with negative pooled effect sizes for the difference between the two response formats (mean d av = -0.65; 95% CI [-0.78; -0.53]). Significantly higher scores were obtained on CE exams. Stem-equivalency of items, low-stakes test situations, written short answer OE question types, studies conducted outside the United States and before the year 2000, and test-takers' achievement motivation and sex were at least partially associated with smaller differences and/or larger relations between scores from OE and CE formats. Limitations and the results' implications for practitioners in achievement and aptitude testing are discussed.

The negative footprint illusion in environmental impact estimates: Methodological considerations.

Past research has consistently shown that carbon footprint estimates of a set of conventional and more environmentally friendly items in combination tend to be lower than estimates of the conventional items alone. This 'negative footprint illusion' is a benchmark for the study of how cognitive heuristics and biases underpin environmentally significant behavior. However, for this to be a useful paradigm, the findings must also be reliable and valid, and an understanding of how methodological details such as response time pressure influence the illusion is necessary. Past research has cast some doubt as to whether the illusion is obtained when responses are made on a ratio/quantitative scale and when a within-participants design is used. Moreover, in past research on the negative footprint illusion, participants have had essentially as much time as they liked to make the estimates. It is yet unknown how time pressure influences the effect. This paper reports an experiment that found the effect when participants were asked to estimate the items' emissions in kilograms CO2 (a ratio scale) under high and under low time pressure, using a within-participants design. Thus, the negative footprint illusion seems to be a reliable and valid phenomenon that generalizes across methodological considerations and is not an artifact of specific details in the experimental setup.

Do different response formats affect how test takers approach a clinical reasoning task? An experimental study on antecedents of diagnostic accuracy using a constructed response and a selected response format.

The use of response formats in assessments of medical knowledge and clinical reasoning continues to be the focus of both research and debate. In this article, we report on an experimental study in which we address the question of how much list-type selected response formats and short-essay type constructed response formats are related to differences in how test takers approach clinical reasoning tasks. The design of this study was informed by a framework developed within cognitive psychology which stresses the importance of the interplay between two components of reasoning-self-monitoring and response inhibition-while solving a task or case. The results presented support the argument that different response formats are related to different processing behavior. Importantly, the pattern of how different factors are related to a correct response in both situations seem to be well in line with contemporary accounts of reasoning. Consequently, we argue that when designing assessments of clinical reasoning, it is crucial to tap into the different facets of this complex and important medical process.

A Comparison of Frequency- and Agreement-Based Response Formats in the Measurement of Burnout and Engagement.

The present research compares and contrasts frequency versus agreement response formats, two approaches to measuring job burnout and work engagement. Construct-based and measurement-based arguments for the superiority of the frequency response format in measuring burnout/engagement are provided, demonstrating that frequency-based measurements will explain relatively more variance in outcome variables. Fair comparison, time order counterbalance, and multiple measuring waves justify the comparison and reduce common method errors of self-report measures. Sample 1 (N = 242) was composed of employees from multiple organizations, while the participants in Sample 2 (N = 281) were employees from one company. Relative importance analysis showed that frequency outperforms the agreement response format in measuring burnout and engagement in both samples. These findings suggest that the frequency response format provides a more valuable method of detecting the dynamic nature of burnout/engagement, which offers methodological guidance for future research involving dynamic constructs. These findings can lead to improvements in the measurement of the dynamic experiences of burnout and engagement. This is one of the first studies to provide evidence whether the dynamic nature of the constructs would have any bearing on the response formats.

Using Harter and Likert Response Formats in Middle Childhood: A Comparison of Attachment Measures.

Self-report questionnaires based on Harter's response format ("Some kids . . . but other kids . . . ") are commonly used in developmental and clinical research settings, but the reliability and validity of this format in middle childhood are still under debate. The current study aimed to test the psychometric proprieties of Harter versus Likert response formats as applied to two attachment questionnaires in a sample of 410 Italian children aged 8 to 10 years. Participants completed the Experiences in Close Relationships-Revised Child version (n = 102, 4-point Likert-type scale; n = 104, adapted Harter version) and the Security Scale (n = 95, Harter's format; n = 109 adapted 4-point Likert version). Results of multigroup confirmatory factor analyses indicated that the two response formats exhibited comparable reliability and factorial validity, although a slight superiority of Harter's format emerged for the Security Scale. External validity was supported for both answer formats. Implications for developmental theory and practice are discussed.

Taking the Test Taker's Perspective: Response Process and Test Motivation in Multidimensional Forced-Choice Versus Rating Scale Instruments.

The multidimensional forced-choice (MFC) format has been proposed as an alternative to the rating scale (RS) response format. However, it is unclear how changing the response format may affect the response process and test motivation of participants. In Study 1, we investigated the MFC response process using the think-aloud technique. In Study 2, we compared test motivation between the RS format and different versions of the MFC format (presenting 2, 3, 4, and 5 items simultaneously). The response process to MFC item blocks was similar to the RS response process but involved an additional step of weighing the items within a block against each other. The RS and MFC response format groups did not differ in their test motivation. Thus, from the test taker's perspective, the MFC format is somewhat more demanding to respond to, but this does not appear to decrease test motivation.

Prediction of performance by non-cognitive traits.

Project A clearly demonstrated that performance is multidimensional and that some aspects are better predicted by noncognitive measures. Substantial research and development in the ensuing years has focused on personality and vocational interests. The articles in this special issue convincingly demonstrate that at least one personality measure developed by military researchers, the Tailored Adaptive Personality Assessment System (TAPAS), is resistant to faking, which was an important concern about earlier single statement instruments. Moreover, several articles report showing that TAPAS predicts retention and important aspects of "will do" performance. On the other hand, these papers show that TAPAS adds little or no incremental validity to "can do" aspects of performance over and above the Armed Services Vocational Aptitude Battery (ASVAB). Three measures of vocational interest are described in articles in this special issue and research has been positive about their ability to predict attrition, rates of promotion and reenlistment, and job satisfaction. A number of topics for further research are noted.

Response Format, Not Semantic Activation, Influences the Failed Retrieval Effect.

In educational settings, tests are mainly used to measure the extent to which learners' knowledge and skill have been acquired. However, the act of taking a test also promotes learning itself. In particular, making errors on tests (i.e., searching for erroneous information) promotes learning. This is called the "failed retrieval effect" (FRE) and has been the subject of considerable study. Previous research shows that enhanced learning does not occur if feedback correcting an error is delayed. This is attributed to the relative absence of activated information. In this study, we manipulated both the amount of information to be retrieved prior to learning and the delay time until feedback is given to investigate their effects on learning. As a result, even when multiple incorrect answers were given to increase the degree of semantic activation, learning was not promoted beyond that found with traditional procedures that rely on only one incorrect response. The timing of feedback (immediate, short-delay, long-delay) also did not impact FRE. However, the manipulation of response format for erroneous information resulted in degraded performance when responses were typed and feedback was delayed. Based on this result, we suggested that the failed retrieval effect was not affected by semantic activation at the time of retrieval but was affected by response format. Moreover, the processing necessary for typing may affect FRE under the delayed feedback condition.

Comparison of Single-Response Format and Forced-Choice Format Instruments Using Thurstonian Item Response Theory.

One of the most cited methodological issues is with the response format, which is traditionally a single-response Likert response format. Therefore, our study aims to elucidate and illustrate an alternative response format and analytic technique, Thurstonian item response theory (IRT), for analyzing data from surveys using an alternate response format, the forced-choice format. Specifically, we strove to give a thorough introduction of Thurstonian IRT at a more elementary level than previous publications in order to widen the possible audience. This article presents analyses and comparison of two versions of a self-report scale, one version using a single-response format and the other using a forced-choice format. Drawing from lessons learned from our study and literature, we present a number of recommendations for conducting research using the forced-choice format and Thurstonian IRT, as well as suggested avenues for future research.

To bin or not to bin? A comparison of symptom frequency response formats in the assessment of health-related quality of life.

PURPOSE: The goal of this study is to compare three different types of retrospective frequency response formats on the Healthy Days Symptoms Module (HDSM). Responses are compared in terms of intra-individual consistency, psychometric value, and participant feedback about each type of response format. METHODS: Respondents each completed three versions of the HDSM, where items were framed to elicit an open-ended frequency, a fixed choice frequency, or a vague quantifier response. Traditional reliability statistics were used to evaluate intra-individual consistency. Differential item functioning (DIF) was used to test for response format effects, and item response theory (IRT) scale scores and standard errors were computed across the three forms to compare psychometric value. Linear mixed modeling was used to examine the associations of IRT scale scores across response formats with respondent characteristics. RESULTS: People are largely consistent in how they respond to items about their health, regardless of the response format, and no DIF was detected between response formats. The IRT scores computed from the "# of days" frequency response formats tend to have better measurement precision than those from vague quantifiers. Open-ended frequencies capture a greater span of individual differences for people reporting fewer symptoms; however, little measurement precision is lost in collapsing the frequencies into categories. CONCLUSIONS: Both the open-ended and fixed choice frequency response formats offer more measurement precision than vague quantifiers. While the open-ended frequency response format may capture more individual differences, respondents tend to report more difficulty with exact frequency recall, and thus, prefer the fixed choice frequency format.

Five-Factor Model Best Describes Narcissistic Personality Inventory Across Different Item Response Formats.

The objective of this research was to validate the Narcissistic Personality Inventory across different response formats, given that several factor structures were proposed, ranging from two to seven factors. The original forced-choice format of the Narcissistic Personality Inventory was given to 410 participants and a modified, i.e., Likert format was given to 423 participants from the general population, along with personality and other narcissism measures. The results showed that the five-factor model proposed by Ackerman et al. had the best model fit in both response formats and that a distinction between adaptive (Leadership, Vanity, and Superiority) and some aspects of maladaptive (Manipulativeness and Exhibitionism) narcissism factors could be established. However, the redundancy of items in certain factors could be problematic and further improvements of the Narcissistic Personality Inventory should include more indicators of some proposed factors, especially of Vanity.

Influence of response instructions and response format on applicant perceptions of a situational judgement test for medical school selection.

BACKGROUND: This study examined the influence of two Situational Judgement Test (SJT) design features (response instructions and response format) on applicant perceptions. Additionally, we investigated demographic subgroup differences in applicant perceptions of an SJT. METHODS: Medical school applicants (N = 372) responded to an online survey on applicant perceptions, including a description and two example items of an SJT. Respondents randomly received one of four SJT versions (should do-rating, should do-pick-one, would do-rating, would do-pick-one). They rated overall favourability and items on four procedural justice factors (face validity, applicant differentiation, study relatedness and chance to perform) and ease-of-cheating. Additionally, applicant perceptions were compared for subgroups based on gender, ethnic background and first-generation university status. RESULTS: Applicants rated would-do instructions as easier to cheat than should-do instructions. Rating formats received more favourable judgements than pick-one formats on applicant differentiation, study-relatedness, chance to perform and ease of cheating. No significant main effect for demographic subgroup on applicant perceptions was found, but significant interaction effects showed that certain subgroups might have more pronounced preferences for certain SJT design features. Specifically, ethnic minority applicants - but not ethnic majority applicants - showed greater preference for should-do than would-do instructions. Additionally, first-generation university students - but not non-first-generation university students - were more favourable of rating formats than of pick-one formats. CONCLUSIONS: Findings indicate that changing SJT design features may positively affect applicant perceptions by promoting procedural justice factors and reducing perceived ease of cheating and that response instructions and response format can increase the attractiveness of SJTs for minority applicants.

The Indecision Model of Psychophysical Performance in Dual-Presentation Tasks: Parameter Estimation and Comparative Analysis of Response Formats.

Psychophysical data from dual-presentation tasks are often collected with the two-alternative forced-choice (2AFC) response format, asking observers to guess when uncertain. For an analytical description of performance, psychometric functions are then fitted to data aggregated across the two orders/positions in which stimuli were presented. Yet, order effects make aggregated data uninterpretable, and the bias with which observers guess when uncertain precludes separating sensory from decisional components of performance. A ternary response format in which observers are also allowed to report indecision should fix these problems, but a comparative analysis with the 2AFC format has never been conducted. In addition, fitting ternary data separated by presentation order poses serious challenges. To address these issues, we extended the indecision model of psychophysical performance to accommodate the ternary, 2AFC, and same-different response formats in detection and discrimination tasks. Relevant issues for parameter estimation are also discussed along with simulation results that document the superiority of the ternary format. These advantages are demonstrated by fitting the indecision model to published detection and discrimination data collected with the ternary, 2AFC, or same-different formats, which had been analyzed differently in the sources. These examples also show that 2AFC data are unsuitable for testing certain types of hypotheses. matlab and R routines written for our purposes are available as Supplementary Material, which should help spread the use of the ternary format for dependable collection and interpretation of psychophysical data.

La medida de la comprensión emocional con el modelo de Rasch

El objetivo fue construir ítems de comprensión emocional desde la perspectiva del Modelo de Rasch, comparando mediante un experimento aleatorizado los formatos de respuesta verbal/imagen. Participaron 204 sujetos de población general. Se contrastó el efecto del formato (verbal/imagen) y del género en las medidas Rasch de comprensión de emociones de los participantes y se analizó el efecto de la distancia social (cerca/lejos) sobre la dificultad Rasch de los ítems. El formato de respuesta tuvo un efecto significativo sobre las medidas independientemente del género: el formato verbal resultó más fácil que el de imagen. El efecto del género fue significativo, favorable a las mujeres. Los ítems que describían situaciones con receptores cercanos resultaron más fáciles que los que incluían receptores lejanos.

The aim of this study was to construct emotion understanding items from the Rasch Model approach by experimenting through comparison between verbal /image response format. The participants were 204 subjects from a Spanish community sample. A randomized experiment was carried out to test the effect of response format (verbal/ image) and participants' gender on the emotion understanding Rasch measurements. The effect on item difficulty of the social distance (close/far) was also contrasted. No interaction effect was found. Response format had a significant effect on measurement regardless of gender: the verbal response format was easier than the image one. There were significant gender differences on emotion understanding favoring women. Items describing situations with close receivers were significantly easier than the items showing far recipients.

The Effect of Response Format on the Psychometric Properties of the Narcissistic Personality Inventory: Consequences for Item Meaning and Factor Structure.

The Narcissistic Personality Inventory (NPI) is currently the most widely used measure of narcissism in social/personality psychology. It is also relatively unique because it uses a forced-choice response format. We investigate the consequences of changing the NPI's response format for item meaning and factor structure. Participants were randomly assigned to one of three conditions: 40 forced-choice items (n = 2,754), 80 single-stimulus dichotomous items (i.e., separate true/false responses for each item; n = 2,275), or 80 single-stimulus rating scale items (i.e., 5-point Likert-type response scales for each item; n = 2,156). Analyses suggested that the "narcissistic" and "nonnarcissistic" response options from the Entitlement and Superiority subscales refer to independent personality dimensions rather than high and low levels of the same attribute. In addition, factor analyses revealed that although the Leadership dimension was evident across formats, dimensions with entitlement and superiority were not as robust. Implications for continued use of the NPI are discussed.

The effect of forced choice on facial emotion recognition: a comparison to open verbal classification of emotion labels.

OBJECTIVE: This article includes the examination of potential methodological problems of the application of a forced choice response format in facial emotion recognition. METHODOLOGY: 33 subjects were presented with validated facial stimuli. The task was to make a decision about which emotion was shown. In addition, the subjective certainty concerning the decision was recorded. RESULTS: The detection rates are 68% for fear, 81% for sadness, 85% for anger, 87% for surprise, 88% for disgust, and 94% for happiness, and are thus well above the random probability. CONCLUSION: This study refutes the concern that the use of forced choice formats may not adequately reflect actual recognition performance. The use of standardized tests to examine emotion recognition ability leads to valid results and can be used in different contexts. For example, the images presented here appear suitable for diagnosing deficits in emotion recognition in the context of psychological disorders and for mapping treatment progress.