Building an intelligent recommendation system for personalized test scheduling in computerized assessments: A reinforcement learning approach.

The introduction of computerized formative assessments in the classroom has opened a new area of effective progress monitoring with more accessible test administrations. With computerized formative assessments, all students could be tested at the same time and with the same number of test administrations within a school year. Alternatively, the decision for the number and frequency of such tests could be made by teachers based on their observations and personal judgments about students. However, this often results in rigid test scheduling that fails to take into account the pace at which students acquire knowledge. To administer computerized formative assessments efficiently, teachers should be provided with systematic guidance regarding effective test scheduling based on each student's level of progress. In this study, we introduce an intelligent recommendation system that can gauge the optimal number and timing of testing for each student. We discuss how to build an intelligent recommendation system using a reinforcement learning approach. Then, we present a case study with a large sample of students' test results in a computerized formative assessment. We show that the intelligent recommendation system can significantly reduce the number of testing for the students by eliminating unnecessary test administrations where students do not show significant progress (i.e., growth). Also, the proposed recommendation system is capable of identifying the optimal test time for students to demonstrate adequate progress from one test administration to another. Implications for future research on personalized assessment scheduling are discussed.

A multi-method exploratory study of health professional students' experiences with compliance behaviours.

BACKGROUND: Research in healthcare, including students as participants, has begun to document experiences with negative compliance, specifically conformity and obedience. There is a growing body of experimental and survey literature, however, currently lacking is a direct measure of the frequency at which health professional students have negative experiences with conformity and obedience integrated with psychological factors, the outcomes of negative compliance, and students' perceptions. METHODS: To develop empirical knowledge about the frequency of negative compliance and student perceptions during health professional education a multi-methods survey approach was used. The survey was administered to health professional students across ten disciplines at four institutions. RESULTS: The results indicated students regularly experience obedience and conformity and are influenced by impression management and displacement of responsibility. Moral distress was identified as a consistent negative outcome. Student self-reported experiences aligned with the empirical findings. CONCLUSIONS: The findings of the present study demonstrate the pervasiveness of experiences with negative compliance during health professional's education along with some attendant psychological factors. The findings have educational and practical implications, as well as pointing to the need for further integration of social and cognitive psychology in explaining compliance in healthcare. The results are likely generalizable to a population level however replication is encouraged to better understand the true frequency of negative compliance at a health professional population level.

From experiences to expectations: A quantitative study on the fear of childbirth among multigravida women.

There is a strong relationship between women's previous birth experiences and their predetermined expectations. Childbirth expectations play an important role in the women's response to the birthing experience and postpartum period. The negative emotions and expectations, such as fear of childbirth, may result in negative experiences in subsequent childbirths. The aim of this study is to examine the relationship between previous birth experiences and the fear of childbirth in current pregnancy. A sample of 309 healthy women with normal pregnancies was recruited for this study. A chi-square test of independence and a multinomial logistic regression were used to explain the association between previous birth experiences and the fear of childbirth in current pregnancy. The level of childbirth fear that pregnant women feel appears to differ based upon their previous pregnancy experiences. Pregnant women who describe their previous births as happy and proud tend to experience a moderate level of childbirth fear about their current pregnancies, whereas pregnant women who remember their previous births as either fearful or painful have lower levels of childbirth fear. The level of childbirth fear tends to decrease as pregnant women have more children. Gestational week does not seem to be influential on the level of childbirth fear. According to the findings of this study, counterintuitively there is a negative relationship between the previous birth experience and childbirth fear. The level of childbirth fear is lower for pregnant women who remember their previous births as a negative experience compared to those who remember their previous births positively.

Procedures to develop a computerized adaptive test to assess patient-reported physical functioning.

PURPOSE: The purpose of this paper is to demonstrate the procedures to develop and implement a computerized adaptive patient-reported outcome (PRO) measure using secondary analysis of a dataset and items from fixed-format legacy measures. METHODS: We conducted secondary analysis of a dataset of responses from 1429 persons with work-related lower extremity impairment. We calibrated three measures of physical functioning on the same metric, based on item response theory (IRT). We evaluated efficiency and measurement precision of various computerized adaptive test (CAT) designs using computer simulations. RESULTS: IRT and confirmatory factor analyses support combining the items from the three scales for a CAT item bank of 31 items. The item parameters for IRT were calculated using the generalized partial credit model. CAT simulations show that reducing the test length from the full 31 items to a maximum test length of 8 items, or 20 items is possible without a significant loss of information (95, 99% correlation with legacy measure scores). CONCLUSIONS: We demonstrated feasibility and efficiency of using CAT for PRO measurement of physical functioning. The procedures we outlined are straightforward, and can be applied to other PRO measures. Additionally, we have included all the information necessary to implement the CAT of physical functioning in the electronic supplementary material of this paper.

The interactive effects of child maltreatment and the FK506 binding protein 5 gene (FKBP5) on dissociative symptoms in adolescence.

The FK506 binding protein 5 gene (FKBP5) has been associated with susceptibility to pathogenic effects of childhood trauma including dissociative symptoms. This study examines the impact of maltreatment on dissociative tendencies in adolescence as moderated by the FKBP5 gene. Dissociative symptoms and variation within FKBP5 were assessed in a high-risk, low socioeconomic status community sample of 279 maltreated and 171 nonmaltreated adolescents. Following the assignment of haplotypes across four single nucleotide polymorphisms (rs3800373, rs9296158, rs1360780, and rs9470080), individuals with one or more copies of the CATT haplotype (N = 230) were grouped together and compared to individuals with zero copies of this haplotype (N = 185). Analyses of covariance were conducted to test hypotheses regarding the effects of developmental timing and the chronicity of maltreatment and the CATT haplotype. We found a significant interactive effect of timing/chronicity of maltreatment and the CATT haplotype on dissociative symptoms. Among adolescents who had no copies of the CATT haplotype, dissociative symptoms were higher for chronically maltreated adolescents who had an infancy onset compared to those who were not maltreated or whose maltreatment experience was either relatively less chronic or not started in infancy. The groups did not differ significantly among subjects who carry one or more copies of the CATT haplotype.

Estimating Between-Person and Within-Person Subscore Reliability with Profile Analysis.

Subscores are of increasing interest in educational and psychological testing due to their diagnostic function for evaluating examinees' strengths and weaknesses within particular domains of knowledge. Previous studies about the utility of subscores have mostly focused on the overall reliability of individual subscores and ignored the fact that subscores should be distinct and have added value over the total score. This study introduces a profile reliability approach that partitions the overall subscore reliability into within-person and between-person subscore reliability. The estimation of between-person reliability and within-person reliability coefficients is demonstrated using subscores from number-correct scoring, unidimensional and multidimensional item response theory scoring, and augmented scoring approaches via a simulation study and a real data study. The effects of various testing conditions, such as subtest length, correlations among subscores, and the number of subtests, are examined. Results indicate that there is a substantial trade-off between within-person and between-person reliability of subscores. Profile reliability coefficients can be useful in determining the extent to which subscores provide distinct and reliable information under various testing conditions.

Psychometric Evaluation of the Wijma Delivery Expectancy/Experience Questionnaire Version B.

Examining the fear of delivery after childbirth is important. The authors' aim is to examine the overall psychometric quality of the Wijma Delivery Expectancy/Experience Questionnaire version B. Reliability and validity of the instrument were evaluated via reliability analysis, exploratory and confirmatory factor analyses, and multidimensional subscore estimation. A six-factor model was proposed to explain instrument results. Factors included concerns about labor pain, lack of positive behaviors, loneliness, lack of positive feelings, concerns about childbirth, and concerns about the baby. Researchers provide psychometric evidence about the quality of the questionnaire to measure fear of delivery after childbirth.

Psycho-Educational Assessments: Theory and Practice.

Psycho-educational assessments, such as intelligence tests, cognitive test batteries, and behavioral measures, serve as invaluable tools for school psychologists and educators [...].

A Psychometric Network Analysis of CHC Intelligence Measures: Implications for Research, Theory, and Interpretation of Broad CHC Scores "Beyond g".

For over a century, the structure of intelligence has been dominated by factor analytic methods that presume tests are indicators of latent entities (e.g., general intelligence or g). Recently, psychometric network methods and theories (e.g., process overlap theory; dynamic mutualism) have provided alternatives to g-centric factor models. However, few studies have investigated contemporary cognitive measures using network methods. We apply a Gaussian graphical network model to the age 9-19 standardization sample of the Woodcock-Johnson Tests of Cognitive Ability-Fourth Edition. Results support the primary broad abilities from the Cattell-Horn-Carroll (CHC) theory and suggest that the working memory-attentional control complex may be central to understanding a CHC network model of intelligence. Supplementary multidimensional scaling analyses indicate the existence of possible higher-order dimensions (PPIK; triadic theory; System I-II cognitive processing) as well as separate learning and retrieval aspects of long-term memory. Overall, the network approach offers a viable alternative to factor models with a g-centric bias (i.e., bifactor models) that have led to erroneous conclusions regarding the utility of broad CHC scores in test interpretation beyond the full-scale IQ, g.

Preschool Children's Loose Parts Play and the Relationship to Cognitive Development: A Review of the Literature.

Play is an integrative process, and the skills acquired in it-overcoming impulses, behavior control, exploration and discovery, problem-solving, reasoning, drawing conclusions, and attention to processes and outcomes are foundational cognitive structures that drive learning and motivation. Loose parts play is a prominent form of play that many scholars and educators explicitly endorse for cognitive development (e.g., divergent thinking, problem-solving). It is unique among play types because children can combine different play types and natural or manufactured materials in one occurrence. While educators and policymakers promote the benefits of loose parts play, no previous research has explored the direct relationship between preschool-age children's indoor loose parts play experiences and cognitive development. We address this gap by bringing together the relevant literature and synthesizing the empirical studies on common play types with loose parts, namely object and exploratory, symbolic and pretend, and constructive play. We also focus on studies that examine children's experiences through loose parts, highlighting the impact of different play types on learning through the reinforcement of cognitive skills, such as executive function, cognitive self-regulation, reasoning, and problem-solving. By examining the existing literature and synthesizing empirical evidence, we aim to deepen our understanding of the relationship between children's play with loose parts and its impact on cognitive development. Ultimately, pointing out the gaps in the literature that would add to the body of knowledge surrounding the benefits of play for cognitive development and inform educators, policymakers, and researchers about the significance of incorporating loose parts play into early childhood education.

Item Selection With Collaborative Filtering in On-The-Fly Multistage Adaptive Testing.

An important design feature in the implementation of both computerized adaptive testing and multistage adaptive testing is the use of an appropriate method for item selection. The item selection method is expected to select the most optimal items depending on the examinees' ability level while considering other design features (e.g., item exposure and item bank utilization). This study introduced collaborative filtering (CF) as a new method for item selection in the on-the-fly assembled multistage adaptive testing framework. The user-based CF (UBCF) and item-based CF (IBCF) methods were compared to the maximum Fisher information method based on the accuracy of ability estimation, item exposure rates, and item bank utilization under different test conditions (e.g., item bank size, test length, and the sparseness of training data). The simulation results indicated that the UBCF method outperformed the traditional item selection methods regarding measurement accuracy. Also, the IBCF method showed the most superior performance in terms of item bank utilization. Limitations of the current study and the directions for future research are discussed.

Rapid assessment of communication consistency: sentiment analysis of public health briefings during the COVID-19 pandemic.

Background: A key component of the initial public health response to the COVID-19 pandemic involved the use of mass media briefings led by public health officials to communicate updates during a time of great uncertainty and rapidly changing information. This study aims to examine the consistency of communications expressed during the public health briefings to generate novel insights about the type, direction, and strength of public health messages. The data source included 131 readily accessible public health briefings alongside the provincial and national new confirmed case counts during the first two waves of rapidly increasing cases during the pandemic in Alberta, Canada. We employed sentiment analysis as a text mining technique to explore the types and frequency of words in public health briefings conveying positive and negative sentiments. Using statistical analyses and data visualizations, we examined how public health messaging shifted with case trends. Results: Our findings indicate consistent public health messaging in terms of sentiments regardless of case count fluctuations, an association of specific words with conveying positive and negative sentiments, and a focus on particular message patterns at different points during the first two waves of the COVID-19 pandemic. Conclusion: Our findings demonstrate the practical implications and methodological advantages of using sentiment analysis as a data analytics tool for rapidly and objectively assessing the consistency of health communications during a public health crisis.

Conformity, obedience, and the Better than Average Effect in health professional students.

Background: Compliance, through conformity and obedience to authority, can produce negative outcomes for patient safety, as well as education. To date, educational interventions for dealing with situations of compliance or positive deviance have shown variable results. Part of the challenge for education on compliance may result from disparities between learners' expectations about their potential for engaging in positive deviance and the actual likelihood of engaging in positive deviance. More specifically, students may demonstrate a Better Than Average Effect (BTAE), the tendency for people to believe they are comparatively better than the average across a wide range of behaviours and skills. Methods: Four vignettes were designed and piloted using cognitive interviews, to investigate the BTAE. Conformity and obedience to authority were each addressed with two vignettes. The vignettes were included in a survey distributed to Canadian health professional students across multiple programs at several different institutions during the Winter 2019 semester. Self-evaluation of behaviour was investigated using a one-sample proportion test. Demographic data were investigated using logistic regression to identify predictors of the BTAE. Results: Participants demonstrated the BTAE for expected behaviour compared to peers for situations of conformity and obedience to authority. Age, sex, and program year were identified as potential predictors for exhibiting the BTAE. Conclusions: This study demonstrated that health professional students expect that they will behave better than average in compliance scenarios. Health professional students are not exempt from this cognitive bias in self-assessment. The results have implications for education on compliance, positive deviance, and patient safety.

Contexte: La conformité, par le biais du respect des normes et de l'obéissance à l'autorité, peut avoir des effets négatifs tant pour la sécurité des patients que pour l'éducation des médecins. À ce jour, les interventions éducatives portant sur la conformité ou la déviance positive se sont soldées par des résultats mitigés. Une des difficultés auxquelles se heurte l'enseignement de la conformité provient de l'écart entre les attentes des apprenants quant à la possibilité d'appliquer la déviance positive et la probabilité qu'ils adoptent réellement ce comportement. Plus précisément, les étudiants peuvent faire preuve de supériorité illusoire, c'est-à-dire la tendance à se croire supérieur à la moyenne des gens par rapport à un éventail de comportements et d'habiletés. Méthodes: Quatre vignettes, deux sur la conformité et deux sur l'obéissance à l'autorité, ont été conçues et mises à l'essai dans le cadre d'entretiens cognitifs afin d'étudier la présence du phénomène de supériorité illusoire. Les vignettes faisaient partie d'une enquête menée auprès d'étudiants de différents programmes en sciences de la santé offerts par divers établissements canadiens au cours de la session d'hiver 2019. L'auto-évaluation du comportement a été examinée à l'aide d'un test de proportion pour échantillon unique. Les données démographiques ont été étudiées à l'aide d'une régression logistique pour identifier les prédicteurs de la supériorité illusoire. Résultats: L'effet de supériorité illusoire par rapport aux pairs était présent chez les participants en ce qui concerne le comportement qu'on attend d'eux dans des situations en lien avec la conformité et l'obéissance à l'autorité. L'âge, le sexe et l'année de formation ont été repérés comme prédicteurs potentiels de la manifestation de supériorité illusoire. Conclusions: Cette étude montre que les étudiants en sciences de la santé s'attendent à avoir un comportement supérieur à la moyenne dans les situations où la conformité est en jeu. Ils ne sont pas exempts de ce biais cognitif dans leur auto-évaluation. Les résultats obtenus ont des implications pour l'éducation en matière de conformité, de déviance positive et de sécurité des patients.

An explanatory mixture IRT model for careless and insufficient effort responding in self-report measures.

Careless and insufficient effort responding (C/IER) on self-report measures results in responses that do not reflect the trait to be measured, thereby posing a major threat to the quality of survey data. Reliable approaches for detecting C/IER aid in increasing the validity of inferences being made from survey data. First, once detected, C/IER can be taken into account in data analysis. Second, approaches for detecting C/IER support a better understanding of its occurrence, which facilitates designing surveys that curb the prevalence of C/IER. Previous approaches for detecting C/IER are limited in that they identify C/IER at the aggregate respondent or scale level, thereby hindering investigations of item characteristics evoking C/IER. We propose an explanatory mixture item response theory model that supports identifying and modelling C/IER at the respondent-by-item level, can detect a wide array of C/IER patterns, and facilitates a deeper understanding of item characteristics associated with its occurrence. As the approach only requires raw response data, it is applicable to data from paper-and-pencil and online surveys. The model shows good parameter recovery and can well handle the simultaneous occurrence of multiple types of C/IER patterns in simulated data. The approach is illustrated on a publicly available Big Five inventory data set, where we found later item positions to be associated with higher C/IER probabilities. We gathered initial supporting validity evidence for the proposed approach by investigating agreement with multiple commonly employed indicators of C/IER.

Exploring Factors That Affected Student Well-Being during the COVID-19 Pandemic: A Comparison of Data-Mining Approaches.

COVID-19-related school closures caused unprecedented and prolonged disruption to daily life, education, and social and physical activities. This disruption in the life course affected the well-being of students from different age groups. This study proposed analyzing student well-being and determining the most influential factors that affected student well-being during the COVID-19 pandemic. With this aim, we adopted a cross-sectional study designed to analyze the student data from the Responses to Educational Disruption Survey (REDS) collected between December 2020 and July 2021 from a large sample of grade 8 or equivalent students from eight countries (n = 20,720), including Burkina Faso, Denmark, Ethiopia, Kenya, the Russian Federation, Slovenia, the United Arab Emirates, and Uzbekistan. We first estimated a well-being IRT score for each student in the REDS student database. Then, we used 10 data-mining approaches to determine the most influential factors that affected the well-being of students during the COVID-19 outbreak. Overall, 178 factors were analyzed. The results indicated that the most influential factors on student well-being were multifarious. The most influential variables on student well-being were students' worries about contracting COVID-19 at school, their learning progress during the COVID-19 disruption, their motivation to learn when school reopened, and their excitement to reunite with friends after the COVID-19 disruption.

Examining Humans' Problem-Solving Styles in Technology-Rich Environments Using Log File Data.

This study investigated how one's problem-solving style impacts his/her problem-solving performance in technology-rich environments. Drawing upon experiential learning theory, we extracted two behavioral indicators (i.e., planning duration for problem solving and human-computer interaction frequency) to model problem-solving styles in technology-rich environments. We employed an existing data set in which 7516 participants responded to 14 technology-based tasks of the Programme for the International Assessment of Adult Competencies (PIAAC) 2012. Clustering analyses revealed three problem-solving styles: Acting indicates a preference for active explorations; Reflecting represents a tendency to observe; and Shirking shows an inclination toward scarce tryouts and few observations. Explanatory item response modeling analyses disclosed that individuals with the Acting style outperformed those with the Reflecting or the Shirking style, and this superiority persisted across tasks with different difficulties.

Linguistic Influences on Cognitive Test Performance: Examinee Characteristics Are More Important than Test Characteristics.

Consideration of the influence of English language skills during testing is an understandable requirement for fair and valid cognitive test interpretation. Several professional standards and expert recommendations exist to guide psychologists as they attempt to engage in best practices when assessing English learners (ELs). Nonetheless, relatively few evidence-based recommendations for practice have been specified for psychologists. To address this issue, we used a mixed-effects modeling approach to examine the influences of test characteristics (i.e., test directions) and examinee characteristics (i.e., expressive and receptive language abilities) on cognitive test performance. Our results suggest that language abilities appear to have a significant influence on cognitive test performance, whereas test characteristics do not influence performance, after accounting for language abilities. Implications for practice include the assessment of expressive and receptive language abilities of EL students prior to administering, scoring, and interpreting cognitive test scores.

A Polytomous Scoring Approach to Handle Not-Reached Items in Low-Stakes Assessments.

In low-stakes assessments, some students may not reach the end of the test and leave some items unanswered due to various reasons (e.g., lack of test-taking motivation, poor time management, and test speededness). Not-reached items are often treated as incorrect or not-administered in the scoring process. However, when the proportion of not-reached items is high, these traditional approaches may yield biased scores and thereby threatening the validity of test results. In this study, we propose a polytomous scoring approach for handling not-reached items and compare its performance with those of the traditional scoring approaches. Real data from a low-stakes math assessment administered to second and third graders were used. The assessment consisted of 40 short-answer items focusing on addition and subtraction. The students were instructed to answer as many items as possible within 5 minutes. Using the traditional scoring approaches, students' responses for not-reached items were treated as either not-administered or incorrect in the scoring process. With the proposed scoring approach, students' nonmissing responses were scored polytomously based on how accurately and rapidly they responded to the items to reduce the impact of not-reached items on ability estimation. The traditional and polytomous scoring approaches were compared based on several evaluation criteria, such as model fit indices, test information function, and bias. The results indicated that the polytomous scoring approaches outperformed the traditional approaches. The complete case simulation corroborated our empirical findings that the scoring approach in which nonmissing items were scored polytomously and not-reached items were considered not-administered performed the best. Implications of the polytomous scoring approach for low-stakes assessments were discussed.

Age and Sex Invariance of the Woodcock-Johnson IV Tests of Cognitive Abilities: Evidence from Psychometric Network Modeling.

The Woodcock-Johnson IV Tests of Cognitive Abilities (WJ IV COG) is a comprehensive assessment battery designed to assess broad and narrow cognitive abilities, as defined by the Cattell-Horn-Carroll (CHC) theory of intelligence. Previous studies examined the invariance of the WJ assessments across sex and age groups using factor analytic methods. Psychometric network modeling is an alternative methodology that can address both direct and indirect relationships among the observed variables. In this study, we employed psychometric network modeling to examine the invariance of the WJ IV COG across sex and age groups. Using a normative sample (n = 4212 participants) representative of the United States population, we tested the extent to which the factorial structure of the WJ IV COG aligned with CHC theory for the school-aged sample. Next, we used psychometric network modeling as a data-driven method to investigate whether the network structure of the WJ IV COG remains similar across different sex and age (age 6 to 19, inclusively) groups. Our results showed that the WJ IV COG maintained the same network structure across all age and sex groups, although the network structure at younger ages indicated weaker relationships among some subtests. Overall, the results provide construct validity evidence for the WJ IV COG, based on both theoretical and data-driven methods.

Effect of Detailed OSCE Score Reporting on Learning and Anxiety in Medical School.

INTRODUCTION: There is growing literature on increasing feedback from Objective Structured Clinical Examinations (OSCEs) and one approach is a score report. The purpose of this study was to implement and evaluate a score report for a second and fourth-year medical school OSCE. METHODS: We developed an electronic OSCE score report that displayed comments and performance by domain within and across stations (checklist items and rating scales were tagged to each domain). Our initial pilot released the score report after pass/fail decisions but subsequent iterations released the score report the same day as the exam. Our evaluation approach included both student surveys and focus groups. RESULTS: Students felt the OSCE score report was accurate, identified strengths and weaknesses, and would likely cause them to take future action, with second-year students more likely to act on the report than fourth year students. The thematic analysis revealed barriers and enablers to utilizing feedback as well as the power of the score report to reduce anxiety. CONCLUSIONS: Our OSCE score report was simple to develop and implement the same day as an OSCE with an overall positive response from students with respect to accuracy and ability to use the information for future learning.