Optimizing a national examination for medical undergraduates via modern automated test assembly approaches.

BACKGROUND: Automated test assembly (ATA) represents a modern methodology that employs data science optimization on computer platforms to automatically create test form, thereby significantly improving the efficiency and accuracy of test assembly procedures. In the realm of medical education, large-scale high-stakes assessments often necessitate lengthy tests, leading to elevated costs in various dimensions (such as examinee fatigue and expenses associated with item development). This study aims to augment the design of the medical education assessments by leveraging modern ATA approaches. METHODS: To achieve the objective, a four-step process employing psychometric methodologies was used to calibrate and analyze the item pool of the Standardized Competence Test for Clinical Medicine Undergraduates (SCTCMU), a nationwide summative test comprising 300 multiple-choice questions (MCQ) in China. Subsequently, two modern ATA approaches were employed to determine the optimal item combination, accounting for both statistical and content requirements specified in the test blueprint. The qualities of the assembled test form, generated using modern ATA approaches, underwent meticulous evaluation. RESULTS: Through an exploration of the psychometric properties of the SCTCMU as a foundational step, the evaluation revealed commendable quality in the item properties. Furthermore, the evaluation of the quality of assembled test form using modern ATA approaches indicated the ability to ascertain the optimal test length within the predefined measurement precision. Specifically, this investigation demonstrates that the application of modern ATA approaches can substantially reduce the test length of assembled test form, while simultaneously maintaining the required statistical and content standards specified in the test blueprint. CONCLUSIONS: This study harnessed modern ATA approaches to facilitate the automatic construction of test form, thereby significantly enhancing the efficiency and precision of test assembly procedures. The utilization of modern ATA approaches offers medical educators a valuable tool to enhance the efficiency and cost-effectiveness of medical education assessment.

Exploring Multiple-Objective Optimization for Efficient and Effective Test Paper Design with Dynamic Programming Guided Genetic Algorithm.

Automatic test paper design is critical in education to reduce workloads for educators and facilitate an efficient teaching process. However, current designs fail to satisfy the realistic teaching requirements of educators, including the consideration of both test quality and efficiency. This is the main reason why teachers still manually construct tests in most teaching environments. In this paper, the quality of tests is quantitatively defined while considering multiple objectives, including a flexible coverage of knowledge points, cognitive levels, and question difficulty. Then, a model based on the technique of linear programming is delicately designed to explore the optimal results for this newly defined problem. However, this technique is not efficient enough, which cannot obtain results in polynomial time. With the consideration of both test quality and generation efficiency, this paper proposes a genetic algorithm (GA) based method, named dynamic programming guided genetic algorithm with adaptive selection (DPGA-AS). In this method, a dynamic programming method is proposed in the population initialization part to improve the efficiency of the genetic algorithm. An adaptive selection method for the GA is designed to avoid prematurely falling into the local optimal for better test quality. The question bank used in our experiments is assembled based on college-level calculus questions from well-known textbooks. The experimental results show that the proposed techniques can construct test papers with both high effectiveness and efficiency. The computation time of the test assembly problem is reduced from 3 hours to 2 seconds for a 5000-size question bank as compared to a linear programming model with similar test quality. The test quality of the proposed method is better than the other baselines.

Estimating and Using Block Information in the Thurstonian IRT Model.

Multidimensional forced-choice (MFC) tests are increasing in popularity but their construction is complex. The Thurstonian item response model (Thurstonian IRT model) is most often used to score MFC tests that contain dominance items. Currently, in a frequentist framework, information about the latent traits in the Thurstonian IRT model is computed for binary outcomes of pairwise comparisons, but this approach neglects stochastic dependencies. In this manuscript, it is shown how to estimate Fisher information on the block level. A simulation study showed that the observed and expected standard errors based on the block information were similarly accurate. When local dependencies for block sizes [Formula: see text] were neglected, the standard errors were underestimated, except with the maximum a posteriori estimator. It is shown how the multidimensional block information can be summarized for test construction. A simulation study and an empirical application showed small differences between the block information summaries depending on the outcome considered. Thus, block information can aid the construction of reliable MFC tests.

A mobile APP-based, customizable automated device for self-administered olfactory testing and an implementation of smell identification test.

Olfactory tests are used for the evaluation of ability to detect and identify common odors in humans psychophysically. Olfactory tests are currently administered by professionals with a set of given odorants. Manual administration of such tests can be labor and cost intensive and data collected as such are confounded with experimental variables, which adds personnel costs and introduces potential errors and data variability. For large-scale and longitudinal studies, manually recorded data must be collected and compiled from multiple sites. It is difficult to standardize the way data are collected and recorded. There is a need for a computerized smell test system for psychophysical and clinical applications. A mobile digital olfactory testing system (DOTS) was developed, consisting of an odor delivery system (DOTS-ODD) and a mobile application program (DOTS-APP) connected wirelessly. The University of Pennsylvania Smell Identification Test was implemented in DOTS and compared to its commercial product on a cohort of 80 normosmic subjects and a clinical cohort of 12 Parkinson's disease patients. A test-retest was conducted on 29 subjects of the normal cohort. The smell identification scores obtained from the DOTS and standard UPSIT commercial test are highly correlated (r = 0.714, P < 0.001), and test-retest reliability coefficient was 0.807 (r = 0.807, P < 0.001). The DOTS is customizable and mobile compatible, which allows for the implementation of standardized olfactory tests and the customization of investigators' experimental paradigms. The DOTS-APP on mobile devices offers capabilities for a broad range of on-site, online, or remote clinical and scientific chemosensory applications.

A Novel System to Increase Yield of Manufacturing Test of an RF Transceiver through Application of Machine Learning.

Electronic manufacturing and design companies maintain test sites for a range of products. These products are designed according to the end-user requirements. The end user requirement, then, determines which of the proof of design and manufacturing tests are needed. Test sites are designed to carry out two things, i.e., proof of design and manufacturing tests. The team responsible for designing test sites considers several parameters like deployment cost, test time, test coverage, etc. In this study, an automated test site using a supervised machine learning algorithm for testing an ultra-high frequency (UHF) transceiver is presented. The test site is designed in three steps. Firstly, an initial manual test site is designed. Secondly, the manual design is upgraded into a fully automated test site. And finally supervised machine learning is applied to the automated design to further enhance the capability. The manual test site setup is required to streamline the test sequence and validate the control and measurements taken from the test equipment and unit under test (UUT) performance. The manual test results showed a high test time, and some inconsistencies were observed when the test operator was required to change component values to tune the UUT. There was also a sudden increase in the UUT quantities and so, to cater for this, the test site is upgraded to an automated test site while the issue of inconsistencies is resolved through the application of machine learning. The automated test site significantly reduced test time per UUT. To support the test operator in selecting the correct component value the first time, a supervised machine learning algorithm is applied. The results show an overall improvement in terms of reduced test time, increased consistency, and improved quality through automation and machine learning.

Quantitative analysis of different respiratory specimens on two automated test systems for detection of SARS-CoV-2 RNA.

Molecular testing of SARS-CoV-2 RNA is essential during the pandemic. Here, we compared the results of different respiratory specimens including anterior nasal swabs, pharyngeal swabs, saliva swabs, and gargle lavage samples to nasopharyngeal swabs on two automated SARS-CoV-2 test systems. Samples were collected and tested simultaneously from a total of 36 hospitalized symptomatic COVID-19 patients. Detection and quantification of SARS-CoV-2 was performed on cobas®6800 (Roche) and NeuMoDx™ (Qiagen) systems. Both assays showed reliable detection and quantification of SARS-CoV-2 RNA, with nasopharyngeal swabs showing the highest sensitivity. SARS-CoV-2 RNA concentrations in other respiratory specimens were lower (mean 2.5 log10 copies/ml) or even undetectable in up to 20%. These data clearly indicate that not all respiratory materials are equally suitable for the management of hospitalized patients, especially, in the late phase of COVID-19, when the viral phase subsides and inflammation becomes the predominant factor, making detection of even lower viral loads increasingly important.

A Short Measure of Acceptability of Intimate Partner Violence Against Women: Development and Validation of the A-IPVAW-8 Scale.

Attitudes of acceptability of intimate partner violence against women (IPVAW) are considered one of the main risk factors of this type of violence. The aim of this study is to develop and validate a short version of the acceptability of IPVAW scale, the A-IPVAW-8, for large scale studies where space and time are limited. A panel of experts were asked to assess item content validity. Two samples were recruited to assemble an 8-item short version of the scale using automated test assembly, and to reassess the psychometric properties of the A-IPVAW-8 in an independent sample. Results showed that the A-IPVAW-8 had adequate internal consistency (α = .72-.76, ω = .73-.81), a stable one-factor latent structure (comparative fit index [CFI] = 0.94, Tucker-Lewis index = 0.92, root mean square error of approximation = 0.077), validity evidences based on its relationships to other variables in both samples, and was also invariant across gender (ΔCFI < |0.02|). This study provides a short, easy-to-use tool to evaluate attitudes of acceptability of IPVAW for large scale studies.

Static and Dynamic Verification of Space Systems Using Asynchronous Observer Agents.

Formal verification of distributed systems is essential, especially in mission-critical systems that cannot be restarted. Such are space systems in which satellites read sensor values and autonomously make actuator decisions based on them, and ground services only set general patterns of behavior. The verification formalism should correspond to the essential characteristics of a distributed system, such as node autonomy and asynchrony of actions and communication, as in our Integrated Model of Distributed Systems (IMDS). It is also crucial that the formalism allows for finding partial deadlocks and checking partial termination, where only a subset of the system nodes is involved while the rest can perform their own tasks at the same time. This article presents the idea of using monitoring agents-observers prepared in the IMDS formalism. Observers check the state of individual system components by polling, allowing verification without knowing the global state of the system. Such an agent is an ideal prototype of a runtime observer that checks if the actual operation of the system corresponds to a design that has previously been proven correct.

Automated Test Assembly for Multistage Testing With Cognitive Diagnosis.

Computer multistage adaptive test (MST) combines the advantages of paper and pencil-based test (P&P) and computer-adaptive test (CAT). As CAT, MST is adaptive based on modules; as P&P, MST can meet the need of test developers to manage test forms and keep test forms parallel. Cognitive diagnosis (CD) can accurately measure students' knowledge states (KSs) and provide diagnostic information, which is conducive to student's self-learning and teacher's targeted teaching. Although MST and CD have a lot of advantages, many factors prevent MST from applying to CD. In this study, we first attempt to employ automated test assembly (ATA) to achieve the objectives of MST in the application of CD (called CD-MST) via heuristic algorithms. The mean correct response probability of all KSs for each item is used to describe the item difficulty of CD. The attribute reliability in CD is defined as the test quantitative target. A simulation study with the G-DINA model (generalized deterministic input noisy "and" gate model) was carried out to investigate the proposed CD-MST, and the results showed that the assembled panels of CD-MST satisfied the statistical and the non-statistical constraints.

Multidimensional Test Assembly Using Mixed-Integer Linear Programming: An Application of Kullback-Leibler Information.

Many educational testing programs require different test forms with minimal or no item overlap. At the same time, the test forms should be parallel in terms of their statistical and content-related properties. A well-established method to assemble parallel test forms is to apply combinatorial optimization using mixed-integer linear programming (MILP). Using this approach, in the unidimensional case, Fisher information (FI) is commonly used as the statistical target to obtain parallelism. In the multidimensional case, however, FI is a multidimensional matrix, which complicates its use as a statistical target. Previous research addressing this problem focused on item selection criteria for multidimensional computerized adaptive testing (MCAT). Yet these selection criteria are not directly transferable to the assembly of linear parallel test forms. To bridge this gap the authors derive different statistical targets, based on either FI or the Kullback-Leibler (KL) divergence, that can be applied in MILP models to assemble multidimensional parallel test forms. Using simulated item pools and an item pool based on empirical items, the proposed statistical targets are compared and evaluated. Promising results with respect to the KL-based statistical targets are presented and discussed.

Usefulness of Automated Latex Turbidimetric Rapid Plasma Reagin Test for Diagnosis and Evaluation of Treatment Response in Syphilis in Comparison with Manual Card Test: a Prospective Cohort Study.

The usefulness of an automated latex turbidimetric rapid plasma reagin (RPR) assay, compared to the conventional manual card test (serial 2-fold dilution method), for the diagnosis of syphilis and evaluation of treatment response remains unknown. We conducted (i) a cross-sectional study and (ii) a prospective cohort study to elucidate the correlation between automated and manual tests and whether a 4-fold decrement is a feasible criterion for successful treatment with the automated test, respectively, in HIV-infected patients, from October 2015 to November 2017. Study i included 518 patients. The results showed strong correlation between the two tests (r = 0.931; P < 0.001). With a manual test titer of ≥1:8 plus a positive Treponema pallidum particle agglutination (TPPA) test as the reference standard for diagnosis, the optimal cutoff value for the automated test was 6.0 RPR units (area under the curve [AUC], 0.998), with positive predictive value (PPV) of 92.5% and negative predictive value (NPV) of 99.4%. Study ii enrolled 66 men with syphilis. Their RPR values were followed up until after 12 months of treatment. At 12 months, 77.3% and 78.8% of the patients achieved a 4-fold decrement in RPR titer by the automated and manual test, respectively. The optimal decrement rate in RPR titer by the automated test for a 4-fold decrement by manual card test was 76.54% (AUC, 0.96) (PPV, 96.1%; NPV, 80.0%). The automated RPR test is a good alternative to the manual test for the diagnosis of syphilis and evaluation of treatment response and is more rapid and can handle more specimens than the manual test without interpersonal variation in interpretation.

Random Changes of Accommodation Stimuli: An Automated Extension of the Flippers Accommodative Facility Test.

PURPOSE: To study the accommodative dynamics for predictable and unpredictable stimuli using manual and automated accommodative facility tests Materials and Methods: Seventeen young healthy subjects were tested monocularly in two consecutive sessions, using five different conditions. Two conditions replicated the conventional monocular accommodative facility tests for far and near distances, performed with manually held flippers. The other three conditions were automated and conducted using an electro-optical system and open-field autorefractor. Two of the three automated conditions replicated the predictable manual accommodative facility tests. The last automated condition was a hybrid approach using a novel method whereby far and near-accommodative-facility tests were randomly integrated into a single test of four unpredictable accommodative demands. RESULTS: The within-subject standard deviations for far- and near-distance-accommodative reversals were (±1,±1) cycles per minute (cpm) for the manual flipper accommodative facility conditions and (±3, ±4) cpm for the automated conditions. The 95% limits of agreement between the manual and the automated conditions for far and near distances were poor: (-18, 12) and (-15, 3). During the hybrid unpredictable condition, the response time and accommodative response parameters were significantly (p < 0.05) larger for accommodation than disaccommodation responses for high accommodative demands only. The response times during the transitions 0.17/2.17 D and 0.50/4.50 D appeared to be indistinguishable between the hybrid unpredictable and the conventional predictable automated tests. CONCLUSIONS: The automated accommodative facility test does not agree with the manual flipper test results. Operator delays in flipping the lens may account for these differences. This novel test, using unpredictable stimuli, provides a more comprehensive examination of accommodative dynamics than conventional manual accommodative facility tests. Unexpectedly, the unpredictability of the stimulus did not to affect accommodation dynamics. Further studies are needed to evaluate the sensitivity of this novel hybrid technique on individuals with accommodative anomalies.

Should We Stop Developing Heuristics and Only Rely on Mixed Integer Programming Solvers in Automated Test Assembly? A Rejoinder to van der Linden and Li (2016).

This rejoinder responds to the commentary by van der Linden and Li entiled "Comment on Three-Element Item Selection Procedures for Multiple Forms Assembly: An Item Matching Approach" on the article "Three-Element Item Selection Procedures for Multiple Forms Assembly: An Item Matching Approach" by Chen. Van der Linden and Li made a strong statement calling for the cessation of test assembly heuristics development, and instead encouraged embracing mixed integer programming (MIP). This article points out the nondeterministic polynomial (NP)-hard nature of MIP problems and how solutions found using heuristics could be useful in an MIP context. Although van der Linden and Li provided several practical examples of test assembly supporting their view, the examples ignore the cases in which a slight change of constraints or item pool data might mean it would not be possible to obtain solutions as quickly as before. The article illustrates the use of heuristic solutions to improve both the performance of MIP solvers and the quality of solutions. Additional responses to the commentary by van der Linden and Li are included.

A review of the 5-Choice Serial Reaction Time (5-CSRT) task in different vertebrate models.

Within cognitive and behavioural research, the 5-Choice Serial Reaction Time task is widely recognized as a valuable test of attention in rats. However, technical and methodological developments required for extending its usefulness are still at an early stage. In view of advances in knowledge about cognition and other areas of biology, issues surrounding attention are increasingly important, and appear to require new methodological approaches. These changes may concern (i) the evolution of the protocol itself, (ii) adaptations in how tasks are implemented (e.g. use of new technologies such as touchscreens), and (iii) applying existing tasks to species presenting an emerging potential. From a primarily methodological perspective, this review focuses on work that has successively built upon the original 5-CSRT task. We address the strengths and weaknesses of new approaches as well as some of the new possibilities they offer.

Multicenter evaluation of the Luminex® ARIES® HSV 1&2 Assay for the detection of herpes simplex virus types 1 and 2 in cutaneous and mucocutaneous lesion specimens.

INTRODUCTION: The ARIES® HSV 1&2 Assay is a new FDA cleared real-time PCR test for detection and differentiation of HSV-1 and HSV-2 DNA from cutaneous and mucocutaneous lesions. The test is performed on the ARIES® System, an automated sample to answer real-time PCR instrument that provides a closed system and simple workflow for performing molecular testing. Areas covered: This article reports the clinical performance of the ARIES® HSV 1&2 Assay assessed on 1963 prospectively collected specimens. Assay sensitivities were 91.1-95% (cutaneous) and 97-98.5% (mucocutaneous), and specificities were 88.8-94.2% (cutaneous) and 93.2-95.4% (mucocutaneous), as compared to the ELVIS® HSV test system. Expert commentary: Detection of HSV DNA by PCR is rapid and more sensitive than traditional culture and immunoassay methods and is being widely adopted in many laboratory settings. Sample to answer molecular platforms like ARIES® will enable routine and non-molecular labs to perform sensitive and rapid molecular testing with ease.

Comment on Three-Element Item Selection Procedures for Multiple Forms Assembly: An Item Matching Approach.

A recent article in this journal addressed the choice between specialized heuristics and mixed-integer programming (MIP) solvers for automated test assembly. This reaction is to comment on the mischaracterization of the general nature of MIP solvers in this article, highlight the quite inefficient modeling of the test-assembly problems used in its empirical examples, and counter these examples by presenting the MIP solutions for a set of 35 real-world multiple-form assembly problems.

Is Joint Attention Detectable at a Distance? Three Automated, Internet-Based Tests.

CONTEXT: Joint attention is the shared focus of two or more individuals on the same object. Sensory cues, such as detecting the direction of another person's gaze, play a major role in establishing joint attention. It may also involve a kind of mental resonance that might be felt by the people involved. OBJECTIVE: The aim of this study was to find out whether people could feel when another person was looking at the same picture at the same time, even when the participants were many miles apart. METHOD: Participants registered online with their names and e-mail addresses, and worked in pairs. After they both logged on for the test they were simultaneously shown one of two photographs, with a 0.5 probability of seeing the same picture. After 20s they were asked if their partner was looking at the same picture or not. After both had registered their guess, the next trial began, with a different pair of pictures. The main outcome measure was the proportion of correct guesses, compared with the 50% mean chance expectation. This test was symmetrical in that all participants were both "senders" and "receivers." RESULTS: In the first experiment, with 11,160 trials, the hit rate was 52.8% (P < 1 × 10(-6)); in the second experiment with 2720 trials, 51.3% (P = .09). The third experiment involved music as well as pictures, and with 8860 trials, the hit rate was 51.9% (P = .0003). Some partners were more than 1000 miles apart, but there were no significant effect of distance. Participants who received immediate feedback about whether their guess was right or wrong did not score significantly better than those without feedback.

Evaluation of new automated syphilis test reagents 'IMMUNOTICLES AUTO3' series: performance, biochemical reactivity, and clinical significance.

BACKGROUND: Automated nontreponemal and treponemal test reagents based on the latex agglutination method (immunoticles auto3 RPR: ITA3RPR and immunoticles auto3 TP: ITA3TP) have been developed to improve the issues of conventional manual methods such as their subjectivity, a massive amount of assays, and so on. We evaluated these reagents in regards to their performance, reactivity to antibody isotype, and their clinical significance. METHODS: ITA3RPR and ITA3TP were measured using a clinical chemistry analyzer. Reactivity to antibody isotype was examined by gel filtration analysis. RESULTS: ITA3RPR and ITA3TP showed reactivity to both IgM- and IgG-class antibodies and detected early infections. ITA3RPR was verified to show a higher reactivity to IgM-class antibodies than the conventional methods. ITA3RPR correlated with VDRL in the high titer range, and measurement values decreased with treatment. ITA3RPR showed a negative result earlier after treatment than conventional methods. ITA3TP showed high specificity and did not give any false-negative reaction. Significant differences in the measurement values of ITA3RPR between the infective and previous group were verified. CONCLUSIONS: The double test of ITA3RPR and ITA3TP enables efficient and objective judgment for syphilis diagnosis and treatments, achieving clinical availability.