Institutional strategies related to test-taking behavior in low stakes assessment.

Low stakes assessment without grading the performance of students in educational systems has received increasing attention in recent years. It is used in formative assessments to guide the learning process as well as in large-scales assessments to monitor educational programs. Yet, such assessments suffer from high variation in students' test-taking effort. We aimed to identify institutional strategies related to serious test-taking behavior in low stakes assessment to provide medical schools with practical recommendations on how test-taking effort might be increased. First, we identified strategies that were already used by medical schools to increase the serious test-taking behavior on the low stakes Berlin Progress Test (BPT). Strategies which could be assigned to self-determination theory of Ryan and Deci were chosen for analysis. We conducted the study at nine medical schools in Germany and Austria with a total of 108,140 observations in an established low stakes assessment. A generalized linear-mixed effects model was used to assess the association between institutional strategies and the odds that students will take the BPT seriously. Overall, two institutional strategies were found to be positively related to more serious test-taking behavior: discussing low test performance with the mentor and consequences for not participating. Giving choice was negatively related to more serious test-taking behavior. At medical schools that presented the BPT as evaluation, this effect was larger in comparison to medical schools that presented the BPT as assessment.

Progress testing anytime and anywhere - Does a mobile-learning approach enhance the utility of a large-scale formative assessment tool?

BACKGROUND: The widespread use of mobile devices among students favors the use of mobile learning scenarios at universities. In this study, we explore whether a time- and location-independent variant of a formative progress test has an impact on the students' acceptance, its validity and reliability and if there is a difference in response processes between the two exam conditions. METHODS: Students were randomly assigned to two groups of which one took the test free of local or temporal fixations, while the other group took the test at the local testing center under usual examination conditions. Beside the generated test data, such as test score, time-on-test, and semester status, students also evaluated the settings. RESULTS: While there was no significant effect on the test score between the two groups, students in the mobile group spent more time on the test and were more likely to use the help of books or online resources. The results of the evaluation show that the acceptability among students is increased by a mobile version of the formative progress test. CONCLUSIONS: The results suggest that the acceptance and motivation to participate in formative tests is enhanced by lifting local and temporal restrictions. The mobile version nonetheless does not have an impact on the students' performance.

A validity argument for progress testing: Examining the relation between growth trajectories obtained by progress tests and national licensing examinations using a latent growth curve approach.

Background: Progress testing is a longitudinal assessment that aims at tracking students' development of knowledge. This approach is used in many medical schools internationally. Although progress tests are longitudinal in nature, and their focus and use of developmental aspects is a key advantage, individual students' learning trajectories themselves play, to date, only a minor role in the use of the information obtained through progress testing. Methods: We investigate in how far between-person differences in initial levels of performance and within-person rate of growth can be regarded as distinct components of students' development and analyze the extent to which these two components are related to performances on national licensing examinations using a latent growth curve model. Results: Both, higher initial levels of performances and steepness of growth are positively related to long-term outcomes as measured by performance on national licensing examinations. We interpret these findings as evidence for progress tests' suitability to monitor students' growth of knowledge across the course of medical training. Conclusions: This study indicates that individual development as obtained by formative progress tests is related to performance in high-stakes assessments. Future studies may put more focus on the use of between-persons differences in growth of knowledge.

Identifying low test-taking effort during low-stakes tests with the new Test-taking Effort Short Scale (TESS) - development and psychometrics.

BACKGROUND: Low-stakes tests are becoming increasingly important in international assessments of educational progress, and the validity of these results is essential especially as these results are often used for benchmarking. Test scores in these tests not only mirror students' ability but also depend on their test-taking effort. One way to obtain more valid scores from participating samples is to identify test-takers with low test-taking effort and to exclude them from further analyses. Self-assessment is a convenient and quick way of measuring test-taking effort. We present the newly developed Test-taking Effort Short Scale (TESS), which comprises three items measuring attainment value/intrinsic value, utility value, and perceived benefits, respectively. METHODS: In a multicenter validation study with N = 1837 medical students sitting a low-stakes progress test we analyzed item and test statistics including construct and external validity. RESULTS: TESS showed very good psychometric properties. We propose an approach using stanine norms to determine a cutoff value for identifying participants with low test-taking effort. CONCLUSION: With just three items, TESS is shorter than most established self-assessment scales; it is thus suited for administration after low-stakes progress testing. However, further studies are necessary to establish its suitability for routine usage in assessment outside progress testing.

Computer versus paper--does it make any difference in test performance?

UNLABELLED: CONSTRUCT: In this study, we examine the differences in test performance between the paper-based and the computer-based version of the Berlin formative Progress Test. In this context it is the first study that allows controlling for students' prior performance. BACKGROUND: Computer-based tests make possible a more efficient examination procedure for test administration and review. Although university staff will benefit largely from computer-based tests, the question arises if computer-based tests influence students' test performance. APPROACH: A total of 266 German students from the 9th and 10th semester of medicine (comparable with the 4th-year North American medical school schedule) participated in the study (paper = 132, computer = 134). The allocation of the test format was conducted as a randomized matched-pair design in which students were first sorted according to their prior test results. The organizational procedure, the examination conditions, the room, and seating arrangements, as well as the order of questions and answers, were identical in both groups. RESULTS: The sociodemographic variables and pretest scores of both groups were comparable. The test results from the paper and computer versions did not differ. The groups remained within the allotted time, but students using the computer version (particularly the high performers) needed significantly less time to complete the test. In addition, we found significant differences in guessing behavior. Low performers using the computer version guess significantly more than low-performing students in the paper-pencil version. CONCLUSIONS: Participants in computer-based tests are not at a disadvantage in terms of their test results. The computer-based test required less processing time. The reason for the longer processing time when using the paper-pencil version might be due to the time needed to write the answer down, controlling for transferring the answer correctly. It is still not known why students using the computer version (particularly low-performing students) guess at a higher rate. Further studies are necessary to understand this finding.

A simple approach of applying blended learning to problem-based learning is feasible, accepted and does not affect evaluation and exam results-a just pre-pandemic randomised controlled mixed-method study.

We tested for feasibility, acceptance, and "non-inferiority" of small-group teaching applying blended learning (i.e., the integration of face-to-face and online instruction) to problem-based learning (bPbL) compared to conventional PbL (cPbL). In a just pre-pandemic, randomised controlled trial, 317 students attended either bPbL or cPbL groups. The first meeting of the bPbL groups took place online via written internet chat, while cPbL groups met on site. All groups met on site the second time. All students had the opportunity to attend lectures either on site or as videos on demand. We analysed student evaluation data, results in a final summative exam, attendance of lectures on site and use of lecture videos. Furthermore, we performed a qualitative analysis of student statements made in semi-structured group interviews about pros and cons of the bPbL approach. There was no difference between students of either bPbL or cPbL groups with respect to exam results (score: 14.3 ± 2.8 vs. 13.8 ± 2.7) or course evaluation. However, relatively more bPbL than cPbL students reported having used lecture videos, while the proportion of those attending lectures on-site was higher among cPbL students. Interviews revealed that some of the bPbL students' experiences were unexpected and feared disadvantages seemed to be less severe than expected. Participation in a blended PbL format did not worsen course evaluations or exam results, but seemed to influence lecture attendance. The combination of face-to-face and digital elements could be suitable as a hybrid approach to digital instruction in the post-pandemic era.

The formation of physician altruism.

We study how patient-regarding altruism is formed by medical education. We elicit and structurally estimate altruistic preferences using experimental data from a large sample of medical students (N = 733) in Germany at different progress stages in their studies. The estimates reveal substantial heterogeneity in altruistic preferences of medical students. Patient-regarding altruism is highest for freshmen, significantly declines for students in the course of medical studies, and tends to increase again for last year students, who assist in clinical practice. Also, patient-regarding altruism is higher for females and positively associated to general altruism. Altruistic medical students have gained prior practical experience in healthcare, have lower income expectations, and are more likely to choose surgery and pediatrics as their preferred specialty.

Can computer-based assessment enhance the acceptance of formative multiple choice exams? A utility analysis.

BACKGROUND: Students' motivation to participate is one of the main challenges in formative assessment. The utility framework identifies potential points of intervention for improving the acceptance of formative assessment [Van Der Vleuten C. 1996. The assessment of professional competence: Developments, research and practical implications. Adv Health Sci Educ 1(1):41-67]. At the Medical Faculty of the University of Cologne, the paper-based version of the Berlin Progress Test has been transformed into computer-based version providing immediate feedback. AIM: To investigate whether the introduction of computer-based assessment (CBA) enhances the acceptance of formative assessment relative to paper-based assessment (PBA). METHODS: In a retrospective cohort study (PBA: N = 2597, CBA: N = 2712), students' overall acceptance of the two forms of assessment was surveyed, their comments were analyzed, and we analyzed their test behavior and categorized students into "serious" or "non-serious" test takers. RESULTS: In the preclinical phase of medical education, no differences were found in overall acceptance of the two forms of assessment (p > 0.05). In the clinical phase, differences in favor of CBA were found in overall acceptance (p < 0.05), the proportion of positive comments (p < 0.001), and the proportion of serious participants (p < 0.001). CONCLUSIONS: Introduction of immediate feedback via CBA can enhance the acceptance and therefore the utility of formative assessment.

Comparison of the evaluation of formative assessment at two medical faculties with different conditions of undergraduate training, assessment and feedback.

Introduction: Both formative and summative assessments have their place in medical curricula: formative assessment to accompany the learning process and summative assessment to ensure that minimum standards are achieved. Depending on the conditions of undergraduate training, assessment and feedback, students place more or less importance on formative assessment, and thus the fulfilment of its function may be questionable. This study describes how the low-stakes formative Berlin Progress Test (BPT) is embedded at two medical faculties with partially different framework conditions and what effects these have on the students' testing efforts and the evaluation of the test, especially the perception of its benefits and (intangible) costs, such as non-participation in contemporaneous activities and emotional impairments. Methods: In this study, the proportion of non-serious BPT participants at two medical faculties (total sample: NF1=1,410, NF2=1,176) in winter term 2015/16 was determined both by the number of unanswered questions on the test itself and in a survey using a standardized instrument (NF1=415, NF2=234). Furthermore, open questions were asked in this survey about perceived benefits and perceived costs, which were analyzed with qualitative and quantitative methods. Results: The BPT is generally better accepted at Faculty 2. This can be seen in the higher proportion of serious test takers, the lower perceived costs and the higher reported benefit, as well as the higher proportion of constructive comments. Faculty 2 students better understood the principle of formative testing and used the results of the BPT as feedback on their own knowledge progress, motivation to learn and reduction of exam fear. Discussion: When medical faculties integrate formative assessments into the curriculum, they have to provide a framework in which these assessments are perceived as an important part of the curriculum. Otherwise, it is questionable whether they can fulfil their function of accompanying the learning process.

Fifteen years of the cologne medical model study course: has the expectation of increasing student interest in general practice specialization been fulfilled?

Background: The 2002 Medical Licensure Act gave German universities certain freedoms for reforming their medical degree courses. The Medical Faculty of the University of Cologne took advantage of this opportunity and introduced a model study course in the winter semester 2003/04 through §41 of the Medical Licensure Act. One of the main reasons for this was that back then there was an increasing shortage of doctors in clinical curative medicine and GP primary care. This study investigates whether the introduction of the Cologne Model Study Course (MSG) can show stronger interest in curative medical work (especially General Practice) compared to students of the standard degree course (RSG). Methodology: The proof of added value was examined through graduate surveys conducted at the University of Cologne and through the proportion of students who completed the PY elective rotation "General Practice". The students of the standard degree course (start of studies prior to winter semester 2003/2004) were compared with students of the model study course (start of studies from winter semester 2003/04 onwards). Measurements were carried out using descriptive frequency tables and correlation analyzes according to Spearman. Results: The students' interest in curative medicine was already high (91%) even before the model study course was introduced and increased only slightly (to 91.9%). There is also only a slight increase in specialization in General Practice (RSG=5.9% vs. MSG=9.2%). However, selection of rotations in General Practice was significantly increased (RSG=1.9% vs. MSG=3.4%, r=0.046 **, p<0.005). Conclusion: The Cologne Model Study Course in Human Medicine has increased awareness of the subject of General Practice among students through a large number of curricular changes. The fact that only marginal effects can be demonstrated shows once more the strong dependence of choosing General Medicine as a career path on other factors (such as gender or the presence of positive role models) and emphasizes the necessity of promoting General Practice student education not only through increased curricular mapping but by additional innovative concepts to maximize the status of General Practice from the perspective of students.

Research into finding a stable prognosis parameter for the detection of students in need of guidance - Realization of equal opportunities through a diversity-oriented study guidance.

Objective: The internationalization of teaching and studying as well as increasing numbers of students with increasingly heterogeneous educational biographies and lifestyles require universities to develop awareness of this diversity and the need for adequate diversity management. For some diversity criteria at least it has been proven that they can influence the individual study success of students. The Dean's Office of the Medical Faculty of the University of Cologne has empirically determined a stable prognosis parameter for study progression on the basis of selected criteria in order to enable early detection of students in need of guidance. This will then be used for targeted, diversity-oriented study guidance. On the one hand a correspondingly adapted guidance offer should take into account individual study progressions. On the other hand, measures to improve the equal opportunities of students with regard to their academic success can be discussed. Methodology: With the help of study progression analyses, study progress of cohorts can be recorded longitudinally. The study progression analysis implemented in the control of faculty teaching serves as a central forecasting and steering tool for the forthcoming concept of diversity-oriented study guidance. The significance measurement of the various features is determined using binary logistic regression analyses. Results: As part of the study progression analyses, the study success rate after the first semester has the strongest influence on the concordance with the minimum duration of study in the pre-clinical phase, followed by the characteristics age at commencement of studies and place of university entrance qualification. The school leaving grade only just misses the required significance level of p <0.05. As a predictor gender provides no explanatory contribution in the considered model. Conclusion: In order to do justice to the heterogeneity among the students, university administrators and lecturers should understand the recognition of diversity as a cross-cutting task and keep an eye on diversity-related aspects and discrimination-critical topics for different target groups as well as individual guidance services in the context of individual study guidance. Within the scope of this study, we were able to empirically prove the stable prognosis parameter study success rate after the first semester allows reliable detection of students in need of guidance. The explanatory contribution is larger than any of the individual criteria examined in this study. The specific causes that led to a delay in studying will be analyzed in the context of downstream and diversity-oriented study guidance. A follow-up study will deal with the question of whether the success of students requiring study guidance can be significantly improved by subsequent study guidance.

Studienstart International of the University of Cologne: The closely supervised semester of study entry for students from third countries using the example of the model degree program for human medicine.

Objective: Due to the increasing degree of internationalization and migration movements, German universities must increasingly pay attention to the cultural diversity of their students. Numerous studies have shown that there is still considerable room for improvement at German universities with regard to the integration of foreign students. Therefore, the University of Cologne aims at improving study conditions for students from third countries (especially non-EU) by introducing a compulsory study entry semester Studienstart International. In this study, we ask if effects on the dropout rates of international students before the introduction of Studienstart International can be observed after the introduction of the study entry semester. In addition, results of a survey of foreign students about familiarizing themselves with the German university educational system and on social contacts with fellow students are presented. Methods: After a one-year voluntary pilot phase in summer semester 2009 and winter semester 2009/10, Studienstart International was made compulsory for Non-EU students in preparation for the study of human medicine. 113 students of the human medicine degree program have taken part in this program since the beginning of winter semester 2016/17. The comparison of dropout rates before and after the introduction of Studienstart International as well as the results from a survey of foreign students are presented descriptively. Statistical significance of the difference in dropout rates between the two groups of students is tested using a Chi2-test. Results: The dropout rates of foreign students fell slightly from 19% to 14% following the introduction of the mandatory study entry semester, but this difference is not statistically significant (Chi2=0.785, df=1, p=0.376). According to the survey, almost three-quarters of foreign students have contact with German fellow students at the university, 55% of whom report daily contact. 68% of respondents have contact with German fellow students outside the university. 97% feel they are well prepared for the degree course and 91% say they know where to go with questions and problems during their studies. Conclusion: With the introduction of the compulsory study entry semester Studienstart International, the University of Cologne aims at improving integration of foreign students. Although dropout rates could not be significantly reduced, survey results suggest that Studienstart International can certainly support international students. Reliable statements on the success of the study cannot yet be made due to the small number of cases. A follow-up study will deal with this question.

A study on effects of and stance over tuition fees.

AIM: Regarding tuition fees (that in Germany already have been abrogated) putative drawbacks like prolonged study duration have been suspected while benefits are not clearly proven. We investigated whether tuition fees (500 Euro per semester) affected the course of studies of Cologne medical students and asked for students' stance over tuition fees. METHODS: Of 1,324 students we analyzed the rate of those passing their first medical exam ("Physikum") within minimum time and students' discontinuation rate, respectively. Regression analysis tested for putative influences of tuition fees and demographic factors. In an additional online survey 400 students answered questions regarding the load by and their stance over tuition fees. RESULTS: We find that fees did not affect rate of Cologne students passing their first medical exam within minimum time or students' discontinuation rate. According to the online survey, at times of tuition fees significantly more students did not attend courses as scheduled. Time spent on earning money was significantly increased. 51% of students who had to pay tuition fees and 71% of those who never had to stated tuition fees to be not justified. More than two thirds of students did not recognize any lasting benefit from tuition fees. CONCLUSION: Tuition fees did not affect discontinuation rate or study duration of Cologne medical students. However, they obviously influenced the study course due to an increased need to pursue a sideline. Cologne medical students rather refused tuition fees and did not recognize their advantages in terms of enhanced quality of studies.

Peer teaching in paediatrics - medical students as learners and teachers on a paediatric course.

BACKGROUND: Peer assisted learning is known as an effective educational strategy in medical teaching. We established a peer assisted teaching program by student tutors with a focus on clinical competencies for students during their practical training on paediatric wards. It was the purpose of this study to investigate the effects of a clinical skills training by tutors, residents and consultants on students evaluations of the teaching quality and the effects of a peer teaching program on self assessed clinical competencies by the students. METHODS: Medical student peers in their 6(th) year were trained by an intensive instruction program for teaching clinical skills by paediatric consultants, doctors and psychologists. 109 students in their 5(th) year (study group) participated in a peer assisted teaching program for training clinical skills in paediatrics. The skills training by student peer teachers were supervised by paediatric doctors. 45 students (control group) participated in a conventional paediatric skills training by paediatric doctors and consultants. Students from both groups, which were consecutively investigated, completed a questionnaire with an evaluation of the satisfaction with their practical training and a self assessment of their practical competencies. RESULTS: The paediatric skills training with student peer teachers received significantly better ratings than the conventional skills training by paediatric doctors concerning both the quality of the practical training and the support by the teaching medical staff. Self assessed learning success in practical skills was higher rated in the peer teaching program than in the conventional training. CONCLUSIONS: The peer assisted teaching program of paediatric skills training was rated higher by the students regarding their satisfaction with the teaching quality and their self assessment of the acquired skills. Clinical skills training by student peer teachers have to be supervised by paediatric doctors. Paediatric doctors seem to be more motivated for their own teaching tasks if they are assisted by student peer teachers. More research is needed to investigate the influence of peer teaching on the motivation of paediatric doctors to teach medical students und the academic performance of the student peers.