Influence of feedback characteristics on perceived learning value of feedback in clerkships: does culture matter?

BACKGROUND: Various feedback characteristics have been suggested to positively influence student learning. It is not clear how these feedback characteristics contribute to students' perceived learning value of feedback in cultures classified low on the cultural dimension of individualism and high on power distance. This study was conducted to validate the influence of five feedback characteristics on students' perceived learning value of feedback in an Indonesian clerkship context. METHODS: We asked clerks in Neurology (n = 169) and Internal Medicine (n = 132) to assess on a 5-point Likert scale the learning value of the feedback they received. We asked them to record whether the feedback provider (1) informed the student what went well, (2) mentioned which aspects of performance needed improvement, (3) compared the student's performance to a standard, (4) further explained or demonstrated the correct performance, and (5) prepared an action plan with the student to improve performance. Data were analyzed using multilevel regression. RESULTS: A total of 250 students participated in this study, 131 from Internal Medicine (response rate 99%) and 119 from Neurology (response rate 70%). Of these participants, 225 respondents (44% males, 56% females) completed the form and reported 889 feedback moments. Students perceived feedback as more valuable when the feedback provider mentioned their weaknesses (ß = 0.153, p < 0.01), compared their performance to a standard (ß = 0.159, p < 0.01), explained or demonstrated the correct performance (ß = 0.324, p < 0.001) and prepared an action plan with the student (ß =0.496, p < 0.001). Appraisal of good performance did not influence the perceived learning value of feedback. No gender differences were found for perceived learning value. CONCLUSIONS: In Indonesia, we could validate four out of the five characteristics for effective feedback. We argue that our findings relate to culture, in particular to the levels of individualism and power distance. The recognized characteristics of what constitutes effective feedback should be validated across cultures.

Exploring cultural differences in feedback processes and perceived instructiveness during clerkships: replicating a Dutch study in Indonesia.

CONTEXT: Cultural differences between countries may entail differences in feedback processes. AIMS: By replicating a Dutch study in Indonesia, we analysed whether differences in processes influenced the perceived instructiveness of feedback. METHODS: Over a two-week period, Indonesian students (n = 215) recorded feedback moments during clerkships, noting who provided the feedback, whether the feedback was based on observations, who initiated the feedback, and its perceived instructiveness. Data were compared with the earlier Dutch study and analysed with χ(2) tests, t-tests and multilevel techniques. Cultural differences were explored using Hofstede's Model, with Indonesia and the Netherlands differing on "power distance" and "individualism." RESULTS: Perceived instructiveness of feedback did not differ significantly between both countries. However, significant differences were found in feedback provider, observation and initiative. Indonesian students perceived feedback as more instructive if provided by specialists and initiated jointly by the supervisor and student (ßresidents = -0.201, p < 0.001 and ßjoint = 0.193, p = 0.001). Dutch students appreciated feedback more when it was based on observation. CONCLUSIONS: We obtained empirical evidence that one model of feedback does not necessarily translate to another culture. Further research is necessary to unravel other possible influences of culture in implementing feedback procedures in different countries.

The effect of implementing undergraduate competency-based medical education on students' knowledge acquisition, clinical performance and perceived preparedness for practice: a comparative study.

BACKGROUND: Little is known about the gains and losses associated with the implementation of undergraduate competency-based medical education. Therefore, we compared knowledge acquisition, clinical performance and perceived preparedness for practice of students from a competency-based active learning (CBAL) curriculum and a prior active learning (AL) curriculum. METHODS: We included two cohorts of both the AL curriculum (n=453) and the CBAL curriculum (n=372). Knowledge acquisition was determined by benchmarking each cohort on 24 interuniversity progress tests against parallel cohorts of two other medical schools. Differences in knowledge acquisition were determined comparing the number of times CBAL and AL cohorts scored significantly higher or lower on progress tests. Clinical performance was operationalized as students' mean clerkship grade. Perceived preparedness for practice was assessed using a survey. RESULTS: The CBAL cohorts demonstrated relatively lower knowledge acquisition than the AL cohorts during the first study years, but not at the end of their studies. We found no significant differences in clinical performance. Concerning perceived preparedness for practice we found no significant differences except that students from the CBAL curriculum felt better prepared for 'putting a patient problem in a broad context of political, sociological, cultural and economic factors' than students from the AL curriculum. CONCLUSIONS: Our data do not support the assumption that competency-based education results in graduates who are better prepared for medical practice. More research is needed before we can draw generalizable conclusions on the potential of undergraduate competency-based medical education.

Key elements in assessing the educational environment: where is the theory?

The educational environment has been increasingly acknowledged as vital for high-quality medical education. As a result, several instruments have been developed to measure medical educational environment quality. However, there appears to be no consensus about which concepts should be measured. The absence of a theoretical framework may explain this lack of consensus. Therefore, we aimed to (1) find a comprehensive theoretical framework defining the essential concepts, and (2) test its applicability. An initial review of the medical educational environment literature indicated that such frameworks are lacking. Therefore, we chose an alternative approach to lead us to relevant frameworks from outside the medical educational field; that is, we applied a snowballing technique to find educational environment instruments used to build the contents of the medical ones and investigated their theoretical underpinnings (Study 1). We found two frameworks, one of which was described as incomplete and one of which defines three domains as the key elements of human environments (personal development/goal direction, relationships, and system maintenance and system change) and has been validated in different contexts. To test its applicability, we investigated whether the items of nine medical educational environment instruments could be mapped unto the framework (Study 2). Of 374 items, 94% could: 256 (68%) pertained to a single domain, 94 (25%) to more than one domain. In our context, these domains were found to concern goal orientation, relationships and organization/regulation. We conclude that this framework is applicable and comprehensive, and recommend using it as theoretical underpinning for medical educational environment measures.

Alternating skills training and clerkships to ease the transition from preclinical to clinical training.

BACKGROUND: The transition from preclinical to clinical training is perceived as stressful with a high workload being the main difficulty. To ease this transition, we implemented a dual learning year, where just-in-time skills training and clerkships alternated. AIMS: To examine the effect of the dual learning year on students' stress and perceptions of workload and skills level, and to compare these data with a baseline measurement from a curriculum in which skills training was provided in advance of clerkships. METHOD: During the first Master year, students completed four rotations in which 5 weeks of skills training and 5-week clerkships alternated. In the second clerkship week of each rotation, students (n = 476) completed questionnaires measuring their experienced workload, perceived skills level and stress. Analysis of variance was used for trend analysis and to determine differences with the baseline measurement (n = 83). RESULTS: During the dual learning year, 'experienced workload' decreased (F(1,471) = 9.408, p < 0.01), 'perceived skills level' increased (F(1,471) = 94.202, p < 0.001) and stress decreased (F(1,471) = 3.309, p < 0.10). 'Experienced workload' was lower (F(5,553) = 7.599, p < 0.001) and 'perceived skills level' was comparable to the baseline measurement. CONCLUSIONS: Compared to the baseline measurement and the results of earlier studies on transition difficulties, alternating just-in-time skills training and clerkships seem to ease the transition from preclinical to clinical training.

The digital pen as a novel device to facilitate the feedback process.

BACKGROUND: To improve clinical performance assessment, checklist data should be immediately available to students to offer them detailed feedback and be stored in a database for quality assurance purposes. AIM: To introduce the digital pen as clinical performance assessment tool, report examiner satisfaction and explore the utility of generated checklist data for quality assurance purposes. METHODS: The digital pen technology transmits examiners' handwritten assessments to a database and exports PDF-files to students' mailboxes. Descriptive statistical analysis of examiner satisfaction and the generated checklist data was performed. RESULTS: The examiners were satisfied with the digital pen. Valuable data were obtained to improve objective structured clinical examination stations and rating criteria, identify training needs for future students and provide examiners with feedback on their rating skills. CONCLUSION: The digital pen technology is a practical device for sending completed checklists to students and providing valuable data for quality assurance purposes.

The reliability of in-training assessment when performance improvement is taken into account.

During in-training assessment students are frequently assessed over a longer period of time and therefore it can be expected that their performance will improve. We studied whether there really is a measurable performance improvement when students are assessed over an extended period of time and how this improvement affects the reliability of the overall judgement. In-training assessment results were obtained from 104 students on rotation at our university hospital or at one of the six affiliated hospitals. Generalisability theory was used in combination with multilevel analysis to obtain reliability coefficients and to estimate the number of assessments needed for reliable overall judgement, both including and excluding performance improvement. Students' clinical performance ratings improved significantly from a mean of 7.6 at the start to a mean of 7.8 at the end of their clerkship. When taking performance improvement into account, reliability coefficients were higher. The number of assessments needed to achieve a reliability of 0.80 or higher decreased from 17 to 11. Therefore, when studying reliability of in-training assessment, performance improvement should be considered.

Time spent on clerkship activities by students in relation to their perceptions of learning environment quality.

CONTEXT: Students' perceptions of their learning environment are of great importance to their learning process. In this study we assessed the time allocated by students to clerkship activities and the relationship between students' allocations of time and their perceptions of the quality of their clinical learning environment. METHODS: Participants were 133 undergraduate students from eight hospitals taking part in four clerkship rotations. All students recorded the time they spent on eight clerkship activities over 2 weeks and completed the Postgraduate Hospital Educational Environment Measure (PHEEM). Partial correlation analyses were undertaken to examine the relationship between the amount of time students spent on each activity and their PHEEM scores. RESULTS: Students spent nearly 8 hours a day on clerkship activities. Most time was spent on observing doctors (40%), followed by participating in consultations without direct supervision (12%). The time students spent on observing doctors (r = 0.206, P < 0.05) and in consultations without direct supervision (r = 0.211, P < 0.05) was significantly related to the students' PHEEM scores. There was a significant relationship at the P < 0.10 level between the time spent on directly supervised activities and students' PHEEM scores (r = 0.165, P < 0.10). CONCLUSIONS: The results suggest that the time spent on activities involving direct patient contact is positively related to students' perceptions of the quality of their learning environment. None of the activities were significantly negatively related to the students' perceptions of their clinical learning environment. Future research should examine the optimal time allocations required to enhance the perceived quality of the clinical learning environment.

Necessary steps in factor analysis: enhancing validation studies of educational instruments. The PHEEM applied to clerks as an example.

BACKGROUND: The validation of educational instruments, in particular the employment of factor analysis, can be improved in many instances. AIMS: To demonstrate the superiority of a sophisticated method of factor analysis, implying an integration of recommendations described in the factor analysis literature, over often employed limited applications of factor analysis. We demonstrate the essential steps, focusing on the Postgraduate Hospital Educational Environment Measure (PHEEM). METHOD: The PHEEM was completed by 279 clerks. We performed Principal Component Analysis (PCA) with varimax rotation. A combination of three psychometric criteria was applied: scree plot, eigenvalues >1.5 and a minimum percentage of additionally explained variance of approximately 5%. Furthermore, four interpretability criteria were used. Confirmatory factor analysis was performed to verify the original scale structure. RESULTS: Our method yielded three interpretable and practically useful dimensions: learning content and coaching, beneficial affective climate and external regulation. Additionally, combining several criteria reduced the risk of overfactoring and underfactoring. Furthermore, the resulting dimensions corresponded with three learning functions essential to high-quality learning, thus strengthening our findings. Confirmatory factor analysis disproved the original scale structure. CONCLUSIONS: Our sophisticated approach yielded several advantages over methods applied in previous validation studies. Therefore, we recommend this method in validation studies to achieve best practice.

Instructiveness of feedback during clerkships: influence of supervisor, observation and student initiative.

BACKGROUND: Several authors assume that the supervisor's role, observation of behaviour and students' active participation are important factors in the instructiveness of feedback. AIM: This study aims to provide empirical evidence for these expectations. METHODS: For two weeks, 142 clerks from eight hospitals recorded for each individual feedback event: who provided the feedback, whether the feedback was based on observation of behaviour, who initiated the feedback moment and the perceived instructiveness of the feedback. Data were analysed with multilevel techniques. RESULTS: The perceived instructiveness of feedback provided by specialists and residents did not differ significantly. However, both were perceived to be more instructive than feedback from nursing and paramedical staff (beta(specialists) = 0.862, p < 0.01; beta(residents) = 0.853, p < 0.01). Feedback on behaviour that had been directly observed was reported to be more instructive than feedback on behaviour that had not been observed (beta(observed) = 0.314, p < 0.001). Feedback which stemmed from student initiative or a joint initiative was experienced to be more instructive than feedback which ensued from the supervisor's initiative (beta(student) = 0.441, p < 0.01; beta(joint) = 0.392, p < 0.01). CONCLUSIONS: The expectations concerning the influence of observation and student initiative on the instructiveness of feedback were confirmed in this empirical study. Expected differences in instructiveness between feedback from specialists and residents were not confirmed.

Transition to clinical training: influence of pre-clinical knowledge and skills, and consequences for clinical performance.

CONTEXT: Many students experience a tough transition from pre-clinical to clinical training and previous studies suggest that this may constrict students' progress. However, clear empirical evidence of this is lacking. The aim of this study was to determine: whether the perceived difficulty of transition influences student performance during the first 2 weeks of clerkships; whether it influences students' overall performance in their first clerkship, and the degree to which the difficulty of transition is influenced by students' pre-clinical knowledge and skills levels. METHODS: Clerks (n = 83) from a university hospital and eight affiliated hospitals completed a questionnaire measuring the perceived difficulty of the transition period. Data collected included student scores on pre-clinical knowledge and skills, their performance during the second week of the first clerkship, and their overall performance in the first clerkship. Univariate and multivariate multiple regression analyses were used to analyse the data. RESULTS: The perceived difficulty of transition was neither predictive of student performance during the transition period (adjusted R(2) = 11.8%, P = NS), nor of their overall clerkship performance (adjusted R(2) = 8.6%, P = NS). Students' pre-clinical knowledge and skills played a minor role in the perceived difficulty of the transition period. CONCLUSIONS: The negative effect of the transition period on student progress suggested in the literature was not found in this study. A possible explanation for the limited influence of students' knowledge and skills on performance during the transition period is that the workload in this period causes a cognitive overload, interfering with students' abilities to apply their pre-clinical knowledge and skills.