The don't know option in progress testing.

Formula scoring (FS) is the use of a don't know option (DKO) with subtraction of points for wrong answers. Its effect on construct validity and reliability of progress test scores, is subject of discussion. Choosing a DKO may not only be affected by knowledge level, but also by risk taking tendency, and may thus introduce construct-irrelevant variance into the knowledge measurement. On the other hand, FS may result in more reliable test scores. To evaluate the impact of FS on construct validity and reliability of progress test scores, a progress test for radiology residents was divided into two tests of 100 parallel items (A and B). Each test had a FS and a number-right (NR) version, A-FS, B-FS, A-NR, and B-NR. Participants (337) were randomly divided into two groups. One group took test A-FS followed by B-NR, and the second group test B-FS followed by A-NR. Evidence for impaired construct validity was sought in a hierarchical regression analysis by investigating how much of the participants' FS-score variance was explained by the DKO-score, compared to the contribution of the knowledge level (NR-score), while controlling for Group, Gender, and Training length. Cronbach's alpha was used to estimate NR and FS-score reliability per year group. NR score was found to explain 27 % of the variance of FS [F(1,332) = 219.2, p < 0.0005], DKO-score, and the interaction of DKO and Gender were found to explain 8 % [F(2,330) = 41.5, p < 0.0005], and the interaction of DKO and NR 1.6 % [F(1,329) = 16.6, p < 0.0005], supporting our hypothesis that FS introduces construct-irrelevant variance into the knowledge measurement. However, NR-scores showed considerably lower reliabilities than FS-scores (mean year-test group Cronbach's alphas were 0.62 and 0.74, respectively). Decisions about FS with progress tests should be a careful trade-off between systematic and random measurement error.

Workplace-based assessment: raters' performance theories and constructs.

Weaknesses in the nature of rater judgments are generally considered to compromise the utility of workplace-based assessment (WBA). In order to gain insight into the underpinnings of rater behaviours, we investigated how raters form impressions of and make judgments on trainee performance. Using theoretical frameworks of social cognition and person perception, we explored raters' implicit performance theories, use of task-specific performance schemas and the formation of person schemas during WBA. We used think-aloud procedures and verbal protocol analysis to investigate schema-based processing by experienced (N = 18) and inexperienced (N = 16) raters (supervisor-raters in general practice residency training). Qualitative data analysis was used to explore schema content and usage. We quantitatively assessed rater idiosyncrasy in the use of performance schemas and we investigated effects of rater expertise on the use of (task-specific) performance schemas. Raters used different schemas in judging trainee performance. We developed a normative performance theory comprising seventeen inter-related performance dimensions. Levels of rater idiosyncrasy were substantial and unrelated to rater expertise. Experienced raters made significantly more use of task-specific performance schemas compared to inexperienced raters, suggesting more differentiated performance schemas in experienced raters. Most raters started to develop person schemas the moment they began to observe trainee performance. The findings further our understanding of processes underpinning judgment and decision making in WBA. Raters make and justify judgments based on personal theories and performance constructs. Raters' information processing seems to be affected by differences in rater expertise. The results of this study can help to improve rater training, the design of assessment instruments and decision making in WBA.

Influence of PBL with open-book tests on knowledge retention measured with progress tests.

The influence of problem-based learning (PBL) and open-book tests on long-term knowledge retention is unclear and subject of discussion. Hypotheses were that PBL as well as open-book tests positively affect long-term knowledge retention. Four progress test results of fifth and sixth-year medical students (n = 1,648) of three medical schools were analyzed. Two schools had PBL driven curricula, and the third one had a traditional curriculum (TC). One of the PBL schools (PBLob) used a combination of open-book (assessing backup knowledge) and closed-book tests (assessing core knowledge); the other two schools (TC and PBLcb) only used closed-book tests. The items of the progress tests were divided into core and backup knowledge. T tests (with Bonferroni correction) were used to analyze differences between curricula. PBL students performed significantly better than TC students on core knowledge (average effect size (av ES) = 0.37-0.74) and PBL students tested with open-book tests scored somewhat higher than PBL students tested without such tests (av ES = 0.23-0.30). Concerning backup knowledge, no differences were found between the scores of the three curricula. Students of the two PBL curricula showed a substantially better long-term knowledge retention than TC students. PBLob students performed somewhat better on core knowledge than PBLcb students. These outcomes suggest that a problem-based instructional approach in particular can stimulate long-term knowledge retention. Distinguishing knowledge into core and backup knowledge and using open-book tests alongside closed-book tests could enhance long-term core knowledge retention.

How Dutch medical specialists perceive the competencies and training needs of medical residents in healthcare management.

BACKGROUND: The Dutch postgraduate medical training has been revised to focus on seven competencies. The role as manager is one of these competencies. Recent studies show that this competency receives little attention during the residency training. In an earlier study, we discovered that residents perceived their competency as managers to be moderate. AIM: In this study, we investigated how medical specialists perceived the managerial competencies of medical residents and their need for management education. METHODS: In September 2010, a 46-item questionnaire was designed which examined medical specialists' perceptions of the competency and needs of residents in the field of medical management. Two hundred ninety-eight specialists were invited via email to participate. RESULTS: Hundred twenty-nine specialists (43.3%) responded to our survey. They rated the residents' competencies in contract negotiating skills, knowledge of the healthcare system, and specialist department poorly. They felt that residents were competent in updating their medical knowledge. Ninety-four percent reported a need for training in management among residents. Preferred topics were time management and healthcare organization. The preferred training method was a workshop given during residency by an extramural expert. CONCLUSION: Dutch medical specialists perceive the management competencies of residents in some areas to be inadequate. They feel that training in medical management during residency is necessary.

Workplace-based assessment: effects of rater expertise.

Traditional psychometric approaches towards assessment tend to focus exclusively on quantitative properties of assessment outcomes. This may limit more meaningful educational approaches towards workplace-based assessment (WBA). Cognition-based models of WBA argue that assessment outcomes are determined by cognitive processes by raters which are very similar to reasoning, judgment and decision making in professional domains such as medicine. The present study explores cognitive processes that underlie judgment and decision making by raters when observing performance in the clinical workplace. It specifically focuses on how differences in rating experience influence information processing by raters. Verbal protocol analysis was used to investigate how experienced and non-experienced raters select and use observational data to arrive at judgments and decisions about trainees' performance in the clinical workplace. Differences between experienced and non-experienced raters were assessed with respect to time spent on information analysis and representation of trainee performance; performance scores; and information processing--using qualitative-based quantitative analysis of verbal data. Results showed expert-novice differences in time needed for representation of trainee performance, depending on complexity of the rating task. Experts paid more attention to situation-specific cues in the assessment context and they generated (significantly) more interpretations and fewer literal descriptions of observed behaviors. There were no significant differences in rating scores. Overall, our findings seemed to be consistent with other findings on expertise research, supporting theories underlying cognition-based models of assessment in the clinical workplace. Implications for WBA are discussed.

Teaching conceptions and approaches to teaching of medical school faculty: the difference between how medical school teachers think about teaching and how they say that they do teach.

BACKGROUND: It is often assumed that the way teachers approach their teaching is determined by the way they think about learning. This study explores how teachers of an undergraduate medical programme (UMP) think about learning, how they approach teaching and whether their conceptions of learning relate to their teaching approaches. METHODS: Quantitative data of academic teachers involved in the undergraduate programme in medicine were collected and analysed. We used a questionnaire designed to measure teachers' conceptions of their own learning (COL) and of student learning as well as teachers' approaches to teaching (AT). RESULTS: Teachers of the medical undergraduate programme hold a variety of COL, of how students learn and their AT. No significant correlations were found between teachers' conceptions of learning and their AT. CONCLUSIONS: Although UMP teachers' ideas on learning and teaching are very diverse, some of their conceptions are interrelated. Teachers' ideas on their own learning is sometimes - but not always - related to how they think about student learning. But most importantly, the way UMP teachers think about learning is not automatically converted into the way they approach teaching.

Exploring the validity and reliability of a questionnaire for evaluating veterinary clinical teachers' supervisory skills during clinical rotations.

BACKGROUND: Feedback to aid teachers in improving their teaching requires validated evaluation instruments. When implementing an evaluation instrument in a different context, it is important to collect validity evidence from multiple sources. AIM: We examined the validity and reliability of the Maastricht Clinical Teaching Questionnaire (MCTQ) as an instrument to evaluate individual clinical teachers during short clinical rotations in veterinary education. METHODS: We examined four sources of validity evidence: (1) Content was examined based on theory of effective learning. (2) Response process was explored in a pilot study. (3) Internal structure was assessed by confirmatory factor analysis using 1086 student evaluations and reliability was examined utilizing generalizability analysis. (4) Relations with other relevant variables were examined by comparing factor scores with other outcomes. RESULTS: Content validity was supported by theory underlying the cognitive apprenticeship model on which the instrument is based. The pilot study resulted in an additional question about supervision time. A five-factor model showed a good fit with the data. Acceptable reliability was achievable with 10-12 questionnaires per teacher. Correlations between the factors and overall teacher judgement were strong. CONCLUSIONS: The MCTQ appears to be a valid and reliable instrument to evaluate clinical teachers' performance during short rotations.

Workplace learning in general practice: supervision, patient mix and independence emerge from the black box once again.

BACKGROUND: Medical students increasingly participate in rotations in primary care settings, such as general practices. Although general practices can differ markedly from hospital settings, research on the instructional quality of general practice rotations is relatively scarce. AIM: We conducted a study to identify which aspects make a significant contribution to the instructional quality of general practice clerkships, as perceived by students. METHOD: After completing their general practice attachment, 155 fifth-year medical students filled out an evaluation questionnaire. Exploratory factor analysis and reliability analysis identified clusters of correlated independent variables. Subsequent regression analysis revealed the relations between the reduced set of independent variables and the dependent variable 'Instructional quality'. RESULTS: Both the quality of supervision and the size and diversity of the patient mix substantially affected students' experienced instructional quality. Opportunities and facilities to perform independently were correlated with instructional quality, but did not affect the instructiveness directly. CONCLUSIONS: Supervision, patient mix and independence are crucial factors for learning in general practice. This is consistent with findings in hospital settings. The perceived quality of instruction hinges on supervision, which is not only the variable most strongly related to instructional quality, but which also affects both the patient mix and students' independence.

The development of shared cognition in paediatric residents analysing a patient video versus a paper patient case.

In a previous study, we established that compared to a written case, a video case enhances observable cognitive processes in the verbal interaction in a postgraduate problem-based learning format. In a new study we examined non-observable cognitive processes using a stimulated recall procedure alongside a reanalysis of the data from the first study. We examined the development of shared cognition as reflected in collaborative concept link formation, an approach to connecting a series of concepts related to a particular topic. Eleven paediatric residents were randomly allocated to two groups. After both analysing the same written case vignette, one group watched a video of the case in the vignette and the other group read a written description of the video. Both groups then reanalysed the vignette. After the group sessions, time-logged transcripts were made of the verbal interaction in both groups and all residents individually took part in a stimulated recall procedure. Causal reasoning concept links were labelled as individual or collaborative depending on whether they originated from individual residents or were directly elicited by verbal utterances from others. The video led to a significantly increased frequency ratio (after intervention: before intervention) of collaborative concept links but did not affect the frequency of individual concept links. This novel process approach to chronological registration of concept link formation offered additional evidence that shared cognition by means of co-elaboration of concept formation is stimulated by the use of patient video recordings in small group learning.

Evaluating the effectiveness of curriculum change. Is there a difference between graduating student outcomes from two different curricula?

BACKGROUND: Changing a curriculum raises the question whether the results, new curriculum student outcomes, are different from old curriculum student outcomes. AIMS: To see whether different curricula produce different outcomes, we compared test and questionnaire results of two cohorts. We wanted to know if there is a difference on knowledge and skills test results, and on the number of times students practiced their skills during their final internships. METHOD: We used two validated test instruments: the Dutch Progress Test (PT) and the Objective Structured Clinical Examination (OSCE). For reporting their skills practice, we asked students of both cohorts to fill out a basic skills questionnaire. RESULTS: We found no significant difference between both cohorts on the results of their knowledge test and their report on skills practice. On the OSCE, students from the new curriculum cohort scored significantly higher than old curriculum students. CONCLUSION: Curriculum change can lead to improvements in graduating students' outcome results.

Does a faculty development programme improve teachers' perceived competence in different teacher roles?

BACKGROUND: Changing concepts of education have led many medical schools to adopt student-centred approaches to teaching, requiring different teaching competencies than more traditional approaches. AIMS: The aim of this study was to investigate whether participation in a faculty development (FD) programme had a positive effect on veterinary medical teachers' perceptions of their competence in seven different teaching roles. METHOD: All teaching staff of a veterinary medical school (251) were invited to complete a questionnaire asking about their perceived competence in seven teaching roles. A stepwise multiple linear regression analysis examined the effect of the completion of a FD programme on perceived teaching competence. RESULTS: There was a significant positive effect of the FD programme on teachers' perceived competence in majority of the teaching roles at issue. CONCLUSIONS: FD appeared to have a positive effect on the perceived competence of veterinary medical teachers. Further research should investigate whether FD actually results in improved teaching performance.

Evaluation of a course on patient records.

OBJECTIVES: To determine whether self-directed learning about (electronic) patient records during a PBL (problem-based learning) block, dealing with the content of disciplines concerned with the diagnosis and therapy of diseases of the abdomen, increased the knowledge of the students with respect to the patient records. METHODS: At the beginning and at the end of the ten-week block the same questionnaire was offered to the students (180). Cohen's d for effect size was used to determine the increase in knowledge. RESULTS: For those students that answered the questionnaire twice (53), a Cohen's d of 0.94 was obtained. CONCLUSIONS: The knowledge of the students concerning the advantages and limitations of (electronic) patient records increased significantly. The corresponding effect size was large.

Relative or Absolute Standards in Assessing Medical Knowledge Using Progress Tests.

Norm-referenced pass/fail decisions are quite common in achievement testing in health sciences education. The use of relative standards has the advantage of correcting for variations in test-difficulty. However, relative standards also show some serious drawbacks, and the use of an absolute and fixed standard is regularly preferred. The current study investigates the consequences of the use of an absolute instead of a relative standard. The performance of the developed standard setting procedure was investigated by using actual progress test scores obtained at the Maastricht medical school in an episode of eight years. When the absolute instead of the relative standard was used 6% of the decisions changed: 2.6% of the outcomes changed from fail to pass, and 3.4% from pass to fail. The failure rate, which was approximately constant when using the relative standard, varied from 2% to 47% for different tests when an absolute standard was used. It is concluded that an absolute standard is precarious because of the variations in difficulty of tests.

Measuring potential predictors of burnout and engagement among young veterinary professionals; construction of a customised questionnaire (the Vet-DRQ).

The Job Demands-Resources model (JD-R model) was used as the theoretical basis of a tailormade questionnaire to measure the psychosocial work environment and personal resources of recently graduated veterinary professionals. According to the JD-R model, two broad categories of work characteristics that determine employee wellbeing can be distinguished: job demands and job resources. Recently, the JD-R model has been expanded by integrating personal resource measures into the model. Three semistructured group interviews with veterinarians active in different work domains were conducted to identify relevant job demands, job resources and personal resources. These demands and resources were organised in themes (constructs). For measurement purposes, a set of questions ('a priori scale') was selected from the literature for each theme. The full set of a priori scales was included in a questionnaire that was administered to 1760 veterinary professionals. Exploratory factor analysis and reliability analysis were conducted to arrive at the final set of validated scales (final scales). 860 veterinarians (73 per cent females) participated. The final set of scales consisted of seven job demands scales (32 items), nine job resources scales (41 items), and six personal resources scales (26 items) which were considered to represent the most relevant potential predictors of work-related wellbeing in this occupational group. The procedure resulted in a tailormade questionnaire: the Veterinary Job Demands and Resources Questionnaire (Vet-DRQ). The use of valid theory and validated scales enhances opportunities for comparative national and international research.

Burnout and engagement, and its predictors in young veterinary professionals: the influence of gender.

With the aim to assess levels of burnout and work engagement in junior veterinarians and identify predictors of burnout and work engagement in male and female veterinarians, 1760 veterinarians who had graduated in The Netherlands between 1999 and 2009, received an electronic questionnaire. Veterinarians numbering 860 (73 per cent females) participated. Levels of exhaustion, cynicism and work engagement were significantly lower compared to the norm group (a random sample of the Dutch working population). Male veterinarians were less exhausted and more engaged than female veterinarians. Exhaustion decreased over the years. Job demands positively related to exhaustion were work-home interference and workload. Job resources positively related to work engagement were opportunities for professional development and skills discretion (ie, the ability to use and develop skills on the job. Personal resources explained more of the variance in work engagement of female and male veterinarians (31 per cent and 42 per cent) than of the variance in exhaustion (19 per cent and 21 per cent) and cynicism (19 per cent and 10 per cent). Personal resources positively related to work engagement were self-efficacy and proactive behaviour. Relative importance analysis revealed differences between men and women in the importance of various job demands, job resources and personal resources in explaining burnout and engagement in young veterinary professionals.

Is an Angoff standard an indication of minimal competence of examinees or of judges?

BACKGROUND: To establish credible, defensible and acceptable passing scores for written tests is a challenge for health profession educators. Angoff procedures are often used to establish pass/fail decisions for written and performance tests. In an Angoff procedure judges' expertise and professional skills are assumed to influence their ratings of the items during standard-setting. The purpose of this study was to investigate the impact of judges' item-related knowledge on their judgement of the difficulty of items, and second, to determine the stability of differences between judges. METHOD: Thirteen judges were presented with two sets of 60 items on different occasions. They were asked to not only judge the difficulty of the items but also to answer them, without the benefit of the answer key. For each of the 120 items an Angoff estimate and an item score were obtained. The relationship between the Angoff estimate and the item score was examined by applying a regression analysis to the 60 items (Angoff estimate, score) for each judge at each occasion. RESULTS AND CONCLUSIONS: This study shows that in standard-setting the individual judgement of the individual item is not only a reflection of the difficulty of the item but also of the inherent stringency of the judge and his/her subject-related knowledge. Considerable variation between judges in their stringency was found, and Angoff estimates were significantly affected by a judge knowing or not knowing the answer to the item. These findings stress the importance of a careful selection process of the Angoff judges when making pass/fail decisions in health professions education. They imply that judges should be selected who are not only capable of conceptualising the 'minimally competent student', but who would also be capable of answering all the items.

Comparison of text and video cases in a postgraduate problem-based learning format.

OBJECTIVE: To investigate whether adding a brief video case instead of an equivalent written text improves the cognitive and metacognitive processes (data exploration, theory building, theory evaluation and metareasoning) of residents in problem-based learning. SUBJECTS: Paediatric residents. SETTING: Department of Paediatrics, Arhus University Hospital, Skejby, Denmark. METHODS: Eleven residents were each assigned to 1 of 2 groups. Both groups analysed an identical vignette of a patient case. Immediately after this, 1 group watched a 2.5-minute video recording and the other group read a description of the same video recording. The groups then reanalysed the case. Thinking processes were tapped by recording and analysing the verbal group interaction. The frequencies of clause categories were calculated and compared using chi-square tests. RESULTS: The verbal interaction showed statistically significant improvements in data exploration, theory building and theory evaluation after the video case. CONCLUSIONS: The greater improvement in clause frequency after the video case compared with the equivalent text suggests an improvement in data exploration, theory building and theory evaluation. The use of patient video recordings may be a valuable supplement to postgraduate paediatric training, particularly pertaining to movement disorders.

Panel expertise for an Angoff standard setting procedure in progress testing: item writers compared to recently graduated students.

INTRODUCTION: An earlier study showed that an Angoff procedure with > or = 10 recently graduated students as judges can be used to estimate the passing score of a progress test. As the acceptability and feasibility of this approach are questionable, we conducted an Angoff procedure with test item writers as judges. This paper reports on the reliability and credibility of this procedure and compares the standards set by the two different panels. METHODS: Fourteen item writers judged 146 test items. Recently graduated students had assessed these items in a previous study. Generalizability was investigated as a function of the number of items and judges. Credibility was judged by comparing the pass/fail rates associated with the Angoff standard, a relative standard and a fixed standard. The Angoff standards obtained by item writers and graduates were compared. RESULTS: The variance associated with consistent variability of item writers across items was 1.5% and for graduate students it was 0.4%. An acceptable error score required 39 judges. Item-Angoff estimates of the two panels and item P-values correlated highly. Failure rates of 57%, 55% and 7% were associated with the item writers' standard, the fixed standard and the graduates' standard, respectively. CONCLUSION: The graduates' and the item writers' standards differed substantially, as did the associated failure rates. A panel of 39 item writers is not feasible. The item writers' passing score appears to be less credible. The credibility of the graduates' standard needs further evaluation. The acceptability and feasibility of a panel consisting of both students and item writers may be worth investigating.

Setting a standard for performance assessment of doctor-patient communication in general practice.

CONTEXT: Continuing professional development (CPD) of general practitioners. OBJECTIVE: Criterion-referenced standards for assessing performance in the real practice of general practitioners (GPs) should be available to identify learning needs or poor performers for CPD. The applicability of common standard setting procedures in authentic assessment has not been investigated. METHODS: To set a standard for assessment of GP-patient communication with video observation of daily practice, we investigated 2 well known examples of 2 different standard setting approaches. An Angoff procedure was applied to 8 written cases. A borderline regression method was applied to videotaped consultations of 88 GPs. The procedures and outcomes were evaluated by the applicability of the procedure, the reliability of the standards and the credibility as perceived by the stakeholders, namely, the GPs. RESULTS: Both methods are applicable and reliable; the obtained standards are credible according to the GPs. CONCLUSIONS: Both modified methods can be used to set a standard for assessment in daily practice. The context in which the standard will be used - i.e. the specific purpose of the standard, the moment the standard must be available or if specific feedback must be given - is important because methods differ in practical aspects.

Cross institutional collaboration in assessment: a case on progress testing.

The practice of assessment is governed by an interesting paradox. On the one hand good assessment requires substantial resources which may exceed the capacity of a single institution and we have reason to doubt the quality of our in-house examinations. On the other hand, our parsimonity with regard to our resources makes us reluctant to pool efforts and share our test material. This paper reports on an initiative to share test material across different medical schools. Three medical schools in The Netherlands have successfully set up a partnership for a specific testing method: progress testing. At present, these three schools collaboratively produce high-quality test items. The jointly produced progress tests are administered concurrently by these three schools and one other school, which buys the test. The steps taken in establishing this partnership are described and results are presented to illustrate the unique sort of information that is obtained by cross-institutional assessment. In addition, plans to improve test content and procedure and to expand the partnership are outlined. Eventually, the collaboration may even extend to other test formats. This article is intended to give evidence of the feasibility and exciting potential of between school collaboration in test development and test administration. Our experiences have demonstrated that such collaboration has excellent potential to combine economic benefit with educational advantages, which exceed what is achievable by individual schools.