The limited use of instructional design guidelines in healthcare simulation scenarios: an expert appraisal.

BACKGROUND: Systematic reviews on simulation training effectiveness have pointed to the need to adhere to evidence-based instructional design (ID) guidelines. ID guidelines derive from sound cognitive theories and aim to optimize complex learning (integration of knowledge, skills, and attitudes) and learning transfer (application of acquired knowledge and skills in the workplace). The purpose of this study was to explore adherence to ID guidelines in simulation training programs for dealing with postpartum hemorrhage (PPH), a high-risk situation and the leading cause of maternal mortality worldwide. METHODS: A total of 40 raters analyzed simulation training programs as described in 32 articles. The articles were divided into four subsets of seven articles and one subset of four articles. Each subset was judged by seven to ten raters on adherence to ID guidelines. The 5-point Likert score rating scale was based on Merrill's First Principles of Instruction and included items relating to key ID features categorized into five subscales: authenticity, activation of prior knowledge, demonstration, application, and integration/transfer. The authors searched for articles published in English between January 2007 and March 2017 in PubMed, Eric, and Google Scholar and calculated the mean Likert-scale score, per subscale, and interrater reliability (IRR). RESULTS: The mean Likert-scale scores calculated for all subscales were < 3.00. For the number of raters used to judge the papers in this study (varying between 7 and 10), the IRR was found to be excellent for the authenticity and integration/transfer subscales, good-to-excellent for the activation of prior knowledge and application subscales, and fair-to-good for the demonstration subscale. CONCLUSION: The results demonstrate a paucity of the description of adherence to evidence-based ID guidelines in current simulation trainings for a high-risk situation such as PPH.

Effects of an in situ instructional design based postpartum hemorrhage simulation training on patient outcomes: an uncontrolled before-and-after study.

Objective: To compare postpartum hemorrhage (PPH) patient outcomes before and after an in situ instructional design based PPH simulation attended by obstetrics and gynecology (OBGYN) residents.Methods: This uncontrolled before-and-after study was conducted in Recife, Brazil including all 1388 women delivering from June to August 2012 and all 1357 delivering from June to August 2013. The 36 OBGYN residents were divided into13 teams of two or three participants and were trained through ID based PPH simulation training with the following eight steps: (1) prior knowledge activation, (2) video demonstration, (3) dual-coding PPH protocol discussion-an image association during the training, (4) training scenario # 1, (5) debriefing, (6) training scenario # 2 with immediate feedback, (7) training scenario # 3, and (8) debriefing with self-assessment. The training scenarios had an increasing level of complexity. The main goal of the training was the adequate management of PPH and situational awareness improvement-the ability to anticipate, recognize, and intercept unfolding error chains. The primary patient outcomes rates used for the before and after comparison were therapeutic uterotonics use within 24 h of birth and blood transfusion. Secondary outcomes were therapeutic oxytocin mean dosage IU within 24 h of birth, postpartum Hb < 6 g/dL, among others. Chi-square test was used for categorical variables comparison and independent t-test for continuous variables.Results: PPH rates were 100 (7.2% of 2012 deliveries) and 80 cases (5.9% of 2013 deliveries), respectively. Comparison of primary post- and pre-simulation outcomes revealed no significant differences. However, in the comparison for therapeutic oxytocin mean dosage IU within 24 h of birth, there was an increase found after the simulation (15.98 ± 7.4 versus 25.1 ± 12.3; p < .001). For all other outcome measures, there were no statistical differences.Conclusions: In situ ID based PPH simulation leads to an increase in the mean dosage of oxytocin after training, in selected cases. This may indicate better situational awareness when managing women with PPH.

High enthusiasm about long lasting mentoring relationships and older mentors.

BACKGROUND: Mentoring plays a pivotal role in workplace-based learning, especially in the medical realm. Organising a formal mentoring programme can be labor and time intensive and generally impractical in resource constrained medical schools with limited numbers of mentors. Hence, informal mentoring offers a valuable alternative, but will be more likely to be effective when mentors and protégés share similar views. It is therefore important to gain more insight into factors influencing perceptions of informal mentoring. This study aims to explore mentors and protégés' perceptions of informal mentoring and how these vary (or not) with gender, age and the duration of the relationship. METHOD: We administered an Informal Mentor Role Instrument (IMRI) to medical practitioners and academics from Egypt, Pakistan and Saudi Arabia. The questionnaire was developed for the study from other validated instruments. It contained 39 items grouped into 7 domains: acceptance, counselling, friendship, parenting, psychological support, role modelling and sociability. RESULTS: A total of 103 mentors and 91 protégés completed the IMRI. Mentors had a better appreciation for the interpersonal aspects of informal mentoring than protégés, especially regarding acceptance, counselling and friendship. Moreover, being older and engaged in a longer mentoring relationship contributed to more positive perceptions of interpersonal aspects of mentoring, regardless of one's role (mentor or protégé). CONCLUSION: It can be concluded that the expectations of mentors and protégés differed regarding the content and aim of the interpersonal characteristics of their mentoring relationship. We recommend mentors and protégés to more explicitly exchange their expectations of the informal mentoring relationship, as typically practiced in formal mentoring. Additionally, in our study, seniority and lasting relationships seem crucial for good informal mentoring. It appears beneficial to foster lasting informal mentoring relationships and to give more guidance to younger mentors.

Exploring medical residents' perceived need for negotiation skills training.

OBJECTIVES: This study explores the optimal focus for negotiation skills development training by investigating how often medical residents negotiate in practice, and how they perceive the effectiveness of their negotiation capabilities. METHODS: An exploratory study was performed using a questionnaire regarding the medical residents' working environment, negotiation frequency, knowledge and skills using a 5-point Likert scale, multiple choice questions and open questions. Exploratory factor analysis with principal component analysis, varimax rotation, reliability analysis, and content analysis were used to reduce the number of variables. Descriptive and interferential statistics and multiple regression analysis were used to analyze the data. RESULTS: We analyzed the responses of 60 medical residents. The findings showed that the perceived development of their negotiation knowledge (M=3.06, SD=0.83) was less than their negotiation skills (M=3.69, SD=0.47). Their attitude during negotiations, especially females, differed substantially in the interactions with nurses than with their supervisors. Medical residents with more working experience, better negotiation skills or who worked in hierarchical environments negotiated more frequently with their supervisors. Medical residents with better collaboration skills and negotiation knowledge demonstrated better negotiation skills. CONCLUSIONS: This study underlines medical residents' need for negotiation training. In addition to the basic negotiation knowledge and skills, training programs in negotiation should focus on the medical residents' awareness of their attitudes during negotiations, combining the assertiveness shown in interactions with supervisors with the empathy and emotional engagement present in interactions with nurses.  Furthermore, attention should be paid to the influence of the environmental hierarchy on negotiation skill development.

Does selection pay off? A cost-benefit comparison of medical school selection and lottery systems.

CONTEXT: Resources for medical education are becoming more constrained, whereas accountability in medical education is increasing. In this constrictive environment, medical schools need to consider and justify their selection procedures in terms of costs and benefits. To date, there have been no studies focusing on this aspect of selection. OBJECTIVES: We aimed to examine and compare the costs and benefits of two different approaches to admission into medical school: a tailored, multimethod selection process versus a lottery procedure. Our goal was to assess the relative effectiveness of each approach and to compare these in terms of benefits and costs from the perspective of the medical school. METHODS: The study was conducted at Maastricht University Medical School, at which the selection process and a weighted lottery procedure ran in parallel for 3 years (2011-2013). The costs and benefits of the selection process were compared with those of the lottery procedure over three student cohorts throughout the Bachelor's programme. The extra costs of selection represented the monetary investment of the medical school in conducting the selection procedure; the benefits were derived from the increase in income generated by the prevention of dropout and the reductions in extra costs facilitated by decreases in the repetition of blocks and objective structured clinical examinations. RESULTS: The tailor-made selection procedure cost about €139 000 when extrapolated to a full cohort of students (n = 286). The lottery procedure came with negligible costs for the medical school. However, the average benefits of selection compared with the lottery system added up to almost €207 000. CONCLUSIONS: This study not only shows that conducting a cost-benefit comparison is feasible in the context of selection for medical school, but also that an 'expensive' selection process can be cost-beneficial in comparison with an 'inexpensive' lottery system. We encourage other medical schools to examine the cost-effectiveness of their own selection processes in relation to student outcomes in order to extend knowledge on this important topic.

Selection into medicine: the predictive validity of an outcome-based procedure.

BACKGROUND: Medical schools must select students from a large pool of well-qualified applicants. A challenging issue set forward in the broader literature is that of which cognitive and (inter)personal qualities should be measured to predict diverse later performance. To address this gap, we designed a 'backward chaining' approach to selection, based on the competences of a 'good doctor'. Our aim was to examine if this outcome-based selection procedure was predictive of study success in a medical bachelor program. METHODS: We designed a multi-tool selection procedure, blueprinted to the CanMEDS competency framework. The relationship between performance at selection and later study success across a three-year bachelor program was examined in three cohorts. Study results were compared between selection-positive and selection-negative (i.e. primarily rejected) students. RESULTS: Selection-positive students outperformed their selection-negative counterparts throughout the entire bachelor program on assessments measuring cognitive (e.g. written exams), (inter)personal and combined outcomes (i.e. OSCEs). Of the 30 outcome variables, selection-positive students scored significantly higher in 11 cases. Fifteen other, non-significant between-group differences were also in favor of the selection-positives. An overall comparison using a sign test indicated a significant difference between both groups (p < 0.001), despite equal pre-university GPAs. CONCLUSIONS: The use of an outcome-based selection approach seems to address some of the predictive validity limitations of commonly-used selection tools. Selection-positive students significantly outperformed their selection-negative counterparts across a range of cognitive, (inter)personal, and mixed outcomes throughout the entire three-year bachelor in medicine.

Development and validation of the TOCO-TURBT tool: a summative assessment tool that measures surgical competency in transurethral resection of bladder tumour.

BACKGROUND: The current shift towards competency-based residency training has increased the need for objective assessment of skills. In this study, we developed and validated an assessment tool that measures technical and non-technical competency in transurethral resection of bladder tumour (TURBT). METHODS: The 'Test Objective Competency' (TOCO)-TURBT tool was designed by means of cognitive task analysis (CTA), which included expert consensus. The tool consists of 51 items, divided into 3 phases: preparatory (n = 15), procedural (n = 21), and completion (n = 15). For validation of the TOCO-TURBT tool, 2 TURBT procedures were performed and videotaped by 25 urologists and 51 residents in a simulated setting. The participants' degree of competence was assessed by a panel of eight independent expert urologists using the TOCO-TURBT tool. Each procedure was assessed by two raters. Feasibility, acceptability and content validity were evaluated by means of a quantitative cross-sectional survey. Regression analyses were performed to assess the strength of the relation between experience and test scores (construct validity). Reliability was analysed by generalizability theory. RESULTS: The majority of assessors and urologists indicated the TOCO-TURBT tool to be a valid assessment of competency and would support the implementation of the TOCO-TURBT assessment as a certification method for residents. Construct validity was clearly established for all outcome measures of the procedural phase (all r > 0.5, p < 0.01). Generalizability-theory analysis showed high reliability (coefficient Phi ≥ 0.8) when using the format of two assessors and two cases. CONCLUSIONS: This study provides first evidence that the TOCO-TURBT tool is a feasible, valid and reliable assessment tool for measuring competency in TURBT. The tool has the potential to be used for future certification of competencies for residents and urologists. The methodology of CTA might be valuable in the development of assessment tools in other areas of clinical practice.

Cross-validation of a learning climate instrument in a non-western postgraduate clinical environment.

BACKGROUND: In postgraduate training, there is a need to continuously assess the learning and working conditions to optimize learning. Students or trainees respond to the learning climate as they perceive it. The Dutch Residency Educational Climate Test (D-RECT) is a learning climate measurement tool with well-substantiated validity. However, it was originally designed for Dutch postgraduate trainees and it remains to be shown whether extrapolation to non-Western settings is viable. The dual objective of this study was to revalidate D-RECT outside of a Western setting and to evaluate the factor structure of a recently revised version of the D-RECT containing 35 items. METHODS: We invited Filipino internal medicine residents from 96 hospitals to complete the revised 35-item D-RECT. Subsequently, we performed a confirmatory factor analysis to check the fit of the 9 scale model of the revised 35-item D-RECT. Inter-rater reliability was assessed using generalizability theory. RESULTS: Confirmatory factor analysis unveiled that the factor structure of the revised 35-item D-RECT provided a reasonable fit to the Filipino data, after removal of 7 items. Five to seven evaluations of individual residents were needed per scale to obtain a reliable result. CONCLUSION: Even in a non-Western setting, the D-RECT exhibited psychometric validity. This study validated the factor structure of the revised 35-item D-RECT after some modifications. We recommend that its application be extended to other Asian countries and specialties.

Selection and ranking of patient video cases in paediatric neurology in relation to learner levels.

BACKGROUND: Teaching and learning with patient video cases may add authenticity, enhance diagnostic accuracy and improve chances of early diagnosis. The aim of this study is firstly to identify selection criteria for key Patient video cases (PVCs), secondly to identify trends in relevance of PVCs for learner levels and thirdly, to rank PVCs for learner levels. METHODS: Based on a literature review, we identified criteria for key PVCs for use in paediatric neurology. We then performed a multi-round Delphi analysis to obtain agreement between 28 expert clinician teachers concerning key PVCs for four learner levels. RESULTS: We identified two major criteria: key PVCs should demonstrate key movements, and these movements should be subtle and/or difficult to note. The expert clinician teachers subsequently assessed a list of 14 topics for key PVCs. We found a clear, increasing trend in relevance scores, from medical students to young residents to experienced residents and specialists. For medical students and residents, epileptic spasms, Down syndrome, developmental delay, cerebral palsy and absence epilepsy were highly ranked. For specialists, conditions like chorea, focal seizures or eye movement disorders topped the ranking list, although ranking was less clear for this group of advanced learners. DISCUSSION AND CONCLUSION: Key PVCs should demonstrate movements that are difficult to note for learners. Ranked lists of key PVCs for teaching and learning at different learner levels are now available and may help institutions build validated local libraries of PVCs.

Stakeholders' perceptions on competency and assessment program of entry-level pharmacists in developing countries.

OBJECTIVES: To assess the stakeholders' perceptions on the competency of entry-level pharmacists and the use of written licensure examination as the primary assessment for licensure decisions on entry-level pharmacists who have completed the Pharmacy Internship Program1 (PIP) in developing countries. METHOD: A cross-sectional survey was conducted among stakeholders in which they completed a web-based 21-item pre-tested questionnaire to determine their views regarding the competency outcomes and assessment program for entry-level pharmacist. RESULTS: The stakeholders rated the entry-level pharmacists to possess all competencies except research skills. Stakeholders suggested improvement of the program by defining the competency framework and training preceptors. However, stakeholders disagree on using written examination as the primary assessment for licensure decision and suggested the incorporation of other performance-based assessments like preceptor's assessment reports. CONCLUSION: Stakeholders are uncertain on entry-level pharmacists in developing countries possessing adequate research competencies and think their assessment program for licensure need more than written examination to assess all required competencies.

Toward a better judgment of item relevance in progress testing.

BACKGROUND: Items must be relevant to ensure item quality and test validity. Since "item relevance" has not been operationalized yet, we developed a rubric to define it. This study explores the influence of this rubric on the assessment of item relevance and on inter-rater agreement. METHODS: Members of the item review committee (RC) and students, teachers, and alumni (STA) reassessed the relevance of 50 previously used progress test (PT) items and decided about their inclusion using a 5-criteria rubric. Data were analyzed at item level using paired samples t-tests, Intraclass Correlation Coefficients (ICC), and linear regression analysis, and at rater level in a generalizability analysis per group. RESULTS: The proportion of items that the RC judged relevant enough to be included decreased substantially from 1.00 to 0.72 (p < 0.001). Agreement between the RC and STA was high, with an ICC of >0.7 across items. The relation between inclusion and relevance was strong (correlation = 0.89, p < 0.001), and did not differ between RC and STA. To achieve an acceptable inter-rater reliability for relevance and inclusion, 6 members must serve on the RC. CONCLUSIONS: Use of the rubric results in a stricter evaluation of items' appropriateness for inclusion in the PT and facilitates agreement between the RC and other stakeholders. Hence, it may help increase the acceptability and validity of the PT.

Electronic assessment of clinical reasoning in clerkships: A mixed-methods comparison of long-menu key-feature problems with context-rich single best answer questions.

BACKGROUND: It remains unclear which item format would best suit the assessment of clinical reasoning: context-rich single best answer questions (crSBAs) or key-feature problems (KFPs). This study compared KFPs and crSBAs with respect to students' acceptance, their educational impact, and psychometric characteristics when used in a summative end-of-clinical-clerkship pediatric exam. METHODS: Fifth-year medical students (n = 377) took a computer-based exam that included 6-9 KFPs and 9-20 crSBAs which assessed their clinical reasoning skills, in addition to an objective structured clinical exam (OSCE) that assessed their clinical skills. Each KFP consisted of a case vignette and three key features using a "long-menu" question format. We explored students' perceptions of the KFPs and crSBAs in eight focus groups and analyzed statistical data of 11 exams. RESULTS: Compared to crSBAs, KFPs were perceived as more realistic and difficult, providing a greater stimulus for the intense study of clinical reasoning, and were generally well accepted. The statistical analysis revealed no difference in difficulty, but KFPs resulted more reliable and efficient than crSBAs. The correlation between the two formats was high, while KFPs correlated more closely with the OSCE score. CONCLUSIONS: KFPs with long-menu exams seem to bring about a positive educational effect without psychometric drawbacks.

Stakeholder perspectives on workplace-based performance assessment: towards a better understanding of assessor behaviour.

Workplace-Based Assessment (WBA) plays a pivotal role in present-day competency-based medical curricula. Validity in WBA mainly depends on how stakeholders (e.g. clinical supervisors and learners) use the assessments-rather than on the intrinsic qualities of instruments and methods. Current research on assessment in clinical contexts seems to imply that variable behaviours during performance assessment of both assessors and learners may well reflect their respective beliefs and perspectives towards WBA. We therefore performed a Q methodological study to explore perspectives underlying stakeholders' behaviours in WBA in a postgraduate medical training program. Five different perspectives on performance assessment were extracted: Agency, Mutuality, Objectivity, Adaptivity and Accountability. These perspectives reflect both differences and similarities in stakeholder perceptions and preferences regarding the utility of WBA. In comparing and contrasting the various perspectives, we identified two key areas of disagreement, specifically 'the locus of regulation of learning' (i.e., self-regulated versus externally regulated learning) and 'the extent to which assessment should be standardised' (i.e., tailored versus standardised assessment). Differing perspectives may variously affect stakeholders' acceptance, use-and, consequently, the effectiveness-of assessment programmes. Continuous interaction between all stakeholders is essential to monitor, adapt and improve assessment practices and to stimulate the development of a shared mental model. Better understanding of underlying stakeholder perspectives could be an important step in bridging the gap between psychometric and socio-constructivist approaches in WBA.

The use of instructional design guidelines to increase effectiveness of postpartum hemorrhage simulation training.

OBJECTIVE: To compare learning outcomes of postpartum hemorrhage simulation training based on either instructional design guidelines or best practice. METHODS: A pretest-post-test non-equivalent groups study was conducted among obstetrics and gynecology residents in Recife, Brazil, from June 8 to August 30, 2013. The instructional design group included 13 teams, whereas the best practice group included seven teams. A standardized task checklist was used for scenario analysis and the proportion of correctly executed tasks compared (post-test minus pretest). RESULTS: The instructional design group scored higher than the best practice group for total number of tasks completed (median difference 0.46 vs 0.17; P<0.001; effect size [r]=0.72). Similar results were observed for communication (median difference 0.56 vs 0.22; P=0.004; r=0.58), laboratory evaluation (median difference 0.83 vs 0.00; P<0.001; r=0.76), and mechanical management (median difference 0.25 vs -0.15; P=0.048; r=0.39). Speed of learning was also increased. The median differences were 0.20 for the instructional design group compared with 0.05 for the best practice group at 60 seconds (P=0.015; r=0.49), and 0.49 versus 0.26 (P=0.001; r=0.65) at 360 seconds. CONCLUSION: The use of simulation training for postpartum hemorrhage that was based on instructional design guidelines yielded better learning outcomes than did training based on best practice.

High educational impact of a national simulation-based urological curriculum including technical and non-technical skills.

BACKGROUND: Although simulation training is increasingly used to meet modern technology and patient safety demands, its successful integration within surgical curricula is still rare. The Dutch Urological Practical Skills (D-UPS) curriculum provides modular simulation-based training of technical and non-technical basic urological skills in the local hospital setting. This study aims to assess the educational impact of implementing the D-UPS curriculum in the Netherlands and to provide focus points for improvement of the D-UPS curriculum according to the participants. METHODS: Educational impact was assessed by means of qualitative individual module-specific feedback and a quantitative cross-sectional survey among residents and supervisors. Twenty out of 26 Dutch teaching hospitals participated. The survey focussed on practical aspects, the D-UPS curriculum in general, and the impact of the D-UPS curriculum on the development of technical and non-technical skills. RESULTS: A considerable survey response of 95 % for residents and 76 % for supervisors was obtained. Modules were attended by junior and senior residents, supervised by a urologist, and peer teaching was used. Ninety percent of supervisors versus 67 % of residents judged the D-UPS curriculum as an important addition to current residency training (p = 0.007). Participants' aggregated general judgement of the modules showed a substantial percentage favorable score (M ± SE: 57 ± 4 %). The impact of training on, e.g., knowledge of materials/equipment and ability to anticipate on complications was high, especially for junior residents (77 ± 5 and 71 ± 7 %, respectively). Focus points for improvement of the D-UPS curriculum according to the participants include adaptation of the training level to residents' level of experience and focus on logistics. CONCLUSION: The simulation-based D-UPS curriculum has a high educational impact. Residents and supervisors consider the curriculum to be an important addition to current residency training. Focus points for improvement of the D-UPS curriculum according to the participants include increased attention to logistics and integration of a spiral learning approach.

A first report of East Asian students' perception of progress testing: a focus group study.

BACKGROUND: Progress testing (PT) is used in Western countries to evaluate students' level of functional knowledge, and to enhance meaning-oriented and self-directed learning. However, the use of PT has not been investigated in East Asia, where reproduction-oriented and teacher-centered learning styles prevail. Here, we explored the applicability of PT by focusing on student perceptions. METHODS: Twenty-four students from Years 2, 3, and 5 at Jichi Medical University in Japan attended a pilot PT session preceded by a brief introduction of its concept and procedures. Variations in obtained test scores were analyzed by year, and student perceptions of PT were explored using focus groups. RESULTS: Formula scores (mean ± standard deviation) in Years 2, 3, and 5 were 12.63 ± 3.53, 35.88 ± 14.53, and 71.00 ± 18.31, respectively. Qualitative descriptive analysis of focus group data showed that students disfavored testing of medical knowledge without tangible goals, but instead favored repetitive assessment of knowledge that had been learned and was tested on a unit basis in the past in order to achieve deep learning. Further, students of all school years considered that post-test explanatory lectures by teachers were necessary. CONCLUSIONS: East Asian students' perceptions indicated that, in addition to their intensive memorization within narrow test domains compartmentalized by end-of-unit tests, the concept of PT was suitable for repetitive memorization, as it helped them to integrate their knowledge and to increase their understanding. Post-test explanatory lectures might lessen their dislike of the intangible goals of PT, but at the expense of delaying the development of self-directed learning. Key issues for the optimization of PT in East Asia may include administration of PT after completed end-of-unit tests and a gradual change in feedback methodology over school years from test-oriented post-test lectures to the provision of literature references only, as a means of enhancing test self-review and self-directed learning.

Content validity of workplace-based portfolios: A multi-centre study.

BACKGROUND: Portfolios are used as tools to coach and assess students in the workplace. This study sought to evaluate the content validity of portfolios as reflected in their capacity to adequately assess achieved competences of medical students during clerkships. METHODS: We reviewed 120 workplace portfolios at three medical universities (Belgium and the Netherlands). To validate their content, we developed a Validity Inventory for Portfolio Assessment (VIPA) based on the CanMEDS roles. Two raters evaluated each portfolio and indicated for each VIPA item whether the portfolio provided sufficient information to enable satisfactory assessment of the item. We ran a descriptive analysis on the validation data and computed Cohen's Kappa to investigate interrater agreement. RESULTS: The portfolios adequately covered the items pertaining to the communicator (90%) and professional (87%) roles. Coverage of the medical expert, collaborator, scholar and manager roles ranged between 75% and 85%. The health advocate role, covering 59%, was clearly less well represented. This role also exhibited little interrater agreement (Kappa < 0.4). CONCLUSIONS: This study lends further credence to the evidence that portfolios can indeed adequately assess the different CanMEDS roles during clerkships, the health advocate role, which was less well represented in the portfolio content, excepted.

The Simbla TURBT Simulator in Urological Residency Training: From Needs Analysis to Validation.

OBJECTIVE: To investigate the value of the physical Simbla Transurethral Resection of a Bladder Tumor (TURBT) simulator as an educational tool within urological residency training, by means of a training needs analysis (TNA) and assessment of its feasibility, acceptability, and face, content, and construct validity. METHODS: To analyze the training needs for TURBT, procedural steps and pitfalls were identified and the TNA was completed during an expert consensus meeting. Participants (n = 76) were divided into three groups based on their experience in TURBT: novices, intermediates, and experts. Participants performed two standardized TURBT procedures on the simulator. Face validity and content validity, as well as feasibility and acceptability, were assessed with a quantitative survey. Construct validity was assessed by comparing the performance of novices, intermediates, and experts on resection time, quality of tumor resection, and overall performance. RESULTS: Of the 21 procedural steps and 17 pitfalls defined in TNA, 13 steps and 8 pitfalls were covered by the Simbla. Participants rated the Simbla's overall realism (face validity) with a score of 8 of 10 (range 6-9). The simulator was judged to be most useful (content validity) for learning eye-hand coordination: score 8 (6-10). All aspects regarding realism and usefulness were rated above the acceptability threshold of 6/10. Intermediates (100%) and experts (96%) considered the Simbla to be a useful educational tool within the urological curriculum. Resection time was longer for novices than for experts (p < 0.05; construct validity). In addition, the overall performance of novices was rated lower compared with intermediates and experts, and novices showed more irradical resections and bladder perforations (all p < 0.05). CONCLUSIONS: The Simbla TURBT simulator is a valid, feasible, and acceptable educational tool for training procedural skills and may be implemented in the urological curriculum to complement learning in clinical practice. TNA is valuable in defining training objectives and evaluating the educational value of a simulator.

Effect of comprehensive oncogenetics training interventions for general practitioners, evaluated at multiple performance levels.

General practitioners (GPs) are increasingly called upon to identify patients at risk for hereditary cancers, and their genetic competencies need to be enhanced. This article gives an overview of a research project on how to build effective educational modules on genetics, assessed by randomized controlled trials (RCTs), reflecting the prioritized educational needs of primary care physicians. It also reports on an ongoing study to investigate long-term increase in genetic consultation skills (1-year follow-up) and interest in and satisfaction with a supportive website on genetics among GPs. Three oncogenetics modules were developed: an online Continuing Professional Development (G-eCPD) module, a live genetic CPD module, and a "GP and genetics" website (huisartsengenetica.nl) providing further genetics information applicable in daily practice. Three assessments to evaluate the effectiveness (1-year follow-up) of the oncogenetic modules were designed: 1.An online questionnaire on self-reported genetic competencies and changes in referral behaviour, 2.Referral rates from GPs to clinical genetics centres and 3.Satisfaction questionnaire and visitor count analytics of supportive genetics website. The setting was Primary care in the Netherlands and three groups of study participants were included in the reported studies:. Assessment 1. 168 GPs responded to an email invitation and were randomly assigned to an intervention or control group, evaluating the G-eCPD module (n = 80) or the live module (n = 88). Assessment 2. Referral rates by GPs were requested from the clinical genetics centres, in the northern and southern parts of the Netherlands (Amsterdam and Maastricht), for the two years before (2010 [n = 2510] and 2011 [n = 2940]) and the year after (2012 [n = 2875]) launch of the oncogenetics CPD modules and the website. Assessment 3. Participants of the website evaluation were all recruited online. When they visited the website during the month of February 2013, a pop-up invitation came up. Of the 1350 unique visitors that month, only 38 completed the online questionnaire. Main outcomes measure showed long-term (self-reported) genetic consultation skills (i.e. increased genetics awareness and referrals to clinical genetics centres) among GPs who participated in the oncogenetic training course, and interest in and satisfaction with the supportive website. 42 GPs (52%) who previously participated in the G-eCPD evaluation study and 50 GPs (57%) who participated in the live training programme responded to the online questionnaire on long-term effects of educational outcome. Previous RCTs showed that the genetics CPD modules achieved sustained improvement of oncogenetic knowledge and consultation skills (3-months follow-up). Participants of these RCTs reported being more aware of genetic problems long term; this was reported by 29 GPs (69%) and 46 GPs (92%) participating in the G-eCPD and live module evaluation studies, respectively (Chisquare test, p<0.005). One year later, 68% of the respondents attending the live training reported that they more frequently referred patients to the clinical genetics centres, compared to 29% of those who attended the online oncogenetics training (Chisquare test, p<0.0005). However, the clinical genetics centres reported no significant change in referral numbers one year after the training. Website visitor numbers increased, as did satisfaction, reflected in a 7.7 and 8.1 (out of 10) global rating of the website (by G-eCPD and live module participants, respectively). The page most often consulted was "family tree drawing". Self-perceived genetic consultation skills increased long-term and GPs were interested in and satisfied with the supportive website. Further studies are necessary to see whether the oncogenetics CPD modules result in more efficient referral. The results presented suggest we have provided a flexible and effective framework to meet the need for effective educational programmes for non-geneticist healthcare providers, enabling improvement of genetic medical care.

Exploring the validity and reliability of a questionnaire for evaluating virtual patient design with a special emphasis on fostering clinical reasoning.

BACKGROUND: Virtual patients (VPs) are increasingly used to train clinical reasoning. So far, no validated evaluation instruments for VP design are available. AIMS: We examined the validity of an instrument for assessing the perception of VP design by learners. METHODS: Three sources of validity evidence were examined: (i) Content was examined based on theory of clinical reasoning and an international VP expert team. (ii) The response process was explored in think-aloud pilot studies with medical students and in content analyses of free text questions accompanying each item of the instrument. (iii) Internal structure was assessed by exploratory factor analysis (EFA) and inter-rater reliability by generalizability analysis. RESULTS: Content analysis was reasonably supported by the theoretical foundation and the VP expert team. The think-aloud studies and analysis of free text comments supported the validity of the instrument. In the EFA, using 2547 student evaluations of a total of 78 VPs, a three-factor model showed a reasonable fit with the data. At least 200 student responses are needed to obtain a reliable evaluation of a VP on all three factors. CONCLUSION: The instrument has the potential to provide valid information about VP design, provided that many responses per VP are available.