Effectiveness of spaced repetition for clinical problem solving amongst undergraduate medical students studying paediatrics in Pakistan.

BACKGROUND: Studies using spaced repetition for teaching and learning in undergraduate clinical rotations such as paediatrics are limited, even more so in the South Asian region. Therefore, this study aimed to identify the effectiveness of utilizing spaced repetition compared to traditional learning methods among undergraduate medical students during their paediatric rotation at a medical university in Pakistan. METHODS: Bahria University Medical and Dental College (BUMDC) conducted quasii-experimental research in Karachi. Four topics were identified from the Year 5 Pediatrics curriculum to be used in the study, using which the study content was developed along with 50 multiple choice questions (MCQs) for assessment. All BUMDC Year 5 medical students rotating in Pediatrics were included and randomly allocated to the control or intervention group. In the control group, they provided the students with traditional study methods consisting of books and lectures to learn topics. In the intervention group, we created an Anki flashcard deck of the same topics to enable learning via spaced repetition. The researchers conducted a pretest and post test assessment of the 50 MCQs in both groups at the beginning and after the four-week study interval. The data were analysed using SPSS 19. RESULTS: A total of 115 BUMDC medical students agreed to participate in the study; 70 (59.1%) were in the intervention group, and 45 (41.7%) were in the control group. The pretest mean score of the control group was 27.96 ± 3.70, and the posttest mean score was 27.22 ± 5.02, with no statistically significant difference at the 95% confidence level. The mean score of the pretest for the intervention group was 27.93 ± 4.53, and that of the posttest was 30.8 ± 4.56, with a statistically significant difference at the 95% confidence level. The intervention showed a significant effect size of 0.8. CONCLUSION: The use of spaced repetitions resulted in significantly greater scores for medical students studying paediatrics than for those using more traditional methods of learning, compromising medical books and lectures. Considering that medical students need to retain a vast amount of information, using spaced repetition through flashcards can be a more effective learning tool that is more cost-efficient and time-efficient than traditional learning methods.

Investigating the internal structure of multiple mini interviews-A perspective from Pakistan.

BACKGROUND: Healthcare professionals require many personal attributes in addition to cognitive abilities and psychomotor skills for competent practice. Multiple Mini- Interviews are being employed globally to assess personality attributes of candidates for selection in health professions education at all level of entry; these attributes are namely, communication skills, critical thinking, honesty, responsibility, health advocacy, empathy and sanctity of life. Considering the high stakes involved for students, faculty, institutions and the society, rigorous quality assurance mechanisms similar to those used for student assessment must be employed for student selection, throughout the continuum of medical education. It is a difficult undertaking as these psychological constructs are difficult to define and measure. Though considered to yield reliable and valid scores, studies providing multiple evidences of internal structure especially dimensionality of Multiple Mini-Interviews are sparse giving rise to questions if they are measuring a single or multiple constructs and even if they are measuring what they are purported to be measuring. OBJECTIVE: The main objective is to provide statistical support of the multi-dimensional nature of our Multiple Mini Interviews, hypothesized a-priori, through CFA. Another objective is to provide multiple evidences for the internal structure. Our study highlights the link between content and internal structure evidences of the constructs, thus establishing that our Multiple Mini Interviews measure what they were intended to measure. METHOD: After securing permission from the Institutional review board, an a-priori seven factor-model was hypothesized based on the attributes considered most essential for the graduating student of the institution. After operationally defining the attributes through extensive literature search, scenarios were constructed to assess them. A 5-point rating scale was used to rate each item on the station. A total 259 students participated in the multiple mini interviews over a period of three days. A training workshop had been arranged for the participating faculty. RESULTS: The reliability coefficient using Cronbach's alpha were calculated (range from 0.73 to 0.94), Standard Error of Measurement (ranged from 0.80 to1.64), and item to station-total correlation ranged from 0.43-0.50 to 0.75-0.83. Inter-station correlation was also determined. Confirmatory factor analysis endorsed the results of Exploratory factor analysis in the study revealing a seven model fit with multiple indices of Goodness-of-fit statistics such as Root mean square error of approximation (RMSEA) value 0.05, Standardized root mean square residual (SRMR) value with less than 0.08. All these indices showed that model fit is good. The Confirmatory factor analysis confirmed the multi-dimensional nature of our MMIs and also confirmed that our stations measured the attributes that they were supposed to measure. CONCLUSION: This study adds to the validity evidence of Multiple Mini-Interviews, in selection of candidates, with required personality traits for healthcare profession. It provides the evidence for the multi-dimensional structure of Multiple Mini interviews administered with multiple evidences for its internal structure and demonstrates the independence of different constructs being measured.

Assessing the validity of admission test into the internship programme at a private university in Pakistan.

OBJECTIVE: To report the development and validity assessment of a comprehensive model of assessment method comprising written test and multiple mini interviews for selection in a medical internship programme. METHODS: The psychometric validation study was conducted at the Aga Khan University, Karachi, and comprised anonymised scores of written admission test, multiple mini interviews and exit written exams for all the interns who completed their internships in 2018 and 2019. Correlation between admission and exit tests, and predictions were assessed. Data was analysed using SPSS 20. RESULTS: There were 160 interns; 80(50%) each in 2018 and 2019. Mean scores were 68.8±4.40% for written tests and 76.7±4.66% for multiple mini interviews. The mean score for exit examination was 68.1±6.84%. The Cronbach's alpha of scores on the written admission test was 0.82 and 0.88 for the two years, respectively, while for the multiple mini interviews, the corresponding values were 0.81 and 0.94. The written admission and exit tests were moderately correlated (0.44) while the correlations of multiple mini interviews scores with written admissions and exit tests were -0.28 and 0.04, respectively. CONCLUSIONS: The selection process should comprise multiple measures of assessment to ensure the selection of the best candidates.

Agreement between simulated patients and faculty: Assessment of communication skills during objective structured clinical examination.

OBJECTIVE: Ensuring competence in communication skills amongst trainees is essential in health professions education. Involving faculty members for the same is a challenge in Obstetrics and Gynecology (OBGYN) due to their clinical commitments. The present study compares scores of OBGYN faculty, non-OBGYN faculty and simulated patients (SPs) on communication skills of postgraduate trainees during formative Objective Structured Clinical Examination (OSCE). METHODS: This is a psychometric study conducted in Feburary 2017 at the Aga Khan University Medical College (AKU-MC). All thirty-two postgraduate trainees of OBGYN gave consent. Each trainee was assessed by OBGYN faculty, non-OBGYN faculty and SP on communication skills at six stations using nine-point itemized rating-scale during formative OBGYN OSCE. The scores were reviewed using descriptive statistics, reliability was calculated using Cronbach's alpha and inter-rater reliability was analyzed using Pearson correlation and intra-class correlation coefficient. RESULTS: The score reliability of each of the examiners was >0.7. The mean scores showed that OBGYN faculty were most stringent while SPs were lenient examiners, however, non-OBGYN faculty scored in between. The inter-rater reliability among any two of the OBGYN, non-OBGYN and SP examiner was >0.84 using Pearson correlation and >0.9 using intra-class correlation. CONCLUSION: The SPs and non-OBGYN clinical faculty can also be used to assess communication and counseling skills on OBGYN OSCEs after required training as examiners.

The journeys of three ASPIRE winning medical schools toward excellence in student assessment.

Introduction: ASPIRE Excellence Awards in Student Assessment are offered to medical schools with innovative and comprehensive assessment programmes adjudged by international experts, using evidence-based criteria. The journeys of three ASPIRE-winning medical schools toward "assessment excellence" are presented. These schools include Aga Khan University Medical College (AKU-MC), Pakistan, Southern Illinois University School of Medicine (SIUSOM), USA, and University of Leeds School of Medicine, UK. Methods: The unfolding journeys highlighting achievements, innovations, and essential components of each assessment programme were compared to identify differences and commonalities. Results: Cultural contextual differences included developed-versus-developing country, east-west, type of regulatory bodies, and institutional-versus-national certifying/licensing examinations, which influence curricula and assessments. In all, 12 essential commonalities were found: alignment with institutional vision; sustained assessment leadership; stakeholder engagement; communication between curriculum and assessment; assessment-for-learning and feedback; longitudinal student profiling of outcome achievement; assessment rigor and robustness; 360° feedback from-and-to assessment; continuous enrichment through rigorous quality assurance; societal sensitivity; influencing others; and a "wow factor." Conclusions: Although the journeys of the three medical schools were undertaken in different cultural contexts, similar core components highlight strong foundations in student assessment. The journeys continue as assessment programmes remain dynamic and measurement science expands. This article may be helpful to other institutions pursuing excellence in assessment.

Multiple mini interviews as a measure of non-cognitive skills for admissions into undergraduate medical education programme in Pakistan: A validity study.

Student selection for Undergraduate Medical Education Programmes (UGME) is a highly selective process globally. Health care practice requires many attributes like communication skills, professionalism, critical thinking and problem solving in addition to cognitive abilities. This study reports the development and administration of Multiple Mini Interviews (MMI), the descriptive and psychometric properties of the MMI station scores and assesses the validity of MMI stations to ascertain if the stations measured the intended attributes. Nine attributes considered most essential for a successful health care professional were selected. A 5 point rating scale was used to rate each item on the station. The scores were then converted into percentage scores. The mean scores on each MMI station ranged from 27.4% to 80.0%. The reliability of stations using Cronbach's alpha ranged from 0.64 to 0.98. MMI can be used to make reliable and valid decisions to select students with desired non cognitive attributes.

Predictive Validity of Family Medicine Resident's Performance at Induction with Their Future Performance During Residency.

OBJECTIVE: To determine the predictive validity of induction assessment scores of family medicine residency program for performance in final year of residency. STUDY DESIGN: Psychometric (predictive-validity) study. PLACE AND DURATION OF STUDY: The Aga Khan University Hospital, Karachi, from 2008 to 2014. METHODOLOGY: All family medicine residents were evaluated. Family Medicine Residency induction assessment scores were used as independent predictor variables, whereas resident assessment scores during the final year residency program were used as dependent or outcome variables. Data was analyzed using SPSS version 19. Linear regression was used to determine predictive validity of induction scores as independent variables with outcome variables at 95% confidence level. RESULTS: There were 33 residents. MBBS scores accounted for 30.1% of variance in final year Objective Structured Clinical Exam (OSCE) scores; whereas, induction written test accounted for 37.1% of variance in final year written test. Induction communication skill scores did not correlate with in-training communication skill scores or with the final year OSCE scores. Induction professionalism scores accounted for 13.7% of the variation in final year OSCE scores, but not with in-training continuous professionalism scores. CONCLUSION: Induction knowledge scores have acceptable predictive value for future knowledge and its application. Other valid and reliable assessment methods, such as multiple mini-interviews, should be explored for assessment of noncognitive domains at induction.

Standard Setting Methods for Pass/Fail Decisions on High-Stakes Objective Structured Clinical Examinations: A Validity Study.

UNLABELLED: CONSTRUCT: Authentic standard setting methods will demonstrate high convergent validity evidence of their outcomes, that is, cutoff scores and pass/fail decisions, with most other methods when compared with each other. BACKGROUND: The objective structured clinical examination (OSCE) was established for valid, reliable, and objective assessment of clinical skills in health professions education. Various standard setting methods have been proposed to identify objective, reliable, and valid cutoff scores on OSCEs. These methods may identify different cutoff scores for the same examinations. Identification of valid and reliable cutoff scores for OSCEs remains an important issue and a challenge. APPROACH: Thirty OSCE stations administered at least twice in the years 2010-2012 to 393 medical students in Years 2 and 3 at Aga Khan University are included. Psychometric properties of the scores are determined. Cutoff scores and pass/fail decisions of Wijnen, Cohen, Mean-1.5SD, Mean-1SD, Angoff, borderline group and borderline regression (BL-R) methods are compared with each other and with three variants of cluster analysis using repeated measures analysis of variance and Cohen's kappa. RESULTS: The mean psychometric indices on the 30 OSCE stations are reliability coefficient = 0.76 (SD = 0.12); standard error of measurement = 5.66 (SD = 1.38); coefficient of determination = 0.47 (SD = 0.19), and intergrade discrimination = 7.19 (SD = 1.89). BL-R and Wijnen methods show the highest convergent validity evidence among other methods on the defined criteria. Angoff and Mean-1.5SD demonstrated least convergent validity evidence. The three cluster variants showed substantial convergent validity with borderline methods. CONCLUSIONS: Although there was a high level of convergent validity of Wijnen method, it lacks the theoretical strength to be used for competency-based assessments. The BL-R method is found to show the highest convergent validity evidences for OSCEs with other standard setting methods used in the present study. We also found that cluster analysis using mean method can be used for quality assurance of borderline methods. These findings should be further confirmed by studies in other settings.

Incidental diagnosis of intradural lumbar disc herniation during discography: a case report.

We describe an incidental finding of intradural lumbar disc herniation diagnosed radiographically during discography. A patient was referred to our center for discography with symptoms of worsening, intractable low back pain radiating to both hips and the left leg which was exacerbated when standing and walking. Magnetic resonance imaging of the lumbar spine revealed multiple disc bulges and lumbar facet arthroses with ligamentum flavum hypertrophy producing moderate central canal and lateral recess stenosis. Discography was performed at three levels (L3-4, L4-5, L5-S1). During fluoroscopically guided injection into L4-5 it was noted that contrast was not contained within the disc and spread intrathecally with a myelographic appearance. Computerized tomography confirmed accurate needle placement and a spread of contrast into the intrathecal space. To the best of our knowledge, this is the first report describing a finding of intradural disc herniation while performing discography. Physicians should be aware of this potential finding while performing this technique.