Professionalism and clinical short answer question marking with machine learning.

Machine learning may assist in medical student evaluation. This study involved scoring short answer questions administered at three centres. Bidirectional encoder representations from transformers were particularly effective for professionalism question scoring (accuracy ranging from 41.6% to 92.5%). In the scoring of 3-mark professionalism questions, as compared with clinical questions, machine learning had a lower classification accuracy (P < 0.05). The role of machine learning in medical professionalism evaluation warrants further investigation.

Medical student knowledge and critical appraisal of machine learning: a multicentre international cross-sectional study.

To utilise effectively tools that employ machine learning (ML) in clinical practice medical students and doctors will require a degree of understanding of ML models. To evaluate current levels of understanding, a formative examination and survey was conducted across three centres in Australia, New Zealand and the United States. Of the 245 individuals who participated in the study (response rate = 45.4%), the majority had difficulty with identifying weaknesses in model performance analysis. Further studies examining educational interventions addressing such ML topics are warranted.

How was the intern year?: self and clinical assessment of four cohorts, from two medical curricula.

BACKGROUND: Problem-based curricula have provoked controversy amongst educators and students regarding outcome in medical graduates, supporting the need for longitudinal evaluation of curriculum change. As part of a longitudinal evaluation program at the University of Adelaide, a mixed method approach was used to compare the graduate outcomes of two curriculum cohorts: traditional lecture-based 'old' and problem-based 'new' learning. METHODS: Graduates were asked to self-assess preparedness for hospital practice and consent to a comparative analysis of their work-place based assessments from their intern year. Comparative data were extracted from 692 work-place based assessments for 124 doctors who graduated from the University of Adelaide Medical School between 2003 and 2006. RESULTS: Self-assessment: Overall, graduates of the lecture-based curriculum rated the medical program significantly higher than graduates of the problem-based curriculum. However, there was no significant difference between the two curriculum cohorts with respect to their preparedness in 13 clinical skills. There were however, two areas where the cohorts rated their preparedness in the 13 broad practitioner competencies as significantly different: problem-based graduates rated themselves as better prepared in their 'awareness of legal and ethical issues' and the lecture-based graduates rated themselves better prepared in their 'understanding of disease processes'.Work-place based assessment: There were no significant differences between the two curriculum cohorts for 'Appropriate Level of Competence' and 'Overall Appraisal'. Of the 14 work-place based assessment skills assessed for competence, no significant difference was found between the cohorts. CONCLUSIONS: The differences in the perceived preparedness for hospital practice of two curriculum cohorts do not reflect the work-place based assessments of their competence as interns. No significant difference was found between the two cohorts in relation to their knowledge and clinical skills. However results suggest a trend in 'communication with peers and colleagues in other disciplines' (χ2 (3, N = 596) =13.10, p = 0.056) that requires further exploration. In addition we have learned that student confidence in a new curriculum may impact on their self-perception of preparedness, while not affecting their actual competence.

Identifying the quality of life effects of urinary incontinence with depression in an Australian population.

BACKGROUND: To explore the additive effect of urinary incontinence, in people with comorbid depression, on health related quality of life. METHODS: Males and females, 15 to 95 years (n = 3010, response rate 70.2%) were interviewed face to face in the 1998 Autumn South Australian Health Omnibus Survey. RESULTS: Self-reported urinary incontinence was found in 20.3% (n=610), and depression as defined by the PRIME-MD in 15.2% (n=459) of the survey population. Urinary incontinence with comorbid depression was found in 4.3% of the overall population. Univariate analysis showed that respondents with urinary incontinence and comorbid depression were more likely to be aged between 15 and 34 years and never married when compared to those with incontinence only. Multivariate analysis demonstrated that in people with incontinence, the risk of having comorbid depression was increased by an overall health status of Fair or Poor, or the perception that their incontinence was moderately or very serious. Respondents reporting that they experienced incontinence with comorbid depression scored significantly lower than those experiencing incontinence without depression on all dimensions of the SF-36.The interaction of the presence of incontinence and the presence of depression was significantly associated with the dimensions of physical functioning. CONCLUSIONS: Depression and incontinence both reduce QOL. When they occur together there appears to be an additive effect which affects both physical and mental health, perhaps by increasing a person's negative perceptions of their illness. Clinicians should identify and manage comorbid depression when treating patients who have incontinence to improve their overall QOL.

Standard setting of script concordance tests using an adapted Nedelsky approach.

BACKGROUND: Standard setting in assessment seeks to apply meaning of achievement to an assessment score. Appropriate standard setting for script concordance tests (SCTs) remains a challenge, with existing methods representing norm-referenced approaches. AIMS: To develop a criterion-referenced standard setting approach for sct using an adapted nedelsky approach, to pilot feasibility, and to compare failure rates with two other methods. METHODS: Second- and third-year medical students were administered a 45-question SCT and results collated. Standard setting was applied using three approaches: (1) norm-referenced (student cohorts), (2) expert-referenced (student cohort compared to expert mean), and (3) adapted Nedelsky approach using answer key normalization. Feasibility and failure rates were measured. RESULTS: All standard setting approaches were feasible, with 60 additional minutes required for the Nedelsky standard setting exercise. Failure rates between the three approaches were similar (Year 2: 8.0-9.8% and Year 3: 2.1-7.6%), with the adapted Nedelsky approach representing an intermediate option (Year 2: 8.0% and Year 3: 3.5%). CONCLUSION: Standard setting SCT using the criterion-referenced method of an adapted Nedelsky approach was found to be both logically justifiable and logistically simple, and produced failure rates comparable to other currently utilized and less objective approaches.

Script concordance testing: from theory to practice: AMEE guide no. 75.

The script concordance test (SCT) is used in health professions education to assess a specific facet of clinical reasoning competence: the ability to interpret medical information under conditions of uncertainty. Grounded in established theoretical models of knowledge organization and clinical reasoning, the SCT has three key design features: (1) respondents are faced with ill-defined clinical situations and must choose between several realistic options; (2) the response format reflects the way information is processed in challenging problem-solving situations; and (3) scoring takes into account the variability of responses of experts to clinical situations. SCT scores are meant to reflect how closely respondents' ability to interpret clinical data compares with that of experienced clinicians in a given knowledge domain. A substantial body of research supports the SCT's construct validity, reliability, and feasibility across a variety of health science disciplines, and across the spectrum of health professions education from pre-clinical training to continuing professional development. In practice, its performance as an assessment tool depends on careful item development and diligent panel selection. This guide, intended as a primer for the uninitiated in SCT, will cover the basic tenets, theoretical underpinnings, and construction principles governing script concordance testing.

Medical student self-reported confidence in obstetrics and gynaecology: development of a core clinical competencies document.

BACKGROUND: Clinical competencies in obstetrics and gynaecology have not been clearly defined for Australian medical students, the growing numbers of which may impact clinical teaching. Our aim was to administer and validate a competencies list, for self-evaluation by medical students of their confidence to manage common clinical tasks in obstetrics and gynaecology; to evaluate students' views on course changes that may result from increasing class sizes. METHODS: A draft list of competencies was peer-reviewed, and discussed at two student focus groups. The resultant list was administered as part of an 81 item online survey. RESULTS: Sixty-eight percent (N = 172) of those eligible completed the survey. Most respondents (75.8%) agreed or strongly agreed that they felt confident and well equipped to recognise and manage most common and important obstetric and gynaecological conditions. Confidence was greater for women, and for those who received a higher assessment grade. Free-text data highlight reasons for lack of clinical experience that may impact perceived confidence. CONCLUSIONS: The document listing competencies for medical students and educators is useful for discussions around a national curriculum in obstetrics and gynaecology in medical schools, including the best methods of delivery, particularly in the context of increasing student numbers.

Summative assessment of 5th year medical students' clinical reasoning by Script Concordance Test: requirements and challenges.

BACKGROUND: The Script Concordance Test (SCT) has not been reported in summative assessment of students across the multiple domains of a medical curriculum. We report the steps used to build a test for summative assessment in a medical curriculum. METHODS: A 51 case, 158-question, multidisciplinary paper was constructed to assess clinical reasoning in 5th-year. 10-16 experts in each of 7 discipline-based reference panels answered questions on-line. A multidisciplinary group considered reference panel data and data from a volunteer group of 6th Years, who sat the same test, to determine the passing score for the 5th Years. RESULTS: The mean (SD) scores were 63.6 (7.6) and 68.6 (4.8) for the 6th Year (n = 23, alpha = 0.78) and and 5th Year (n = 132, alpha =0.62) groups (p < 0.05), respectively. The passing score was set at 4 SD from the expert mean. Four students failed. CONCLUSIONS: The SCT may be a useful method to assess clinical reasoning in medical students in multidisciplinary summative assessments. Substantial investment in training of faculty and students and in the development of questions is required.

Urodynamic diagnoses and quality of life in women presenting for evaluation of urinary incontinence.

BACKGROUND: Several population-based and clinical studies report that stress incontinence has less impact on quality of life (QoL) than urge incontinence and overactive bladder. AIMS: This research aimed to determine if this relationship held true for urodynamic diagnoses. METHODS: Quality of life was evaluated by the King's Health Questionnaire prior to urodynamic testing in 326 women presenting with lower urinary tract symptoms and who completed a 48-h frequency volume chart. Urodynamic results were categorised as 'normal', 'sensory abnormalities only', 'idiopathic detrusor overactivity (IDO) only', 'urodynamic stress incontinence (USI) only' or 'mixed (IDO and USI)'. QoL data were compared using these diagnostic categories. RESULTS: Women in mixed, USI and IDO categories had significantly worse QoL scores in the domain Severity Measures than women in sensory or normal categories (P < 0.0001). Incontinence Impact was significantly worse in mixed and IDO categories compared with normal (P = 0.006) but not compared with women with USI. Sleep/Energy scores were significantly worse for women in mixed and IDO categories compared with women with USI (P = 0.003). Significant differences between urodynamic categories were also observed in the domains Role Limitations, Social Limitations and General Health. CONCLUSIONS: Mixed incontinence had the greatest adverse effect on QoL; however, any abnormal urodynamic diagnosis was associated with a significantly adverse effect. Although a normal urodynamic result was associated with less impact on QoL than an abnormal result, there was still an effect present. The optimal management (eg conservative vs surgical management) of women with a normal urodynamic result is yet to be established.

The modified essay question: its exit from the exit examination?

BACKGROUND: Exit examinations in medicine are 'high stakes' examinations and as such must satisfy a number of criteria including psychometric robustness, fairness and reliability in the face of legal or other challenges. AIMS: We have undertaken a critical review of the exit examination from the University of Adelaide focussing on the written components. This examination consisted of an objective structure clinical examination (OSCE), a multiple choice question (MCQ) paper and a modified essay question (MEQ) paper. METHODS: The two written papers were assessed for item writing flaws and taxonomic level using modified Bloom's criteria. Curriculum experts independently assessed adequacy of the examination for validity and fidelity. RESULTS: The overall examination had good fidelity and validity. The results of the MEQ and MCQ were strongly and positively correlated and there was a weak negative correlation between these papers and the OSCE. The MEQ had a higher proportion of questions focussed on recall of knowledge and the questions were more structurally flawed compared with the MCQs. The MEQ re-marking process resulted in lower scores than were awarded by the original, discipline-based expert markers. The MEQ paper failed to achieve its primary purpose of assessing higher cognitive skills. CONCLUSION: The University of Adelaide's MBBS programme has since dropped the MEQ paper from its exit examination and is evaluating in its place the Script Concordance test.

Urodynamics or history? Clinical decision-making in women presenting with urinary incontinence.

BACKGROUND: Preoperative urodynamic testing is recommended by experts, but evidence for clinical benefit is limited. AIMS: To determine whether urodynamic stress incontinence could be confirmed in women presenting with a history of stress incontinence. METHODS: History and urodynamic diagnoses of 444 women presenting to a general unit between June 2003 and July 2007 are described. RESULTS: Urodynamic Stress Incontinence (USI) or mixed USI and detrusor overactivity were observed in 61% reporting any stress incontinence. 15% who reported no stress incontinence had USI. Compared with a normal urodynamic diagnosis, incontinence occurred more frequently in all other diagnostic categories (P < 0.0001) and was worst in the USI and mixed groups (P < 0.0004). CONCLUSIONS: Urodynamic Stress Incontinence is a more severe entity than stress incontinence diagnosed by history. It is yet to be established if this information benefits patients.

Electronic voting to encourage interactive lectures: a randomised trial.

BACKGROUND: Electronic Voting Systems have been used for education in a variety of disciplines. Outcomes from these studies have been mixed. Because results from these studies have been mixed, we examined whether an EVS system could enhance a lecture's effect on educational outcomes. METHODS: A cohort of 127 Year 5 medical students at the University of Adelaide was stratified by gender, residency status and academic record then randomised into 2 groups of 64 and 63 students. Each group received consecutive 40-minute lectures on two clinical topics. One group received the EVS for both topics. The other group received traditional teaching only. Evaluation was undertaken with two, 15-question multiple-choice questionnaires (MCQ) assessing knowledge and problem solving and undertaken as a written paper immediately before and after the lectures and repeated online 8-12 weeks later. Standardised institutional student questionnaires were completed for each lecture and independent observers assessed student behaviour during the lectures. Lecturer's opinions were assessed by a questionnaire developed for this study. RESULTS: Two-thirds of students randomised to EVS and 59% of students randomised to traditional lectures attended. One-half of the students in the EVS group and 41% in the traditional group completed all questionnaires. There was no difference in MCQ scores between EVS and traditional lectures (p = 0.785). The cervical cancer lectures showed higher student ranking in favour of EVS in all parameters. The breast cancer lectures showed higher ranking in favour of traditional lectures in 5 of 7 parameters (p < 0.001). The observed higher-order lecturer-students interactions were increased in the EVS lecture for one lecturer and reduced for the other. Both lecturers felt that the EVS lectures were difficult to prepare, that they were able to keep to time in the traditional lectures, that the educational value of both lecture styles was similar, and that they were neutral-to-slightly favourably disposed to continue with the EVS technology. The 2 lecturers disagreed regarding the ease of preparation of the traditional lecture, their ability to keep to time in the EVS lecture, and personal satisfaction with the EVS lecture. The lecturers felt that EVS encouraged student participation and helped identify where students were having difficulty. CONCLUSION: In this setting, EVS technology used in large group lectures did not offer significant advantages over the more traditional lecture format.

Controlled surface plasmon enhanced fluorescence from 1D gold gratings via azimuth rotations.

We demonstrate a method to maximise the fluorescence enhancement from a dye using gold coated diffraction gratings. Rotations about the azimuth provides a convenient approach to maximise the coupling between the grating and excitation source while achieving enhancements comparable to traditional optical configurations where the grating and in plane light vectors are parallel. This approach yields a 30 fold enhancement in the fluorescence signal over metal free substrates, while opening up the range of possible orientations and configurations suitable for fluorescence enhancement applications.

Utilization of preoperative urodynamic studies by Canadian gynaecologists.

OBJECTIVES: (1) To determine the proportion of Canadian gynaecologists who have urodynamic studies (UDS) readily available; (2) to assess the preoperative UDS utilization patterns of Canadian gynaecologists in various clinical scenarios of stress urinary incontinence (SUI). METHODS: A mail survey was sent to all 1266 obstetrician/gynaecologist members of the Society of Obstetricians and Gynaecologists of Canada (SOGC), as part of an international survey with the U.K., U.S.A., New Zealand, and Australia. Survey questions included geographic availability of UDS and specific types of urodynamics testing available. Utilization of urodynamics prior to anti-incontinence surgery was assessed in four clinical scenarios: (A) a history of uncomplicated primary stress urinary incontinence, (B) a history of mixed stress and urgency incontinence, (C) stress incontinence with a history of straining to void, and (D) recurrent stress incontinence. RESULTS: Of the 1266 surveys mailed, 230 (18%) were returned. Forty-four of the respondents did not manage incontinent women. Of the 186 respondents who did, 27% reported a special interest or additional postgraduate training in urogynaecology, and 73% were general gynaecologists. UDS were available in the same centre or city to 79% of respondents. For uncomplicated primary stress incontinence, 47% would always obtain preoperative UDS, 42% would not, and 11% would only if UDS were readily available. For the total group of respondents, in scenario A 47% would always obtain preoperative UDS, 42% would not, and 11% would only if UDS were readily available. Among general gynaecologists, in scenario A, if UDS were available in their city or centre of practice, 54% would always obtain preoperative UDS, as compared to 5% by general gynaecologists who did not have UDS available in their city or centre of practice. CONCLUSION: The data collected in this survey are compromised because of the low response rate. However, most (79%) of the respondents who manage women with urinary incontinence had access to urodynamics in the same centre or city. The utilization of urodynamics prior to surgical treatment of uncomplicated primary pure stress incontinence varies depending on the availability of such testing. An uncomplicated history of primary SUI and decreased availability of testing were two factors seen to have an effect of decreasing utilization of preoperative UDS. Hospital management did not always comply with SOGC guidelines for preoperative testing for SUI. Updating and dissemination of these guidelines is necessary to ensure "best" practice and highest quality of care for all women with SUI in Canada.

Improved quality of life in women treated for urinary incontinence by an authorised continence nurse practitioner.

The goal of this study was to evaluate the effect of conservative therapy for women with urinary incontinence by an authorised Continence Nurse Practitioner (CNP) on quality of life (QoL) measured by a validated questionnaire and to correlate simple analogue scores with the QoL domains. Pre- and post-treatment Kings Health Questionnaires (KHQ) and post-treatment analogue scores for "wetness" and "satisfaction" with treatment were sent to women managed by a CNP. We received 85 completed questionnaires (73.9% response rate). Two KHQ domains (Incontinence Impact and Emotions) showed a significant improvement after treatment. The analogue scores correlated moderately with the majority of KHQ domains. There is a positive effect of CNP treatment on QoL in women with urinary incontinence. Simple analogue scores are not sufficiently strongly correlated with KHQ domains to be a satisfactory substitute for validated QoL questionnaires.

Incisional hernia after a tension-free vaginal tape procedure.

A case is presented of an incisional hernia of the inguinal canal presenting 9 months after a tension-free vaginal tape (TVT) procedure and anterior vaginal repair. The TVT and repair procedure was complicated by prolonged postoperative urinary retention requiring midline incision of the tape for resumption of normal voiding. The patient had a hysterectomy several years earlier via a Pfannenstiel incision. No other risk factors for hernia were identified. There are no previous reports of TVT-related incisional hernia. We conclude that incisional hernia is a rare complication of the TVT procedure and that the characteristics of the TVT tape may contribute to late occurrence of herniation.

Utilization of preoperative urodynamic investigations by gynecologists who frequently operate for female urinary incontinence.

Our objective was to determine the availability and utilization of urodynamic investigations by gynecologists in the preoperative evaluation of women with urinary incontinence. Gynecologists in the UK, USA, Australia, New Zealand and Canada were asked what urodynamic investigations were required for four clinical scenarios. Analysis was restricted to frequent operators. Urodynamic investigations were available to 70% of frequent operators. For uncomplicated stress incontinence, cystometry was utilized by 72% of subspecialists and 44% of generalists ( P<0.001) who had access to urodynamic investigations. For stress incontinence and straining to void, uroflowmetry was utilized by 73% of subspecialists and 46% of generalists ( P<0.001) who had access to urodynamic investigations. We concluded that many gynecologists who frequently operate for female urinary incontinence do not have access to urodynamic investigations or do not utilize urodynamic investigations, or utilize investigations in a way that may be inadequate for the clinical problem. There are differences in utilization between subspecialists and generalists that are not explained by access. These observations could be explained by poor understanding or a lack of belief in the value of urodynamic investigations.