Capability ... what's in a word? Rural Doctors Network of New South Wales Australia is shifting to focus on the capability of rural health professionals.

Rural health services, and the workforces that provide those services, are under unprecedented pressure due to insufficient health workforce numbers and distribution of health workforce weighted to urban areas. This creates health service access issues in rural areas, compounding existing health inequalities between rural and urban people. Many approaches to date have aimed to rectify these issues, with moderate success. In this article we present a call to action to pursue a complementary approach: supporting the capability of the rural health workforce. We hypothesise that further exploring what it means to be a 'capable' rural health professional and what processes or conditions support or erode capability may additionally bolster efforts toward strong rural and remote health systems. The Capability Approach is a theory proposed by Amartya Sen, who was awarded the Nobel Memorial Prize in Economic Sciences in 1998 for this work. Although the Capability Approach inspired, for instance, the UN's Human Development Index, it has not been deeply explored in the context of rural health workforce. While still untested, a focus on capability may assist us in taking a broader view, which encompasses functioning and the freedom to pursue different functioning combinations. The feasible freedom and opportunities are paramount to the concept of capability. We posit that competence is static and the responsibility of the practitioner (and their education), but that capability is fluid and multi-dimensional and the responsibility of the practitioner, community and system. Therefore, we hypothesise that a focus on a Capability Approach, which modulates the relation between the contextual factors and outcomes, may provide us with greater understanding and avenues for action when we aim to improve outcomes such as rural health service sustainability. Developing a list of appropriate capabilities and setting strategies to support capability and its more nuanced domains may present unique opportunities for influence, and these may have positive effects on the rural health workforce. Of course it will need to be determined if improving rural primary health professionals' capability has positive impacts upon quality and access to care, and whether supporting capability is sustainable and worthy of investment.

Hypothesis: improving literacy about health workforce will improve rural health workforce recruitment, retention and capability.

BACKGROUND: One of the key barriers to health in rural areas is health workforce. Poor understanding and communication about health workforce across all stakeholder groups (including the broad community) is very common and can negatively affect the health workforce, recruitment, experiences and outcomes. HYPOTHESIS: In this paper, we propose the concept of literacy about health workforce. We propose this as a specific, actionable extension of the existing and well accepted health literacy concept. We hypothesise that improving literacy about health workforce will improve, in particular, rural health workforce recruitment, retention and capability. IMPLICATIONS OF THE HYPOTHESIS: We propose that literacy about health workforce is important for all members of the health and broader system (e.g. local GP, mayor, workforce agency, health manager, Aboriginal health worker, carers, community health facilitators, patients, schools, local businesses, cultural and recreation groups) because we hypothesise their literacy about health workforce affects their capacity to make informed decisions and take action to manage their health workforce needs in direct synchrony with the community's health needs. We hypothesise that improving literacy about health workforce will improve the effectiveness and efficiency of attracting, recruiting, training, and retaining a high quality, capable, health workforce, and further, will support the development and acceptance of innovative solutions to health workforce crises such as new models of care. This hypothesis is action orientated, is testable and includes the consideration of methods to engage and improve literacy of those within and external to the health workforce.

The Australian Medical Schools Assessment Collaboration: benchmarking the preclinical performance of medical students.

OBJECTIVES: To report the level of participation of medical schools in the Australian Medical Schools Assessment Collaboration (AMSAC); and to measure differences in student performance related to medical school characteristics and implementation methods. DESIGN: Retrospective analysis of data using the Rasch statistical model to correct for missing data and variability in item difficulty. Linear model analysis of variance was used to assess differences in student performance. SETTING AND PARTICIPANTS: 6401 preclinical students from 13 medical schools that participated in AMSAC from 2011 to 2013. MAIN OUTCOME MEASURES: Rasch estimates of preclinical basic and clinical science knowledge. RESULTS: Representation of Australian medical schools and students in AMSAC more than doubled between 2009 and 2013. In 2013 it included 12 of 19 medical schools and 68% of medical students. Graduate-entry students scored higher than students entering straight from school. Students at large schools scored higher than students at small schools. Although the significance level was high (P < 0.001), the main effect sizes were small (4.5% and 2.3%, respectively). The time allowed per multiple choice question was not significantly associated with student performance. The effect on performance of multiple assessments compared with the test items as part of a single end-of-year examination was negligible. The variables investigated explain only 12% of the total variation in student performance. CONCLUSIONS: An increasing number of medical schools are participating in AMSAC to monitor student performance in preclinical sciences against an external benchmark. Medical school characteristics account for only a small part of overall variation in student performance. Student performance was not affected by the different methods of administering test items.

Internet-based multiple mini-interviews for candidate selection for graduate entry programmes.

CONTEXT: Multiple mini-interviews (MMIs) have been used by The University of Sydney graduate medical and dental programmes since 2006. In 2011, interviews with international candidates were conducted using Skype (iMMI), whereas interviews with local candidates were conducted in person. We determined whether the MMI scores derived from both methods were comparable. We describe the feasibility, acceptability and cost-effectiveness of the iMMI. METHODS: We compared 2011 international student internet-based iMMI results with data from 2009 international student MMIs and 2011 local student MMIs. Analyses of variance (anovas) were used to investigate equivalence of the two formats by exploring whether the medium of interviewing resulted in significantly different mean scores and variance for the in-person MMI and the iMMI. Acceptability of the process was informed by feedback surveys from interviewers and candidates, and cost savings were estimated. RESULTS: No significant difference was found between the 2011 iMMI scores for international candidates and MMI scores in 2009 (p > 0.05). There was no significant difference between the MMI scores for local and international candidates in 2011 (p > 0.05); the MMI scores for international candidates had greater variation (p < 0.01). Using generalisability theory, the reliability of the nine-question iMMI was 0.76 and for the MMI was 0.70. Delivery of the iMMI occurred smoothly and candidates and interviewers gave positive feedback on its format and delivery. Cost savings have been estimated to be over AU$50 000, representing an 84% saving. CONCLUSIONS: We believe this is the first study reporting an internet-based MMI for a high stakes interview. We have shown that interviewers were able to make valid and reliable decisions about candidates through the iMMI in a process that was acceptable to participants, producing comparable results to the in-person MMI with a saving of resources. The slightly wider variance in iMMI scores warrants further investigation.

Should candidate scores be adjusted for interviewer stringency or leniency in the multiple mini-interview?

CONTEXT: There are significant levels of variation in candidate multiple mini-interview (MMI) scores caused by interviewer-related factors. Multi-facet Rasch modelling (MFRM) has the capability to both identify these sources of error and partially adjust for them within a measurement model that may be fairer to the candidate. METHODS: Using facets software, a variance components analysis estimated sources of measurement error that were comparable with those produced by generalisability theory. Fair average scores for the effects of the stringency/leniency of interviewers and question difficulty were calculated and adjusted rankings of candidates were modelled. RESULTS: The decisions of 207 interviewers had an acceptable fit to the MFRM model. For one candidate assessed by one interviewer on one MMI question, 19.1% of the variance reflected candidate ability, 8.9% reflected interviewer stringency/leniency, 5.1% reflected interviewer question-specific stringency/leniency and 2.6% reflected question difficulty. If adjustments were made to candidates' raw scores for interviewer stringency/leniency and question difficulty, 11.5% of candidates would see a significant change in their ranking for selection into the programme. Greater interviewer leniency was associated with the number of candidates interviewed. CONCLUSIONS: Interviewers differ in their degree of stringency/leniency and this appears to be a stable characteristic. The MFRM provides a recommendable way of giving a candidate score which adjusts for the stringency/leniency of whichever interviewers the candidate sees and the difficulty of the questions the candidate is asked.

Validating a multiple mini-interview question bank assessing entry-level reasoning skills in candidates for graduate-entry medicine and dentistry programmes.

CONTEXT: The multiple mini-interview (MMI) was initially designed to test non-cognitive characteristics related to professionalism in entry-level students. However, it may be testing cognitive reasoning skills. Candidates to medical and dental schools come from diverse backgrounds and it is important for the validity and fairness of the MMI that these background factors do not impact on their scores. METHODS: A suite of advanced psychometric techniques drawn from item response theory (IRT) was used to validate an MMI question bank in order to establish the conceptual equivalence of the questions. Bias against candidate subgroups of equal ability was investigated using differential item functioning (DIF) analysis. RESULTS: All 39 questions had a good fit to the IRT model. Of the 195 checklist items, none were found to have significant DIF after visual inspection of expected score curves, consideration of the number of applicants per category, and evaluation of the magnitude of the DIF parameter estimates. CONCLUSIONS: The question bank contains items that have been studied carefully in terms of model fit and DIF. Questions appear to measure a cognitive unidimensional construct, 'entry-level reasoning skills in professionalism', as suggested by goodness-of-fit statistics. The lack of items exhibiting DIF is encouraging in a contemporary high-stakes admission setting where candidates of diverse personal, cultural and academic backgrounds are assessed by common means. This IRT approach has potential to provide assessment designers with a quality control procedure that extends to the level of checklist items.

Experiences of the multiple mini-interview: a qualitative analysis.

CONTEXT: Multiple mini-interviews (MMIs) are increasingly used in high-stakes medical school selection. Yet there is little published research about participants' experiences and understandings of the process. We report the findings from an international qualitative study on candidate and interviewer experiences of the MMI for entry into a graduate-entry medical school. METHODS: Qualitative data from six interviewer focus groups and 442 candidate and 75 interviewer surveys were analysed using framework analysis. Multiple researchers (n = 3) analysed a proportion of the data and developed a thematic framework capturing content-related (i.e. what was said) themes that emerged from the data. This thematic framework was then used to code the complete dataset. RESULTS: Several key themes were identified, including participants' perspectives on having: (i) a one-to-one interview; (ii) multiple assessment opportunities; (iii) a standardised, scenario-based interview; (iv) a mini-interview, and on (v) the attributes currently measured by the MMI, and (vi) other attributes that should be assessed. CONCLUSIONS: We gained a deeper understanding of participants' experiences of a high-stakes, decision-making process for selection into a graduate-entry medical school. We discuss our findings in the light of the existing literature and make recommendations to address the issue of differing participant expectations and understandings of the MMI, and to improve the credibility and acceptability of the process.

Factors affecting the utility of the multiple mini-interview in selecting candidates for graduate-entry medical school.

CONTEXT: We wished to determine which factors are important in ensuring interviewers are able to make reliable and valid decisions about the non-cognitive characteristics of candidates when selecting candidates for entry into a graduate-entry medical programme using the multiple mini-interview (MMI). METHODS: Data came from a high-stakes admissions procedure. Content validity was assured by using a framework based on international criteria for sampling the behaviours expected of entry-level students. A variance components analysis was used to estimate the reliability and sources of measurement error. Further modelling was used to estimate the optimal configurations for future MMI iterations. RESULTS: This study refers to 485 candidates, 155 interviewers and 21 questions taken from a pre- prepared bank. For a single MMI question and 1 assessor, 22% of the variance between scores reflected candidate-to-candidate variation. The reliability for an 8-question MMI was 0.7; to achieve 0.8 would require 14 questions. Typical inter-question correlations ranged from 0.08 to 0.38. A disattenuated correlation with the Graduate Australian Medical School Admissions Test (GAMSAT) subsection 'Reasoning in Humanities and Social Sciences' was 0.26. CONCLUSIONS: The MMI is a moderately reliable method of assessment. The largest source of error relates to aspects of interviewer subjectivity, suggesting interviewer training would be beneficial. Candidate performance on 1 question does not correlate strongly with performance on another question, demonstrating the importance of context specificity. The MMI needs to be sufficiently long for precise comparison for ranking purposes. We supported the validity of the MMI by showing a small positive correlation with GAMSAT section scores.

When should students learn about ethics, professionalism and patient safety?

BACKGROUND: Medical education curricula are required to teach ethics, professionalism and patient safety, but there is no clear evidence as to when these topics should be introduced. The Personal and Professional Development (PPD) theme incorporates these topics, and is integrated throughout our postgraduate medical programme, but we were particularly interested in knowing when and how to introduce them to year-1 students. We describe an intensive PPD programme in the context of broader issues associated with the appropriate timing of PPD curricula in a medical programme. METHODS: The PPD Intensive was held over 2 days (267 students in 2008 and 299 in 2009). Attendance was mandatory. Teaching included a mixture of didactic, small group and interactive sessions. Students completed a formative case-based assessment and evaluation questionnaire. RESULTS: In response to questions about a patient narrative, many students focused on the health provider perspective, even when asked specifically to put themselves in the patient's position when answering questions about the case. Students showed a superficial understanding of patient safety factors, even those who were previous health professionals. The Intensive was evaluated highly, but from the perspective of faculty staff there were issues relating to timing and lack of clinical context to give real meaning and purpose to the basic concepts underpinning the learning areas. DISCUSSION: The Intensive was designed to demonstrate to students the importance of PPD. A counter position is that students lack the clinical context required for effective learning of PPD topic areas. We discuss these conflicting beliefs and the changes made to the Intensive programme in 2010. As a result of the lessons, we changed the 2010 Intensive programme to strongly emphasise professionalism; patient safety was moved to the later years.