The reliability of the College of Intensive Care Medicine of Australia and New Zealand "Hot Case" examination.

BACKGROUND: High stakes examinations used to credential trainees for independent specialist practice should be evaluated periodically to ensure defensible decisions are made. This study aims to quantify the College of Intensive Care Medicine of Australia and New Zealand (CICM) Hot Case reliability coefficient and evaluate contributions to variance from candidates, cases and examiners. METHODS: This retrospective, de-identified analysis of CICM examination data used descriptive statistics and generalisability theory to evaluate the reliability of the Hot Case examination component. Decision studies were used to project generalisability coefficients for alternate examination designs. RESULTS: Examination results from 2019 to 2022 included 592 Hot Cases, totalling 1184 individual examiner scores. The mean examiner Hot Case score was 5.17 (standard deviation 1.65). The correlation between candidates' two Hot Case scores was low (0.30). The overall reliability coefficient for the Hot Case component consisting of two cases observed by two separate pairs of examiners was 0.42. Sources of variance included candidate proficiency (25%), case difficulty and case specificity (63.4%), examiner stringency (3.5%) and other error (8.2%). To achieve a reliability coefficient of > 0.8 a candidate would need to perform 11 Hot Cases observed by two examiners. CONCLUSION: The reliability coefficient for the Hot Case component of the CICM second part examination is below the generally accepted value for a high stakes examination. Modifications to case selection and introduction of a clear scoring rubric to mitigate the effects of variation in case difficulty may be helpful. Increasing the number of cases and overall assessment time appears to be the best way to increase the overall reliability. Further research is required to assess the combined reliability of the Hot Case and viva components.

Using transfer learning and dimensionality reduction techniques to improve generalisability of machine-learning predictions of mosquito ages from mid-infrared spectra.

BACKGROUND: Old mosquitoes are more likely to transmit malaria than young ones. Therefore, accurate prediction of mosquito population age can drastically improve the evaluation of mosquito-targeted interventions. However, standard methods for age-grading mosquitoes are laborious and costly. We have shown that Mid-infrared spectroscopy (MIRS) can be used to detect age-specific patterns in mosquito cuticles and thus can be used to train age-grading machine learning models. However, these models tend to transfer poorly across populations. Here, we investigate whether applying dimensionality reduction and transfer learning to MIRS data can improve the transferability of MIRS-based predictions for mosquito ages. METHODS: We reared adults of the malaria vector Anopheles arabiensis in two insectaries. The heads and thoraces of female mosquitoes were scanned using an attenuated total reflection-Fourier transform infrared spectrometer, which were grouped into two different age classes. The dimensionality of the spectra data was reduced using unsupervised principal component analysis or t-distributed stochastic neighbour embedding, and then used to train deep learning and standard machine learning classifiers. Transfer learning was also evaluated to improve transferability of the models when predicting mosquito age classes from new populations. RESULTS: Model accuracies for predicting the age of mosquitoes from the same population as the training samples reached 99% for deep learning and 92% for standard machine learning. However, these models did not generalise to a different population, achieving only 46% and 48% accuracy for deep learning and standard machine learning, respectively. Dimensionality reduction did not improve model generalizability but reduced computational time. Transfer learning by updating pre-trained models with 2% of mosquitoes from the alternate population improved performance to ~ 98% accuracy for predicting mosquito age classes in the alternative population. CONCLUSION: Combining dimensionality reduction and transfer learning can reduce computational costs and improve the transferability of both deep learning and standard machine learning models for predicting the age of mosquitoes. Future studies should investigate the optimal quantities and diversity of training data necessary for transfer learning and the implications for broader generalisability to unseen datasets.

Multi-vendor evaluation of artificial intelligence as an independent reader for double reading in breast cancer screening on 275,900 mammograms.

BACKGROUND: Double reading (DR) in screening mammography increases cancer detection and lowers recall rates, but has sustainability challenges due to workforce shortages. Artificial intelligence (AI) as an independent reader (IR) in DR may provide a cost-effective solution with the potential to improve screening performance. Evidence for AI to generalise across different patient populations, screening programmes and equipment vendors, however, is still lacking. METHODS: This retrospective study simulated DR with AI as an IR, using data representative of real-world deployments (275,900 cases, 177,882 participants) from four mammography equipment vendors, seven screening sites, and two countries. Non-inferiority and superiority were assessed for relevant screening metrics. RESULTS: DR with AI, compared with human DR, showed at least non-inferior recall rate, cancer detection rate, sensitivity, specificity and positive predictive value (PPV) for each mammography vendor and site, and superior recall rate, specificity, and PPV for some. The simulation indicates that using AI would have increased arbitration rate (3.3% to 12.3%), but could have reduced human workload by 30.0% to 44.8%. CONCLUSIONS: AI has potential as an IR in the DR workflow across different screening programmes, mammography equipment and geographies, substantially reducing human reader workload while maintaining or improving standard of care. TRIAL REGISTRATION: ISRCTN18056078 (20/03/2019; retrospectively registered).

Evaluating depression anxiety and stress assessment before and during the COVID-19 pandemic using generalisability theory.

The current study investigated the assessment of depression, anxiety, and stress during normal and COVID-19 pandemic conditions. Generalisability theory (G-theory) was applied to examine stable and dynamic aspects of psychological distress and the overall reliability of the Depression, Anxiety and Stress Scales (DASS-21), using data from two independent samples collected on three occasions with 2- to 4-week intervals. The US data (n = 115) was collected before the COVID-19 pandemic, and the New Zealand (NZ) data (n = 114) was obtained during the pandemic. The total DASS-21 demonstrated excellent reliability in measuring enduring symptoms of psychological distress (G = .94-.96) across both samples. While all DASS-21 subscales demonstrated good reliability with the pre-pandemic US sample, the subscales' reliability was below an acceptable level for the NZ sample. Findings from this study indicate that the overall psychological distress is enduring and can be reliably measured by the DASS-21 across different conditions and populations, while shifts across depression, anxiety and stress levels are likely during emergency and uncertainty, as seen in the COVID-19 pandemic.

Revascularisation for Symptomatic Peripheral Artery Disease: External Applicability of the VOYAGER PAD Trial.

OBJECTIVE: In the VOYAGER PAD trial, rivaroxaban 2.5 mg plus aspirin significantly reduced the primary composite efficacy outcome of acute limb ischaemia, major amputation, myocardial infarction, ischaemic stroke, or cardiovascular death compared with aspirin alone. However, patients enrolled in the trial may not reflect patients encountered in daily clinical practice. This study described the proportion of patients eligible for VOYAGER PAD within the nationwide Danish Vascular Registry (DVR), reasons for ineligibility, and outcomes according to eligibility. METHODS: In total, 32 911 patients who underwent lower extremity revascularisation for symptomatic peripheral arterial disease (PAD) in the DVR (2000-2016) were identified. Trial inclusion and exclusion criteria were applied, and the three year cumulative incidence of primary and secondary trial outcomes was estimated. RESULTS: Altogether, 27.1% of patients with PAD in the DVR were "VOYAGER eligible". Of those not included, 30.7% had at least one exclusion criterion ("VOYAGER excluded"), and an additional 42.3% did not fulfil the inclusion criteria ("VOYAGER not included"). The main reasons for exclusion were atrial fibrillation (32.3%), poorly regulated hypertension (20.6%), requirement for long term dual antiplatelet therapy (10.9%), cytochrome P450 inhibitors or inducers (9.7%), and renal failure (9.3%). The three year rate of the primary efficacy outcome was 10.08 per 100 person years among the "VOYAGER eligible", 16.32 among "VOYAGER excluded", and 6.98 among the "VOYAGER not included". For the primary safety outcome of thrombolysis in myocardial infarction (TIMI) major bleeding, rates were 2.24, 3.76, and 1.17, respectively. Rates of secondary endpoints were also consistently lower for patients who did not meet the inclusion criteria (predominantly due to central aorto-iliac procedures) and highest for "VOYAGER excluded" patients. "VOYAGER eligible" patients experienced a higher cumulative incidence of most endpoints than patients enrolled in the control arm of the VOYAGER PAD trial. CONCLUSION: Among patients in routine clinical practice, 27.1% were eligible for the VOYAGER PAD trial. These patients were older, had more severe vascular symptoms, higher bleeding risk, and worse prognosis than trial participants.

Assessing transferability in systematic reviews of health economic evaluations - a review of methodological guidance.

OBJECTIVE: For assessing cost-effectiveness, Health Technology Assessment (HTA) organisations may use primary economic evaluations (P-HEs) or Systematic Reviews of Health Economic evaluations (SR-HEs). A prerequisite for meaningful results of SR-HEs is that the results from existing P-HEs are transferable to the decision context (e.g, HTA jurisdiction). A particularly pertinent issue is the high variability of costs and resource needs across jurisdictions. Our objective was to review the methods documents of HTA organisations and compare their recommendations on considering transferability in SR-HE. METHODS: We systematically hand searched the webpages of 158 HTA organisations for relevant methods documents from 8th January to 31st March 2019. Two independent reviewers performed searches and selected documents according to pre-defined criteria. One reviewer extracted data in standardised and piloted tables and a second reviewer checked them for accuracy. We synthesised data using tabulations and in a narrative way. RESULTS: We identified 155 potentially relevant documents from 63 HTA organisations. Of these, 7 were included in the synthesis. The included organisations have different aims when preparing a SR-HE (e.g. to determine the need for conducting their own P-HE). The recommendations vary regarding the underlying terminology (e.g. transferability/generalisability), the assessment approaches (e.g. structure), the assessment criteria and the integration in the review process. CONCLUSION: Only few HTA organisations address the assessment of transferability in their methodological recommendations for SR-HEs. Transferability considerations are related to different purposes. The assessment concepts and criteria are heterogeneous. Developing standards to consider transferability in SR-HEs is desirable.

Supporting the implementation of new healthcare technologies by investigating generalisability of pilot studies using area-level statistics.

BACKGROUND: Implementation of new technologies into national health care systems requires careful capacity planning. This is sometimes informed by data from pilot studies that implement the technology on a small scale in selected areas. A critical consideration when using implementation pilot studies for capacity planning in the wider system is generalisability. We studied the feasibility of using publicly available national statistics to determine the degree to which results from a pilot might generalise for non-pilot areas, using the English human papillomavirus (HPV) cervical screening pilot as an exemplar. METHODS: From a publicly available source on population indicators in England ("Public Health Profiles"), we selected seven area-level indicators associated with cervical cancer incidence, to produce a framework for post-hoc pilot generalisability analysis. We supplemented these data by those from publicly available English Office for National Statistics modules. We compared pilot to non-pilot areas, and pilot regimens (pilot areas using the previous standard of care (cytology) vs. the new screening test (HPV)). For typical process indicators that inform real-world capacity planning in cancer screening, we used standardisation to re-weight the values directly observed in the pilot, to better reflect the wider population. A non-parametric quantile bootstrap was used to calculate 95% confidence intervals (CI) for differences in area-weighted means for indicators. RESULTS: The range of area-level statistics in pilot areas covered most of the spectrum observed in the wider population. Pilot areas were on average more deprived than non-pilot areas (average index of multiple deprivation 24.8 vs. 21.3; difference: 3.4, 95% CI: 0.2-6.6). Participants in HPV pilot areas were less deprived than those in cytology pilot areas, matching area-level statistics. Differences in average values of the other six indicators were less pronounced. The observed screening process indicators showed minimal change after standardisation for deprivation. CONCLUSIONS: National statistical sources can be helpful in establishing the degree to which the types of areas outside pilot studies are represented, and the extent to which they match selected characteristics of the rest of the health care system ex-post. Our analysis lends support to extrapolation of process indicators from the HPV screening pilot across England.

Dynamic meta-analysis: a method of using global evidence for local decision making.

BACKGROUND: Meta-analysis is often used to make generalisations across all available evidence at the global scale. But how can these global generalisations be used for evidence-based decision making at the local scale, if the global evidence is not perceived to be relevant to local decisions? We show how an interactive method of meta-analysis-dynamic meta-analysis-can be used to assess the local relevance of global evidence. RESULTS: We developed Metadataset ( www.metadataset.com ) as a proof-of-concept for dynamic meta-analysis. Using Metadataset, we show how evidence can be filtered and weighted, and results can be recalculated, using dynamic methods of subgroup analysis, meta-regression, and recalibration. With an example from agroecology, we show how dynamic meta-analysis could lead to different conclusions for different subsets of the global evidence. Dynamic meta-analysis could also lead to a rebalancing of power and responsibility in evidence synthesis, since evidence users would be able to make decisions that are typically made by systematic reviewers-decisions about which studies to include (e.g. critical appraisal) and how to handle missing or poorly reported data (e.g. sensitivity analysis). CONCLUSIONS: In this study, we show how dynamic meta-analysis can meet an important challenge in evidence-based decision making-the challenge of using global evidence for local decisions. We suggest that dynamic meta-analysis can be used for subject-wide evidence synthesis in several scientific disciplines, including agroecology and conservation biology. Future studies should develop standardised classification systems for the metadata that are used to filter and weight the evidence. Future studies should also develop standardised software packages, so that researchers can efficiently publish dynamic versions of their meta-analyses and keep them up-to-date as living systematic reviews. Metadataset is a proof-of-concept for this type of software, and it is open source. Future studies should improve the user experience, scale the software architecture, agree on standards for data and metadata storage and processing, and develop protocols for responsible evidence use.

Shifting research culture to address the mismatch between where trials recruit and where populations with the most disease live: a qualitative study.

BACKGROUND: Research participation is beneficial to patients, clinicians and healthcare services. There is currently poor alignment between UK clinical research activity and local prevalence of disease. The National Institute of Health Research is keen to encourage chief investigators (CIs) to base their research activity in areas of high patient need, to support equity, efficiency and capacity building. We explored how CIs choose sites for their trials and suggest ways to encourage them to recruit from areas with the heaviest burden of disease. METHODS: Qualitative, semi-structured telephone interviews with a purposive sample of 30 CIs of ongoing or recently completed multi-centre trials, all of which were funded by the UK National Institute of Health Research. RESULTS: CIs want to deliver world-class trials to time and budget. Approaching newer, less research-active sites appears risky, potentially compromising trial success. CIs fear that funders may close the trial if recruitment (or retention) is low, with potential damage to their research reputation. We consider what might support a shift in CI behaviour. The availability of 'heat maps' showing the disparity between disease prevalence and current research activity will help to inform site selection. Embedded qualitative research during trial set up and early, appropriate patient and public involvement and engagement can provide useful insights for a more nuanced and inclusive approach to recruitment. Public sector funders could request more granularity in recruitment reports and incentivise research activity in areas of greater patient need. Accounts from the few CIs who had 'broken the mould' suggest that nurturing new sites can be very successful in terms of efficient recruitment and retention. CONCLUSION: While improvements in equity and capacity building certainly matter to CIs, most are primarily motivated by their commitment to delivering successful trials. Highlighting the benefits to trial delivery is therefore likely to be the best way to encourage CIs to focus their research activity in areas of greatest need.

Do interventions to improve adherence to antiretroviral therapy recognise diversity? A systematic review.

People living with HIV (PLWH) are often culturally and linguistically diverse populations; these differences are associated with differing barriers to antiretroviral therapy (ART) adherence. Cultural competence measures the extent to which trial design recognises this diversity. This systematic review aimed to determine whether adherence trial participants represent the diversity of PLWH. Randomised Controlled Trials in Organisation for Economic Co-operation and Development countries to improve ART adherence were eligible. We searched MEDLINE, EMBASE, and Cochrane Database of Systematic Reviews. For all included trials, we searched for their development, testing and evaluation studies. We compared trial participant characteristics with nationally reported PLWH data. We appraised trial cultural competence against ten criteria; scoring each criterion as 0, 1 or 2 indicating cultural blindness, pre-competence or competence respectively. For 80 included trials, a further 13 studies presenting development/testing/evaluation data for the included trials were identified. Only one of the 80 included studies reported trial participants representative of the country's population of PLWH. The median (IQ) cultural competence score was 2.5 (1.0, 4.0) out of 20. HIV adherence trial participants are not reflective of the population with HIV, which may be due to limited adoption of culturally competent research methods.

Are efficient learners of verbal stimuli also efficient and precise learners of visuospatial stimuli?

People differ in how quickly they learn information and how long they remember it, and these two variables are correlated such that people who learn more quickly tend to retain more of the newly learned information. Zerr and colleagues [2018. Learning efficiency: Identifying individual differences in learning rate and retention in healthy adults. Psychological Science, 29(9), 1436-1450] termed the relation between learning rate and retention as learning efficiency, with more efficient learners having both a faster acquisition rate and better memory performance after a delay. Zerr et al. also demonstrated in separate experiments that how efficiently someone learns is stable across a range of days and years with the same kind of stimuli. The current experiments (combined N = 231) replicate the finding that quicker learning coincides with better retention and demonstrate that the correlation extends to multiple types of materials. We also address the generalisability of learning efficiency: A person's efficiency with learning Lithuanian-English (verbal-verbal) pairs predicts their efficiency with Chinese-English (visuospatial-verbal) and (to a lesser extent) object-location (visuospatial-visuospatial) paired associates. Finally, we examine whether quicker learners also remember material more precisely by using a continuous measure of recall accuracy with object-location pairs.

Reliability of simulation-based assessment for practicing physicians: performance is context-specific.

INTRODUCTION: Even physicians who routinely work in complex, dynamic practices may be unprepared to optimally manage challenging critical events. High-fidelity simulation can realistically mimic critical clinically relevant events, however the reliability and validity of simulation-based assessment scores for practicing physicians has not been established. METHODS: Standardised complex simulation scenarios were developed and administered to board-certified, practicing anesthesiologists who volunteered to participate in an assessment study during formative maintenance of certification activities. A subset of the study population agreed to participate as the primary responder in a second scenario for this study. The physicians were assessed independently by trained raters on both teamwork/behavioural and technical performance measures. Analysis using Generalisability and Decision studies were completed for the two scenarios with two raters. RESULTS: The behavioural score was not more reliable than the technical score. With two raters > 20 scenarios would be required to achieve a reliability estimate of 0.7. Increasing the number of raters for a given scenario would have little effect on reliability. CONCLUSIONS: The performance of practicing physicians on simulated critical events may be highly context-specific. Realistic simulation-based assessment for practicing physicians is resource-intensive and may be best-suited for individualized formative feedback. More importantly, aggregate data from a population of participants may have an even higher impact if used to identify skill or knowledge gaps to be addressed by training programs and inform continuing education improvements across the profession.

A sinister subject: Quantifying handedness-based recruitment biases in current neuroimaging research.

Approximately ten per cent of humans are left-handed or ambidextrous (adextral). It has been suggested that, despite their sizable representation at the whole-population level, this demographic is largely avoided by researchers within the neuroimaging community. To date, however, no formal effort has been made to quantify the extent to which adextrals are excluded from neuroimaging-based research. Here, we aimed to address this question in a review of over 1,000 recent articles published in high-impact, peer-reviewed, neuroimaging-focused journals. Specifically, we sought to ascertain whether, and the extent to which adextrals are underrepresented in neuroimaging study samples, and to delineate potential trends in this bias. Handedness data were available for over 30,000 research subjects; only around 3%-4% of these individuals were adextral-considerably less than the 10% benchmark one would expect if neuroimaging samples were truly representative of the general population. This observation was generally consistent across different areas of research, but was modulated by the demographic characteristics of neuroimaging participants. The epistemological and ethical implications of these findings are discussed.

How to improve reliability of cut-off scores in dental competency exam: A comparison of rating methods in standard setting.

AIM: The aim of this study was to find ways to improve reliability of cut-off scores that are typically used to make high-stake decisions in dental education by empirically comparing two different rating methods, Yes/No and Percentage methods. MATERIAL AND METHODS: The two rating methods are commonly used when the Angoff's method is applied to determine a cut-off score that divides the examinees into minimally competent group (pass) and incompetent group (fail). The expert panel data were collected using both methods from 11 to 13 panel members in two consecutive years, respectively; The data were analysed within the generalisability theory framework to quantify relative influences of each factor (eg panel, item, rating rounds) on the variability of cut-off scores, standard error of measurement and panel agreement. RESULTS: The results suggest that (a) the two methods can make a substantial difference in overall success rates for college senior students, (b) item-related variance components are generally large and whilst rater-related variance components are small, (c) standard errors of measurement for the cut-off scores decreased from Cohort 1 to Cohort 2 as the number of items are increased and as the expert panel members are more trained and (d) the Percentage method yielded higher agreement amongst the panel in both years. The results provide practical guidelines for dental educators who make efforts to control the quality of final competency exams and cut-off scores with respect to standard setting practices and panel data analysis. CONCLUSION: It can be concluded that evaluations with Percentage method results in more reliable outcomes compared to those with Yes/No method when criterion-referenced assessment is applied to determine the cut-off scores of competency tests at schools.

Pragmatic trials and implementation science: grounds for divorce?

BACKGROUND: The paper opens with a brief history of two of the major intellectual components of the recent utilitarian turn in clinical research, namely 'pragmatic trials' and 'implementation science'. The two schools of thought developed independently and the paper scrutinises their mutual compatibilities and incompatibilities, asking: i) what do the leading advocates of pragmatic trials assume about the transfer of research findings to real-world practice and ii) what role pragmatic trials can and should play in the evaluation of implementation science strategies. METHODS: The paper utilises 'explication de texte': i) providing a close reading of the inferential logics contained in major published expositions of the two paradigms, and ii) interrogating the conclusions of a pragmatic trial of an intervention providing guidelines on retinal screening aimed at family practitioners. RESULTS: The paper is in two parts. Part 1 unearths some significant incommensurability - the pragmatic trial literature retains an antiquated view of knowledge transfer and is overly optimistic about the wide applicability the findings of pragmatic trials to 'real world' conditions. Part 2 of the paper outlines an empirical strategy to better penetrate the mechanisms of knowledge transfer and to tackle the issue of the generalisabilty of research findings in implementation science. CONCLUSIONS: Pragmatism, classically, is about problem solving and the melding of perspectives. The core research requirement in implementation science is a fundamental shift from the narrow shoulders of pragmatic trials to a model of explanation building based upon a multi-case, multi-method body of evidence.

Assessing the applicability of public health intervention evaluations from one setting to another: a methodological study of the usability and usefulness of assessment tools and frameworks.

BACKGROUND: Public health interventions can be complicated, complex and context dependent, making the assessment of applicability challenging. Nevertheless, for them to be of use beyond the original study setting, they need to be generalisable to other settings and, crucially, research users need to be able to identify to which contexts it may be applicable. There are many tools with set criteria for assessing generalisability/applicability, yet few seem to be widely used and there is no consensus on which should be used, or when. This methodological study aimed to test these tools to assess how easy they were to use and how useful they appeared to be. METHODS: We identified tools from an existing review and an update of its search. References were screened on pre-specified criteria. Included tools were tested by using them to assess the applicability of a Swedish weight management intervention to the English context. Researcher assessments and reflections on the usability and utility of the tools were gathered using a standard pro-forma. RESULTS: Eleven tools were included. Their length, content, style and time required to complete varied. No tool was considered ideal for assessing applicability. Their limitations included unrealistic criteria (requiring unavailable information), a focus on implementation to the neglect of transferability (i.e. little focus on potential effectiveness in the new setting), overly broad criteria (associated with low reliability), and a lack of an explicit focus on how interventions worked (i.e. their mechanisms of action). CONCLUSION: Tools presenting criteria ready to be used may not be the best method for applicability assessments. They are likely to be either too long or incomplete, too focused on differences and fail to address elements that matter for the specific topic of interest. It is time to progress from developing lists of set criteria that are not widely used in the literature, to creating a new approach to applicability assessment. Focusing on mechanisms of action, rather than solely on characteristics, could be a useful approach, and one that remains underutilised in current tools. New approaches to assessing generalisability that evolve away from checklist style assessments need to be developed, tested, reported and discussed.

An exploration of the ecological validity of the Virtual Action Planning-Supermarket (VAP-S) with people with schizophrenia.

People with schizophrenia often have functional limitations that affect their daily activities due to executive function deficits. One way to assess these deficits is through the use of virtual reality programmes that reproduce real-life instrumental activities of daily living (IADLs). One such programme is the Virtual Action Planning-Supermarket (VAP-S). This exploratory study aimed to examine the ecological validity of this programme, specifically, how task performance in both virtual and natural environments compares. Case studies were used and involved five participants with schizophrenia, who were familiar with grocery shopping. They were assessed during both the VAP-S shopping task and a real-life grocery shopping task using an observational assessment tool, the Perceive, Recall, Plan and Perform (PRPP) System of Task Analysis. The results show that when difficulties were present in the virtual task, difficulties were also observed in the real-life task. For some participants, greater difficulties were observed in the virtual task. These difficulties could be explained by the presence of perceptual deficits and problems remembering the required sequenced actions in the virtual task. In conclusion, performance on the VAP-S by these five participants was generally comparable to the performance in a natural environment.

Data quality evaluation for observational multiple sclerosis registries.

OBJECTIVE: Objective and reproducible evaluation of data quality is of paramount importance for studies of 'real-world' observational data. Here, we summarise a standardised data quality, density and generalisability process implemented by MSBase, a global multiple sclerosis (MS) cohort study. METHODS: Error rate, data density score and generalisability score were developed using all 35,869 patients enrolled in MSBase as of November 2015. The data density score was calculated across six domains (follow-up, demography, visits, MS relapses, paraclinical data and therapy) and emphasised data completeness. The error rate evaluated syntactic accuracy and consistency of data. The generalisability score evaluated believability of the demographic and treatment information. Correlations among the three scores and the number of patients per centre were evaluated. RESULTS: Errors were identified at the median rate of 3 per 100 patient-years. The generalisability score indicated the samples' representativeness of the known MS epidemiology. Moderate correlation between the density and generalisability scores (ρ = 0.58) and a weak correlation between the error rate and the other two scores (ρ = -0.32 to -0.33) were observed. The generalisability score was strongly correlated with centre size (ρ = 0.79). CONCLUSION: The implemented scores enable objective evaluation of the quality of observational MS data, with an impact on the design of future analyses.

Describing the participants in a study.

This paper reviews the use of descriptive statistics to describe the participants included in a study. It discusses the practicalities of incorporating statistics in papers for publication in Age and Aging, concisely and in ways that are easy for readers to understand and interpret.

Reliability and acceptability of six station multiple mini-interviews: past-behavioural versus situational questions in postgraduate medical admission.

BACKGROUND: The multiple mini-interview (MMI) is increasingly used for postgraduate medical admissions and in undergraduate settings. MMIs use mostly Situational Questions (SQs) rather than Past-Behavioural Questions (PBQs). A previous study of MMIs in this setting, where PBQs and SQs were asked in the same order, reported that the reliability of PBQs was non-inferior to SQs and that SQs were more acceptable to candidates. The order in which the questions are asked may affect reliability and acceptability of an MMI. This study investigated the reliability of an MMI using both PBQs and SQs, minimising question order bias. Acceptability of PBQs and SQs was also assessed. METHODS: Forty candidates applying for a postgraduate medical admission for 2016-2017 were included; 24 examiners were used. The MMI consisted of six stations with one examiner per station; a PBQ and a SQ were asked at every station, and the order of questions was alternated between stations. Reliability was analysed for scores obtained for PBQs or SQs separately, and for both questions. A post-MMI survey was used to assess the acceptability of PBQs and SQs. RESULTS: The generalisability (G) coefficients for PBQs only, SQs only, and both questions were 0.87, 0.96, and 0.80, respectively. Decision studies suggested that a four-station MMI would also be sufficiently reliable (G-coefficients 0.82 and 0.94 for PBQs and SQs, respectively). In total, 83% of participants were satisfied with the MMI. In terms of face validity, PBQs were more acceptable than SQs for candidates (p = 0.01), but equally acceptable for examiners (88% vs. 83% positive responses for PBQs vs. SQs; p = 0.377). Candidates preferred PBQs to SQs when asked to choose one, though this difference was not significant (p = 0.081); examiners showed a clear preference for PBQs (p = 0.007). CONCLUSIONS: Reliability and acceptability of six-station MMI were good among 40 postgraduate candidates; modelling suggested that four stations would also be reliable. SQs were more reliable than PBQs. Candidates found PBQs more acceptable than SQs and examiners preferred PBQs when they had to choose between the two. Our findings suggest that it is better to ask both PBQs and SQs during an MMI to maximise acceptability.