Predicting reliability through structured expert elicitation with the repliCATS (Collaborative Assessments for Trustworthy Science) process.

As replications of individual studies are resource intensive, techniques for predicting the replicability are required. We introduce the repliCATS (Collaborative Assessments for Trustworthy Science) process, a new method for eliciting expert predictions about the replicability of research. This process is a structured expert elicitation approach based on a modified Delphi technique applied to the evaluation of research claims in social and behavioural sciences. The utility of processes to predict replicability is their capacity to test scientific claims without the costs of full replication. Experimental data supports the validity of this process, with a validation study producing a classification accuracy of 84% and an Area Under the Curve of 0.94, meeting or exceeding the accuracy of other techniques used to predict replicability. The repliCATS process provides other benefits. It is highly scalable, able to be deployed for both rapid assessment of small numbers of claims, and assessment of high volumes of claims over an extended period through an online elicitation platform, having been used to assess 3000 research claims over an 18 month period. It is available to be implemented in a range of ways and we describe one such implementation. An important advantage of the repliCATS process is that it collects qualitative data that has the potential to provide insight in understanding the limits of generalizability of scientific claims. The primary limitation of the repliCATS process is its reliance on human-derived predictions with consequent costs in terms of participant fatigue although careful design can minimise these costs. The repliCATS process has potential applications in alternative peer review and in the allocation of effort for replication studies.

Systematic development of quality indicators for skin cancer management in primary care: a mixed-methods study protocol.

INTRODUCTION: Australia has the highest incidence of skin cancer in the world, with two out of three Australians expected to be diagnosed with skin cancer in their lifetime. Such incidence necessitates large-scale, effective skin cancer management practices. General practitioners (in mainstream practice and in skin cancer clinics) play an important role in skin cancer care provision, making decisions based on relevant evidence-based guidelines, protocols, experience and training. Diversity in these decision-making practices can result in unwarranted variation. Quality indicators are frequently implemented in healthcare contexts to measure performance quality at the level of the clinician and healthcare practice and mitigate unwarranted variation. Such measurements can facilitate performance comparisons between peers and a standard benchmark, often resulting in improved processes and outcomes. A standardised set of quality indicators is yet to be developed in the context of primary care skin cancer management. AIMS: This research aims to identify, develop and generate expert consensus on a core set of quality indicators for skin cancer management in primary care. METHODS: This mixed-methods study involves (1) a scoping review of the available evidence on quality indicators in skin cancer management in primary care, (2) identification and development of a core set of quality indicators through interviews/qualitative proforma surveys with participants, and (3) a focus group involving discussion of quality indicators according to Nominal Group Technique. Qualitative and quantitative data will be collected and analysed using thematic and descriptive statistical analytical methods. ETHICS AND DISSEMINATION: Approval was granted by the university's Research Ethics Committee (HREC no. 520211051532420). Results from this study will be widely disseminated in publications, study presentations, educational events and reports.

Selective laryngoscopy before thyroidectomy: a risk assessment.

BACKGROUND: Patients presenting for thyroidectomy may have an unrecognized pre-existing vocal cord palsy (VCP). This raises the danger of bilateral VCP if a patient sustains an injury to the RLN on the sole functioning side. Part of the rationale for routine preoperative laryngoscopy is to eliminate such a risk. This paper endeavours to quantify the relevant potential risk. METHODS: Patients who underwent laryngoscopy prior to thyroid or parathyroid surgery in an endocrine surgical unit over a 5 year period were identified. Literature review revealed four papers in which VCP prevalence in patients without risk factors was reported. Using our data, combined with that of these other authors, the background rate of pre-existing VCP was ascertained, and the subsequent risk of bilateral VCP estimated. RESULTS: Of our 632 patients who underwent preoperative laryngoscopy, there were four patients (0.63%) who were found to have a unilateral VCP, but all had voice symptoms or previous neck surgery. When patients with these risk factors are excluded, our data combined with the published data provides a pre-existing VCP rate of 0.2%. Calculations estimate that if preoperative laryngoscopy is omitted in patients with no risk factors, the risk of bilateral VCP, due to the nerve on the sole functioning side being injured, would be between 1/50000 and 1/150000, depending on an individual surgeon's level of experience. CONCLUSION: Selective use of laryngoscopy prior to thyroidectomy would result in an acceptably low statistical risk of bilateral VCP. Routine laryngoscopy for all patients is not necessary.

Assessment of insulin sensitivity during hyperbaric oxygen treatment.

INTRODUCTION: Previous studies using a hyperinsulinaemic, euglycaemic glucose clamp have demonstrated an increase in peripheral insulin sensitivity in men with and without Type-2 diabetes mellitus on the third and thirtieth hyperbaric oxygen treatment (HBOT) session. In two studies using different techniques for assessment of insulin sensitivity, we investigated the onset and duration of this insulin-sensitising effect of HBOT. METHODS: Men who were obese or overweight but without diabetes were recruited. One study performed a hyperinsulinaemic euglycaemic glucose clamp (80 mU.m-2.min-1) at baseline and during the first HBOT exposure (n = 9) at a pressure of 203 kPa. Data were analysed by paired t-test. The other study assessed insulin sensitivity by a frequently sampled intravenous glucose tolerance test (FSIGT) at three time points: baseline, during the third HBOT and 24-hours post-HBOT (n = 9). Results were analysed by repeated-measures ANOVA. RESULTS: There was a significant 23% increase in insulin sensitivity by clamp measured during the first HBOT exposure. The FSIGT showed no significant changes in insulin sensitivity. CONCLUSIONS: The hyperinsulinaemic, euglycaemic glucose clamp demonstrated a significant increase in peripheral insulin sensitivity during a single, 2-hour HBOT session in a group of men who were obese or overweight but without diabetes. As an alternate technique for assessing insulin sensitivity during HBOT, the FSIGT failed to show any changes during the third HBOT and 24-hours later, however modification of the study protocol should be considered.

An Elemental Impurities Excipient Database: A Viable Tool for ICH Q3D Drug Product Risk Assessment.

To support the practical implementation of the International Council for Harmonisation (ICH) Q3D guideline, which describes a risk-based approach to the control of elemental impurities in drug products, a consortium of pharmaceutical companies has established a database to collate the results of analytical studies of the levels of elemental impurities within pharmaceutical excipients. This database currently includes the results of 26,723 elemental determinations for 201 excipients and represents the largest known, and still rapidly expanding, collection of data of this type. Analysis of the database indicates good coverage of excipients relevant to real-world drug product formulations and tested element profiles consistent with ICH Q3D recommendations. The database includes the results from multiple analytical studies for an excipient and thus incorporates within it an indication of both excipient supplier and batch-to-batch variability as well as any variability associated with the different testing organizations and methods employed. The data confirm the findings of earlier smaller studies that elemental impurity concentrations in excipients are generally low and when used in typical proportions in formulated drug products are unlikely to pose a significant patient safety risk. The database is now in active use as one line of evidence in ICH Q3D risk assessments.

The rationale for and use of assessment frameworks: improving assessment and reporting quality in medical education.

An assessment framework provides a structured conceptual map of the learning outcomes of a programme of study along with details of how achievement of the outcomes can be measured. The rationale for using frameworks to underpin the targeting of essential content components is especially relevant for the medical education community. Frameworks have the capacity to improve validity and reliability in assessment, allowing test developers to more easily create robust assessment instruments. The framework used by the Australian Medical Assessment Collaboration (AMAC) is an interesting and relevant case study for the international community as it draws and builds on established processes in higher education assessment. The AMAC experience offers an insight into important considerations for designing assessment frameworks and implementing frameworks in differing contexts. There are lessons which have the potential to improve assessment and reporting practice and quality in not only medical education, but in other domains of assessment. Prior to implementing any programme of assessment, the framework considerations outlined here will hopefully improve the quality of assessment and reporting practice by making implicit assumptions explicit, and allowing more critical reflection and evaluation throughout assessment processes.

Access to cardiac rehabilitation does not equate to attendance.

BACKGROUND/AIMS: Timely access to appropriate cardiac care is critical for optimizing positive outcomes after a cardiac event. Attendance at cardiac rehabilitation (CR) remains less than optimal (10%-30%). Our aim was to derive an objective, comparable, geographic measure reflecting access to cardiac services after a cardiac event in Australia. METHODS: An expert panel defined a single patient care pathway and a hierarchy of the minimum health services for CR and secondary prevention. Using geographic information systems a numeric/alpha index was modelled to describe access before and after a cardiac event. The aftercare phase was modelled into five alphabetical categories: from category A (access to medical service, pharmacy, CR, pathology within 1 h) to category E (no services available within 1 h). RESULTS: Approximately 96% or 19 million people lived within 1 h of the four basic services to support CR and secondary prevention, including 96% of older Australians and 75% of the indigenous population. Conversely, 14% (64,000) indigenous people resided in population locations that had poor access to health services that support CR after a cardiac event. CONCLUSION: Results demonstrated that the majority of Australians had excellent 'geographic' access to services to support CR and secondary prevention. Therefore, it appears that it is not the distance to services that affects attendance. Our 'geographic' lens has identified that more research on socioeconomic, sociological or psychological aspects to attendance is needed.

Developing outcomes assessments for collaborative, cross-institutional benchmarking: progress of the Australian Medical Assessment Collaboration.

BACKGROUND: The Australian Medical Assessment Collaboration (AMAC) began in 2010. This article charts the development of the collaboration over its initial years. AIMS: AMAC was instigated as a way of improving the quality of medical education through the recognition of the need for tools for comparison and evaluation of learning outcomes, acknowledgement of the need for high quality assessment, and to share expertise in these areas. In a climate of increasing regulation and accountability, this collaboration was formed as a means of increasing assessment practices by, with and for medical schools. METHOD: This article provides an overview of the background issues stimulating the development of AMAC, discussion of the formation of the collaboration and reflection on the lessons learnt through these processes. RESULTS: In a relatively short space of time, AMAC has fostered substantial collaboration among schools; developed an Assessment Framework, items and an online assessment; and provided benchmarking reports to students and schools. CONCLUSION: The intention here is to provide guidance for others (within the medical education community and those in other disciplines) with similar intentions and aims, by outlining the developmental pathway of the project and the systematic lessons that the collaboration team has learnt in establishing AMAC.

Comparing badger (Meles meles) management strategies for reducing tuberculosis incidence in cattle.

Bovine tuberculosis (bTB), caused by Mycobacterium bovis, continues to be a serious economic problem for the British cattle industry. The Eurasian badger (Meles meles) is partly responsible for maintenance of the disease and its transmission to cattle. Previous attempts to manage the disease by culling badgers have been hampered by social perturbation, which in some situations is associated with increases in the cattle herd incidence of bTB. Following the licensing of an injectable vaccine, we consider the relative merits of management strategies to reduce bTB in badgers, and thereby reduce cattle herd incidence. We used an established simulation model of the badger-cattle-TB system and investigated four proposed strategies: business as usual with no badger management, large-scale proactive badger culling, badger vaccination, and culling with a ring of vaccination around it. For ease of comparison with empirical data, model treatments were applied over 150 km(2) and were evaluated over the whole of a 300 km(2) area, comprising the core treatment area and a ring of approximately 2 km. The effects of treatment were evaluated over a 10-year period comprising treatment for five years and the subsequent five year period without treatment. Against a background of existing disease control measures, where 144 cattle herd incidents might be expected over 10 years, badger culling prevented 26 cattle herd incidents while vaccination prevented 16. Culling in the core 150 km(2) plus vaccination in a ring around it prevented about 40 cattle herd breakdowns by partly mitigating the negative effects of culling, although this approach clearly required greater effort. While model outcomes were robust to uncertainty in parameter estimates, the outcomes of culling were sensitive to low rates of land access for culling, low culling efficacy, and the early cessation of a culling strategy, all of which were likely to lead to an overall increase in cattle disease.

Collaborative research between academia and industry using a large clinical trial database: a case study in Alzheimer's disease.

BACKGROUND: Large clinical trials databases, developed over the course of a comprehensive clinical trial programme, represent an invaluable resource for clinical researchers. Data mining projects sponsored by industry that use these databases, however, are often not viewed favourably in the academic medical community because of concerns that commercial, rather than scientific, goals are the primary purpose of such endeavours. Thus, there are few examples of sustained collaboration between leading academic clinical researchers and industry professionals in a large-scale data mining project. We present here a successful example of this type of collaboration in the field of dementia. METHODS: The Donepezil Data Repository comprised 18 randomised, controlled trials conducted between 1991 and 2005. The project team at Pfizer determined that the data mining process should be guided by a diverse group of leading Alzheimer's disease clinical researchers called the "Expert Working Group." After development of a list of potential faculty members, invitations were extended and a group of seven members was assembled. The Working Group met regularly with Eisai/Pfizer clinicians and statisticians to discuss the data, identify issues that were currently of interest in the academic and clinical communities that might lend themselves to investigation using these data, and note gaps in understanding or knowledge of Alzheimer's disease that these data could address. Leadership was provided by the Pfizer Clinical Development team leader; Working Group members rotated responsibility for being lead and co-lead for each investigation and resultant publication. RESULTS: Six manuscripts, each published in a leading subspecialty journal, resulted from the group's work. Another project resulted in poster presentations at international congresses and two were cancelled due to resource constraints. CONCLUSIONS: The experience represents a particular approach to optimising the value of data mining of large clinical trial databases for the combined purpose of furthering clinical research and improving patient care. Fruitful collaboration between industry and academia was fostered while the donepezil data repository was used to advance clinical and scientific knowledge. The Expert Working Group approach warrants consideration as a blueprint for conducting similar research ventures in the future.

Assessment for selection for the health care professions and specialty training: consensus statement and recommendations from the Ottawa 2010 Conference.

Assessment for selection in medicine and the health professions should follow the same quality assurance processes as in-course assessment. The literature on selection is limited and is not strongly theoretical or conceptual. For written testing, there is evidence of the predictive validity of Medical College Admission Test (MCAT) for medical school and licensing examination performance. There is also evidence for the predictive validity of grade point average, particularly in combination with MCAT for graduate entry but little evidence about the predictive validity of school leaver scores. Interviews have not been shown to be robust selection measures. Studies of multiple mini-interviews have indicated good predictive validity and reliability. Of other measures used in selection, only the growing interest in personality testing appears to warrant future work. Widening access to medical and health professional programmes is an increasing priority and relates to the social accountability mandate of medical and health professional schools. While traditional selection measures do discriminate against various population groups, there is little evidence on the effect of non-traditional measures in widening access. Preparation and outreach programmes show most promise. In summary, the areas of consensus for assessment for selection are small in number. Recommendations for future action focus on the adoption of principles of good assessment and curriculum alignment, use of multi-method programmatic approaches, development of interdisciplinary frameworks and utilisation of sophisticated measurement models. The social accountability mandate of medical and health professional schools demands that social inclusion, workforce issues and widening of access are embedded in the principles of good assessment for selection.

The clinical consequences and challenges of hypertension in urban-dwelling black Africans: insights from the Heart of Soweto Study.

BACKGROUND: There is a paucity of data to describe advanced forms of cardiovascular disease (CVD) in urban black Africans with hypertension (HT). METHODS: Chris Hani Baragwanath Hospital services the black African community of 1.1 million people in Soweto, South Africa. We prospectively collected detailed demographic and clinical data from all de novo presentations to the hospital's Cardiology Unit in 2006. RESULTS: Overall, 761 black African patients (56% of de novo cases) presented with a diagnosis of HT with more women (63%, aged 58.5±14.9 years) than men (aged 58.0±15.6 years). On presentation, 396 women (82%) versus 187 men (67%) had dizziness, palpitations and/or chest pain (OR 1.23, 95% 1.12-1.34: p<0.0001). HT was the primary diagnosis in 266 cases (35%). In the rest (n=495), non-ischaemic forms of heart failure were common (54% of total) while only 6.2% had coronary artery disease. Concurrent left ventricular hypertrophy, renal dysfunction and anaemia were present in 39%, 24% and 11% of cases, respectively, with a similar age-adjusted pattern of co-morbidity according to sex. However, men were more likely to present with impaired systolic function (OR 2.13, 95% CI 1.50 to 3.00; p<0.0001). CONCLUSIONS: In the absence of effective primary and secondary prevention strategies, these unique data highlight the potentially devastating impact of advanced forms of hypertensive heart disease in urban black African communities with more women than men affected.

Meeting local complex health needs by building the capacity of general practice: the University of Queensland GP super clinic model.

The GP Super Clinics Program is a highly topical and controversial initiative with varying levels of support within the policy, consumer and health care communities. Here, we describe the GP super clinic initiative of the University of Queensland (UQ), and how it aims to enhance primary-care capacity in the regions where clinics are based. The UQ GP super clinic model has considered the concerns of general practitioners, patients and other stakeholders, and addresses the needs of these groups while providing an excellent opportunity for the university to be involved in innovative service delivery, community-based education, primary-care service design and evaluation.