Revising ethical guidance for the evaluation of programmes and interventions not initiated by researchers.

Public health and service delivery programmes, interventions and policies (collectively, 'programmes') are typically developed and implemented for the primary purpose of effecting change rather than generating knowledge. Nonetheless, evaluations of these programmes may produce valuable learning that helps determine effectiveness and costs as well as informing design and implementation of future programmes. Such studies might be termed 'opportunistic evaluations', since they are responsive to emergent opportunities rather than being studies of interventions that are initiated or designed by researchers. However, current ethical guidance and registration procedures make little allowance for scenarios where researchers have played no role in the development or implementation of a programme, but nevertheless plan to conduct a prospective evaluation. We explore the limitations of the guidance and procedures with respect to opportunistic evaluations, providing a number of examples. We propose that one key missing distinction in current guidance is moral responsibility: researchers can only be held accountable for those aspects of a study over which they have control. We argue that requiring researchers to justify an intervention, programme or policy that would occur regardless of their involvement prevents or hinders research in the public interest without providing any further protections to research participants. We recommend that trial consent and ethics procedures allow for a clear separation of responsibilities for the intervention and the evaluation.

The predictive validity of the Living Goods selection tools for community health workers in Kenya: cohort study.

BACKGROUND: Ensuring that selection processes for Community Health Workers (CHWs) are effective is important due to the scale and scope of modern CHW programmes. However they are relatively understudied. While community involvement in selection should never be eliminated entirely, there are other complementary methods that could be used to help identify those most likely to be high-performing CHWs. This study evaluated the predictive validity of three written tests and two individual sections of a one-to-one interview used for selection into CHW posts in eight areas of Kenya. METHODS: A cohort study of CHWs working for Living Goods in eight local areas of Kenya was undertaken. Data on the selection scores, post-training assessment scores and subsequent on-the-job performance (number of household and pregnancy registrations, number of child assessments, proportion of on-time follow-ups and value of goods sold) were obtained for 547 CHWs. Kendall's tau-b correlations between each selection score and performance outcome were calculated. RESULTS: None of the correlations between selection scores and outcomes reached the 0.3 threshold of an "adequate" predictor of performance. Correlations were higher for the written components of the selection process compared to the interview components, with some small negative correlations found for the latter. CONCLUSIONS: If the measures of performance included in this study are considered critical, then further work to develop the CHW selection tools is required. This could include modifying the content of both tools or increasing the length of the written tests to make them more reliable, for if a test is not reliable then it cannot be valid. Other important outcomes not included in this study are retention in post and quality of care. Other CHW programme providers should consider evaluating their own selection tools in partnership with research teams.

Cost-effectiveness of health care service delivery interventions in low and middle income countries: a systematic review.

BACKGROUND: Low and middle income countries (LMICs) face severe resource limitations but the highest burden of disease. There is a growing evidence base on effective and cost-effective interventions for these diseases. However, questions remain about the most cost-effective method of delivery for these interventions. We aimed to review the scope, quality, and findings of economic evaluations of service delivery interventions in LMICs. METHODS: We searched PUBMED, MEDLINE, EconLit, and NHS EED for studies published between 1st January 2000 and 30th October 2016 with no language restrictions. We included all economic evaluations that reported incremental costs and benefits or summary measures of the two such as an incremental cost effectiveness ratio. Studies were grouped by both disease area and outcome measure and permutation plots were completed for similar interventions. Quality was judged by the Drummond checklist. RESULTS: Overall, 3818 potentially relevant abstracts were identified of which 101 studies were selected for full text review. Thirty-seven studies were included in the final review. Twenty-three studies reported on interventions we classed as "changing by whom and where care was provided", specifically interventions that entailed task-shifting from doctors to nurses or community health workers or from facilities into the community. Evidence suggests this type of intervention is likely to be cost-effective or cost-saving. Nine studies reported on quality improvement initiatives, which were generally found to be cost-effective. Quality and methods differed widely limiting comparability of the studies and findings. CONCLUSIONS: There is significant heterogeneity in the literature, both methodologically and in quality. This renders further comparisons difficult and limits the utility of the available evidence to decision makers.

The Dose-Response Relationship Between Training Load and Aerobic Fitness in Academy Rugby Union Players.

PURPOSE: To identify the dose-response relationship between measures of training load (TL) and changes in aerobic fitness in academy rugby union players. METHOD: Training data from 10 academy rugby union players were collected during a 6-wk in-season period. Participants completed a lactate-threshold test that was used to assess VO2max, velocity at VO2max, velocity at 2 mmol/L (lactate threshold), and velocity at 4 mmol/L (onset of lactate accumulation; vOBLA) as measures of aerobic fitness. Internal-TL measures calculated were Banister training impulse (bTRIMP), Edwards TRIMP, Lucia TRIMP, individualized TRIMP (iTRIMP), and session RPE (sRPE). External-TL measures calculated were total distance, PlayerLoad™, high-speed distance >15 km/h, very-high-speed distance >18 km/h, and individualized high-speed distance based on each player's vOBLA. RESULTS: A second-order-regression (quadratic) analysis found that bTRIMP (R2 = .78, P = .005) explained 78% of the variance and iTRIMP (R2 = .55, P = .063) explained 55% of the variance in changes in VO2max. All other HR-based internal-TL measures and sRPE explained less than 40% of variance with fitness changes. External TL explained less than 42% of variance with fitness changes. CONCLUSIONS: In rugby players, bTRIMP and iTRIMP display a curvilinear dose-response relationship with changes in maximal aerobic fitness.

Variation in passing standards for graduation-level knowledge items at UK medical schools.

OBJECTIVES: Given the absence of a common passing standard for students at UK medical schools, this paper compares independently set standards for common 'one from five' single-best-answer (multiple-choice) items used in graduation-level applied knowledge examinations and explores potential reasons for any differences. METHODS: A repeated cross-sectional study was conducted. Participating schools were sent a common set of graduation-level items (55 in 2013-2014; 60 in 2014-2015). Items were selected against a blueprint and subjected to a quality review process. Each school employed its own standard-setting process for the common items. The primary outcome was the passing standard for the common items by each medical school set using the Angoff or Ebel methods. RESULTS: Of 31 invited medical schools, 22 participated in 2013-2014 (71%) and 30 (97%) in 2014-2015. Schools used a mean of 49 and 53 common items in 2013-2014 and 2014-2015, respectively, representing around one-third of the items in the examinations in which they were embedded. Data from 19 (61%) and 26 (84%) schools, respectively, met the inclusion criteria for comparison of standards. There were statistically significant differences in the passing standards set by schools in both years (effect sizes (f2 ): 0.041 in 2013-2014 and 0.218 in 2014-2015; both p < 0.001). The interquartile range of standards was 5.7 percentage points in 2013-2014 and 6.5 percentage points in 2014-2015. There was a positive correlation between the relative standards set by schools in the 2 years (Pearson's r = 0.57, n = 18, p = 0.014). Time allowed per item, method of standard setting and timing of examination in the curriculum did not have a statistically significant impact on standards. CONCLUSIONS: Independently set standards for common single-best-answer items used in graduation-level examinations vary across UK medical schools. Further work to examine standard-setting processes in more detail is needed to help explain this variability and develop methods to reduce it.

Evaluation of the effect of socio-economic status on performance in a Multiple Mini Interview for admission to medical school.

INTRODUCTION: Widening participation in Medicine is a key policy priority as it helps promote a diverse and representative workforce and improves patient care. The selection process employed can influence the socio-economic composition of the student cohort and this study therefore evaluated whether Multiple Mini Interview (MMI) performance was influenced by school type or area-level Higher Education (HE) participation rates. METHODS: MMI performance for all UK applicants was recorded and consent to link performance with socio-economic data was requested using an applicant questionnaire. Station-level and total MMI scores, and offer rates were compared between applicants from non-selective, non-fee-paying schools and from selective and/or fee-paying schools; and between applicants from each quintile of area-level HE participation. RESULTS: 793 applicants were included in the analysis. MMI performance and offer rates were slightly higher for applicants from non-selective, non-fee-paying schools and/or from lower HE participation quintiles, but the effects were small and not statistically significant. CONCLUSIONS: The MMI did not favour applicants from selective and/or fee-paying schools, or from areas with high HE participation rates. Work to evaluate whether the relationship between MMI and medical school performance is the same for all sub-groups of students is now required.

The effects of laryngeal mask airway passage simulation training on the acquisition of undergraduate clinical skills: a randomised controlled trial.

BACKGROUND: Effective use of the laryngeal mask airway (LMA) requires learning proper insertion technique in normal patients undergoing routine surgical procedures. However, there is a move towards simulation training for learning practical clinical skills, such as LMA placement. The evidence linking different amounts of mannequin simulation training to the undergraduate clinical skill of LMA placement in real patients is limited. The purpose of this study was to compare the effectiveness in vivo of two LMA placement simulation courses of different durations. METHODS: Medical students (n = 126) enrolled in a randomised controlled trial. Seventy-eight of these students completed the trial. The control group (n = 38) received brief mannequin training while the intervention group (n = 40) received additional more intensive mannequin training as part of which they repeated LMA insertion until they were proficient. The anaesthetists supervising LMA placements in real patients rated the participants' performance on assessment forms. Participants completed a self-assessment questionnaire. RESULTS: Additional mannequin training was not associated with improved performance (37% of intervention participants received an overall placement rating of > 3/5 on their first patient compared to 48% of the control group, X2 = 0.81, p = 0.37). The agreement between the participants and their instructors in terms of LMA placement success rates was poor to fair. Participants reported that mannequins were poor at mimicking reality. CONCLUSIONS: The results suggest that the value of extended mannequin simulation training in the case of LMA placement is limited. Educators considering simulation for the training of practical skills should reflect on the extent to which the in vitro simulation mimics the skill required and the degree of difficulty of the procedure.

Development of a modified Cohen method of standard setting.

BACKGROUND: A new 'Cohen' approach to standard setting was recently described where the pass mark is calculated as 60% of the score of the student at the 95th percentile, after correcting for guessing. AIM: This article considers how two potential criticisms of the Cohen method can be addressed and proposes a modified version, with the assumptions tested using local data. METHODS: The modified version removes the correction for guessing and uses the score of the 90th, rather than the 95th percentile student as the reference point, based on the cumulative density functions for 32 modules from one medical school; and incorporates an indirect criterion-referenced passing standard by changing the 60% multiplier to the ratio of the cut score to the score of the student at the 90th percentile on exams that have been standard set using modified Angoff. RESULTS: The assumption that the performance of the 90th percentile student is consistent over time holds for multiple choice questions. Applying the modified Cohen method to the 32 modules generally reduced the variation in failure rate across modules, compared to a fixed pass mark of 50%. CONCLUSION: The results suggest that the modified Cohen method holds much promise as an economical approach to standard setting.