Position statement of the GMA committee "teaching evaluation".

The evaluation of teaching can be an essential driver for curriculum development. Instruments for teaching evaluation are not only used for the purpose of quality assurance but also in the context of medical education research. Therefore, they must meet the common requirements for reliability and validity. This position paper from the GMA Teaching Evaluation Committee discusses strategic and methodological aspects of evaluation in the context of undergraduate medical education and related courses; and formulates recommendations for the further development of evaluation. First, a four-step approach to the design and implementation of evaluations is presented, then methodological and practical aspects are discussed in more detail. The focus here is on target and confounding variables, survey instruments as well as aspects of implementation and data protection. Finally, possible consequences from evaluation data for the four dimensions of teaching quality (structural and procedural aspects, teachers and outcomes) are discussed.

A Weighted Evaluation Study of Clinical Teacher Performance at Five Hospitals in the UK.

INTRODUCTION: Evaluation of individual teachers in undergraduate medical education helps clinical teaching fellows identify their own strengths and weaknesses. In addition, evaluation data can be used to guide career decisions. In order for evaluation results to adequately reflect true teaching performance, a range of parameters should be considered when designing data collection tools. METHODS: Clinical teaching fellows at five London teaching hospitals were evaluated by third-year students they had supervised during a ten-week clinical attachment. The questionnaire addressed (a) general teaching skills and (b) student learning outcome measured via comparative self-assessments. Teachers were ranked using different algorithms with various weights assigned to these two factors. RESULTS: A total of 133 students evaluated 14 teaching fellows. Overall, ratings on teaching skills were largely favourable, while the perceived increase in student performance was modest. Considerable variability across teachers was observed for both factors. Teacher rankings were strongly influenced by the weighting algorithm used. Depending on the algorithm, one teacher was assigned any rank between #2 and #10. CONCLUSION: Both parts of the questionnaire address different outcomes and thus highlight specific strengths and weaknesses of individual teachers. Programme directors need to carefully consider the weight assigned to individual components of teacher evaluations in order to ensure a fair appraisal of teacher performance.

A prediction-based method to estimate student learning outcome: Impact of response rate and gender differences on evaluation results.

BACKGROUND: Low response rates threaten the reliability and validity of student evaluations of teaching. Previous research has shown that asking students to predict how satisfied their fellow students were with a course produces reliable results at lower response rates. The aim of this study was to investigate whether this prediction-based method can also be used to evaluate student learning outcome. METHODS: Before and after a cardiorespiratory module, 128 fourth-year medical students provided self-assessments and predictions of performance on 27 specific learning objectives and took formative tests on the respective contents. Pre-post performance gain was compared across all three modalities. RESULTS: Formative exam results indicated a performance gain of 63.0%. Self-assessed and prediction-based performance gains were identical (67.8%) but both slightly overestimated actual performance gain. Irrespective of the method used, a response rate of 20% was sufficient to produce reliable results. Compared to male students, females greatly overestimated their peers' performance which led to inflated performance gain values. CONCLUSIONS: Student self-assessments and predictions are equally valid sources of learning outcome measures, and low response rates are sufficient to produce stable results. When using a prediction-based approach, a tendency to overestimate learning outcome in female students needs to be taken into account.

National survey of evaluation practices and performance-guided resource allocation at German medical schools.

Background: Little is known about evaluation practices as well as performance-oriented allocation of resources according to teaching quality at German medical schools. For this reason, the Association of the Scientific Medical Societies in Germany and the German Association of Medical Faculties aimed to analyse current practices at German medical schools. Methods: Data were collected by a questionnaire which was sent to all medical schools in Germany. Results: 30 medical schools with 33 undergraduate medical programs participated in the survey (response rate: 83%). The evaluation tools used at these schools mainly assessed structural and procedural aspects of teaching and were designed to obtain overall student ratings of teaching quality. Evaluation tools were quite heterogeneous across the sample, and some uncertainty remained with regard to the psychometric properties of these tools and whether they meet international quality standards. Various algorithms underlying resource allocation for teaching are being used, but most focus on quantity rather than quality of teaching. Conclusion: A nationwide agreement on a generalizable definition of high-quality teaching is desirable. At the same time, reliable and valid tools measuring teaching quality need to be identified and/or created. This could be accomplished through a wider collaboration of medical schools and could represent an advancement for the allocation of resources for high-quality teaching.

Adjuvant treatment patterns and outcomes in patients with stage IB-IIIA non-small cell lung cancer in France, Germany, and the United Kingdom based on the LuCaBIS burden of illness study.

OBJECTIVES: To inform health-technology assessments of new adjuvant treatments, we describe treatment patterns in patients with complete resection of stage IB-IIIA non-small cell lung cancer (NSCLC) in France, Germany, and the United Kingdom (UK). MATERIALS AND METHODS: Data were collected via medical record abstraction. Patients were aged ≥18 years with completely resected stage IB-IIIA NSCLC, diagnosed between 01 January 2009 and 31 December 2011. Median follow-up was 26 months. Adjuvant treatment patterns and clinical outcomes were summarized descriptively. RESULTS: Among the 831 patients studied, 239 (29%) had stage IB disease, 179 (22%) had stage IIA disease, 165 (20%) had stage IIB disease, and 248 (30%) had stage IIIA disease. Adjuvant systemic therapy was received by 402 patients (48.4%), (France, 61.8%; Germany, 51.9%; UK, 33.4%). Use of adjuvant therapy increased with increasing stage of disease. Cisplatin/vinorelbine and carboplatin/vinorelbine were the most frequently prescribed adjuvant regimens. Median disease-free survival was 48.0 months (95% confidence interval [CI] 42.3-not estimable); the 25th percentile was 13.2 months (95% CI, 11.0-15.3). 204 patients (24%) died during the follow-up period. The median overall survival was not reached, the 25th percentile was 31.2 months (95% CI 26.8-36.0 months). 272 patients (33%) had disease recurrence during the follow-up period. For 86 of those patients, the first recurrence was local or regional with no distant metastasis and 14 had further progression to metastatic disease during the follow-up time. For the other 186 patients, the first recurrence involved distant metastases. A total of 200 patients had metastatic disease at any time during study follow-up. CONCLUSIONS: Less than half the patients with stage IB-IIIA NSCLC in this observational study received adjuvant systemic therapy. A high rate of first recurrence with distant metastatic disease was observed, emphasising the need for more effective systemic adjuvant therapies in this population.

Economic burden of resected (stage IB-IIIA) non-small cell lung cancer in France, Germany and the United Kingdom: A retrospective observational study (LuCaBIS).

OBJECTIVES: New adjuvant treatments are being developed for patients with resected non-small cell lung cancer (NSCLC). Due to scarcity of real-world data available for treatment costs and resource utilization, health technology and cost-effectiveness assessments can be limited. We estimated the burden and cost-of-illness associated with completely resected stage IB-IIIA NSCLC in France, Germany and the United Kingdom (UK). MATERIALS AND METHODS: Eligible patients were aged ≥18 years with completely resected stage IB-IIIA NSCLC between August 2009 and July 2012. Patients (living or deceased) were enrolled at clinical sites by a systematic sampling method. Data were obtained from medical records and patient surveys. Direct, indirect and patient out-of-pocket expenses were estimated by multiplying resource use by country-specific unit costs. National annual costs were estimated based on disease prevalence data available from published sources. RESULTS: 39 centers provided data from 831 patients of whom patient surveys were evaluable in 306 patients. Median follow-up was 26 months. The mean total direct costs per patient during follow-up were: €19,057 (France), €14,185 (Germany), and €8377 (UK). The largest cost drivers were associated with therapies received (€12,375 France; €3694 UK), and hospitalization/emergency costs (€7706 Germany). Monthly direct costs per patient were the highest during the distant metastasis/terminal illness phase in France (€15,562) and Germany (€6047) and during the adjuvant treatment period in the UK (€2790). Estimated mean total indirect costs per patient were: €696 (France), €2476 (Germany), and €1414 (UK). Estimates for the annual national direct cost were €478.4 million (France), €574.6 million (Germany) and €325.8 million (UK). CONCLUSION: To our knowledge, this is the first comprehensive study describing the burden of illness for patients with completely resected stage IB-IIIA NSCLC. The economic burden was substantial in all three countries. Treatment of NSCLC is associated with large annual national costs, mainly incurred during disease progression.

Progression-free survival/time to progression as a potential surrogate for overall survival in HR+, HER2- metastatic breast cancer.

BACKGROUND: Several recent randomized controlled trials (RCTs) in hormone receptor-positive (HR+), human epidermal growth factor receptor 2-negative (HER2-) metastatic breast cancer (MBC) have demonstrated significant improvements in progression-free survival (PFS); however, few have reported improvement in overall survival (OS). The surrogacy of PFS or time to progression (TTP) for OS has not been formally investigated in HR+, HER2- MBC. METHODS: A systematic literature review of RCTs in HR+, HER2- MBC was conducted to identify studies that reported both median PFS/TTP and OS. The correlation between PFS/TTP and OS was evaluated using Pearson's product-moment correlation and Spearman's rank correlation. Subgroup analyses were performed to explore possible reasons for heterogeneity. Errors-in-variables weighted least squares regression (LSR) was used to model incremental OS months as a function of incremental PFS/TTP months. An exploratory analysis investigated the impact of three covariates (chemotherapy vs hormonal/targeted therapy, PFS vs TTP, and first-line therapy vs second-line therapy or greater) on OS prediction. The lower 95% prediction band was used to determine the minimum incremental PFS/TTP months required to predict OS benefit (surrogate threshold effect [STE]). RESULTS: Forty studies were identified. There was a statistically significant correlation between median PFS/TTP and OS (Pearson =0.741, P=0.000; Spearman =0.650, P=0.000). These results proved consistent for chemotherapy and hormonal/targeted therapy. Univariate LSR analysis yielded an R2 of 0.354 with 1 incremental PFS/TTP month corresponding to 1.13 incremental OS months. Controlling the type of treatment (chemotherapy vs hormonal/targeted therapy), line of therapy (first vs subsequent), and progression measure (PFS vs TTP) led to an improved R2 of 0.569 with 1 PFS/TTP month corresponding to 0.78 OS months. The STE for OS benefit was 5-6 months of incremental PFS/TTP. CONCLUSION: We demonstrated a significant association between PFS/TTP and OS, which may justify the use of PFS/TTP as a surrogate for OS benefit in HR+, HER2- MBC.

Is progression-free survival a more relevant endpoint than overall survival in first-line HR+/HER2- metastatic breast cancer?

BACKGROUND: Hormone receptor-positive (HR+), human epidermal growth factor receptor 2-negative (HER2-), metastatic breast cancer (MBC) accounts for 73% of all MBCs. Endocrine therapy (ET) is the basis of first-line (1L) therapy for patients with HR+/HER2- MBC. Novel therapies have demonstrated improvements in progression-free survival (PFS) compared to ET. The clinical relevance of PFS is being debated, as there is no proven direct correlation with overall survival (OS) benefit to date. We reviewed studies of HR+/HER2- MBC to assess PFS and other factors that influence OS and treatment response, and health-related quality of life (HRQoL). METHODS: The Embase®, Medline®, and Cochrane databases were systematically searched to identify studies in adult women with HR+/HER2- MBC, published between January 2006 and January 2017, and written in English. Phase II and III randomized controlled trials (RCTs), observational, and retrospective studies were included. RESULTS: Seventy-nine RCTs were identified: 58 (73%) in the 1L+ setting and 21 (27%) in second-line or greater settings. PFS hazard ratios (HRs) were reported in 61 (77%) studies; 31 (39%) reported significant PFS improvements. OS was reported in 44 (41%) studies; 12 (15%) reported significant OS improvements. Significant improvements in both PFS and OS were reported in only 6 (8%) studies (1 Phase II; 5 Phase III). Patients with HER2- MBC received, on average, ≥5 lines of therapy, with no consistent treatment pathway. Baseline characteristics, prior therapies, and the type and number of post-progression therapies significantly impacted OS. PFS, response rates, and HRQoL decreased with each line of therapy (EuroQol 5 Dimensions: 0.78 1L vs. 0.70 post-progression). CONCLUSION: Few RCTs in HR+/HER2- MBC have demonstrated significant improvements in OS. Factors other than choice of 1L therapy impact OS, including post-progression therapies, which cannot be controlled in RCTs. This study emphasizes the importance of PFS improvement in 1L treatment of HR+/HER2- MBC.

Health economic evaluations of non-pharmacological interventions for persons with dementia and their informal caregivers: a systematic review.

BACKGROUND: This systematic review aims to review the literature on trial-based economic evaluations of non-pharmacological interventions directly targeted at persons with dementia as well as persons with mild cognitive impairment and their respective caregivers. METHODS: A systematic literature research was conducted for the timeframe from 2010 to 2016 in the following databases: Centre for Reviews and Dissemination, EconLit, Embase, Cochrane Library, PsycINFO and PubMed. Study quality was assessed according to the Drummond criteria. RESULTS: In total sixteen publications were identified. Health economic evaluations indicated the cost-effectiveness of physical exercise interventions and occupational therapy. There was also evidence to suggest that psychological and behavioral therapies are cost-effective. Health economic studies investigating psychosocial interventions mainly targeted towards informal caregivers showed inconsistent results. CONCLUSIONS: Due to the increasing prevalence of dementia non-pharmacological interventions and their health economic impact are of increasing importance for health care decision-makers and HTA agencies.

Diagnosis of cognitive decline and dementia in rural areas - A scoping review.

OBJECTIVES: Due to the demographic change, the global prevalence of dementia will continually rise. Barriers to diagnosis and care are still high. But timely diagnosis is associated with valuable benefits and can promote timely and optimal management. Receiving an early diagnosis is especially in rural areas a problem due to the limited access to assessments. Therefore, the aim of our scoping review is to investigate different interventions targeted at rural living elderly to screen and diagnose cognitive decline and dementia. METHODS: A scoping review was conducted in line with the framework of Arksey and O'Malley. The following databases were systematically searched: PubMed, PsycINFO, Cochrane Library, and ScienceDirect. The interventions were categorized in four main categories (interventions for general practitioners/institutions; online/mobile offers; telehealth applications; telephone-based screenings). RESULTS: Thirty studies were included. The four categories show different scopes of application. Telehealth applications show that it is feasible and valid to diagnose dementia via videoconference. Assessments described in three other categories show that remotely used tools are appropriate to screen for mild cognitive impairment or cognitive decline, but are not valid to establish a dementia diagnosis. CONCLUSIONS: Telehealth applications can appropriately be used to diagnose dementia. However, most of the studies included only small sample sizes and did not test the applications explicitly in rural or remote populations. Therefore, studies taking these limitations into account are needed. On top, only two RCTs are included in this review indicating that more high quality studies in this field are needed.

Economic burden of advanced melanoma in France, Germany and the UK: a retrospective observational study (Melanoma Burden-of-Illness Study).

The aim of this study was to estimate the cost-of-illness associated with completely resected stage IIIB/IIIC melanoma with macroscopic lymph node involvement, overall and by disease phase, in France, Germany and the UK. This retrospective observational study included patients aged older than or equal to 18 years first diagnosed with stage IIIB/IIIC cutaneous melanoma between 1 January 2009 and 31 December 2011. Data were obtained from medical records and a patient survey. Direct costs, indirect costs and patient out-of-pocket expenses were estimated in euros (€) (and British pounds, £) by collecting resource use and multiplying by country-specific unit costs. National annual costs were estimated using national disease prevalence from the European cancer registry and other published data. Forty-nine centres provided data on 558 patients (58.2% aged <65 years, 53.6% stage IIIB disease at diagnosis). The mean follow-up duration was 27 months (France), 26 months (Germany) and 22 months (UK). The mean total direct cost per patient during follow-up was €23 582 in France, €32 058 in Germany and €37 970 (£31 123) in the UK. The largest cost drivers were melanoma drugs [mean €14 004, €21 269, €29 750 (£24 385), respectively] and hospitalization/emergency treatment [mean: €6634, €6950, €3449 (£2827), respectively]. The total mean indirect costs per patient were €129 (France), €4,441 (Germany) and €1712 (£1427) (UK). Estimates for annual national direct cost were €13.1 million (France), €30.2 million (Germany) and €27.8 (£22.8) million (UK). The economic burden of stage IIIB/IIIC melanoma with macroscopic lymph node involvement was substantial in all three countries. Total direct costs were the highest during the period with distant metastasis/terminal illness.

Treatment patterns and outcomes of Stage IIIB/IIIC melanoma in France, Germany and the UK: A retrospective and prospective observational study (MELABIS).

AIM: Real-world data on treatment patterns/outcomes in patients with advanced melanoma, while scarce, are useful for health technology assessments that govern patient access in many countries. We collected retrospective data on treatment patterns among patients in France, Germany and the UK with Stage IIIB/IIIC melanoma with macroscopic lymph node involvement, whose primary melanoma and regional lymph node metastases had been completely resected. METHODS: Patients ≥18 years were diagnosed between 1 January 2009 and 31 December 2011. Data were obtained from patients' medical records and a patient survey. RESULTS: Forty-nine centres provided data on 558 patients: 53.6% had Stage IIIB disease; 58.2% were of working age (<65 years), 22.5% reported a change in employment status due to melanoma, 8% were on long-term sick leave; and 35.1% were deceased over the study period. Overall median distant metastases-free survival was 23.4 months and median disease-free survival was 13.3 months. Hospitalisation frequency increased during distant metastatic/terminal disease phases. Adjuvant therapy was received by 7.0% (14/199) of patients in France, 2.6% (5/195) in the UK, and 33.5% (55/164) in Germany. Low-dose interferon was used more frequently than other regimens. High-dose interferon was associated with discontinuation in 28.6% and dose delay/reduction in 33.3% of patients. CONCLUSIONS: Rapid disease progression combined with increased use of healthcare resources in later phases of disease result in a high burden-of-illness for patients and healthcare providers. The use of adjuvant interferon therapy varies considerably in this population in the countries studied, highlighting the need for improved treatments for melanoma.