Content Validation and Evaluation of an Endovascular Teamwork Assessment Tool.

OBJECTIVE/BACKGROUND: To modify, content validate, and evaluate a teamwork assessment tool for use in endovascular surgery. METHODS: A multistage, multimethod study was conducted. Stage 1 included expert review and modification of the existing Observational Teamwork Assessment for Surgery (OTAS) tool. Stage 2 included identification of additional exemplar behaviours contributing to effective teamwork and enhanced patient safety in endovascular surgery (using real-time observation, focus groups, and semistructured interviews of multidisciplinary teams). Stage 3 included content validation of exemplar behaviours using expert consensus according to established psychometric recommendations and evaluation of structure, content, feasibility, and usability of the Endovascular Observational Teamwork Assessment Tool (Endo-OTAS) by an expert multidisciplinary panel. Stage 4 included final team expert review of exemplars. RESULTS: OTAS core team behaviours were maintained (communication, coordination, cooperation, leadership team monitoring). Of the 114 OTAS behavioural exemplars, 19 were modified, four removed, and 39 additional endovascular-specific behaviours identified. Content validation of these 153 exemplar behaviours showed that 113/153 (73.9%) reached the predetermined Item-Content Validity Index rating for teamwork and/or patient safety. After expert team review, 140/153 (91.5%) exemplars were deemed to warrant inclusion in the tool. More than 90% of the expert panel agreed that Endo-OTAS is an appropriate teamwork assessment tool with observable behaviours. Some concerns were noted about the time required to conduct observations and provide performance feedback. CONCLUSION: Endo-OTAS is a novel teamwork assessment tool, with evidence for content validity and relevance to endovascular teams. Endo-OTAS enables systematic objective assessment of the quality of team performance during endovascular procedures.

Innovative tools for quality assessment: integrated quality criteria for review of multiple study designs (ICROMS).

OBJECTIVES: With the aim to facilitate a more comprehensive review process in public health including patient safety, we established a tool that we have termed ICROMS (Integrated quality Criteria for the Review Of Multiple Study designs), which unifies, integrates and refines current quality criteria for a large range of study designs including qualitative research. STUDY DESIGN: Review, pilot testing and expert consensus. METHODS: The tool is the result of an iterative four phase process over two years: 1) gathering of established criteria for assessing controlled, non-controlled and qualitative study designs; 2) pilot testing of a first version in two systematic reviews on behavioural change in infection prevention and control and in antibiotic prescribing; 3) further refinement and adding of additional study designs in the context of the European Centre for Disease Prevention and Control funded project 'Systematic review and evidence-based guidance on organisation of hospital infection control programmes' (SIGHT); 4) scrutiny by the pan-European expert panel of the SIGHT project, which had the objective of ensuring robustness of the systematic review. RESULTS: ICROMS includes established quality criteria for randomised studies, controlled before-and-after studies and interrupted time series, and incorporates criteria for non-controlled before-and-after studies, cohort studies and qualitative studies. The tool consists of two parts: 1) a list of quality criteria specific for each study design, as well as criteria applicable across all study designs by using a scoring system; 2) a 'decision matrix', which specifies the robustness of the study by identifying minimum requirements according to the study type and the relevance of the study to the review question. The decision matrix directly determines inclusion or exclusion of a study in the review. ICROMS was applied to a series of systematic reviews to test its feasibility and usefulness in the appraisal of multiple study designs. The tool was applicable across a wide range of study designs and outcome measures. CONCLUSION: ICROMS is a comprehensive yet feasible appraisal of a large range of study designs to be included in systematic reviews addressing behaviour change studies in patient safety and public health. The tool is sufficiently flexible to be applied to a variety of other domains in health-related research. Beyond its application to systematic reviews, we envisage that ICROMS can have a positive effect on researchers to be more rigorous in their study design and more diligent in their reporting.

Decision-making in Colorectal Cancer Tumor Board meetings: results of a prospective observational assessment.

BACKGROUND: The management of colorectal cancer increasingly involves multidisciplinary tumor boards. In cases where these occur, the quality can be variable. Despite this, there are no uniform measures to evaluate them. The aim of this study was to evaluate the performance of colorectal cancer tumor boards, via real-time prospective observation. METHODS: An observational tool, termed Colorectal Multidisciplinary Team Metric for Observation of Decision-Making (cMDT-MODe), was used to assess decision-making in 267 cases. The presentation of case history, radiological and pathological information, as well as contributions to decision making of the various team members were analyzed using descriptive statistics and t-tests. Interobserver agreement was assessed using intraclasscorrelation coefficients. RESULTS: Tumor board meetings lasted 76 min, were attended by approximately 16 specialists each, and reviewed an average of 24 cancer cases (3 min per case review). Regarding the quality of presented information to the team, case history information was rated the highest (mean 4.57), followed by radiological information (mean 4.22) and pathological information (mean 3.81). Regarding each team-member's contribution to discussion, surgeons were scored the highest (mean 4.81), followed by radiologists (mean 4.41) and meeting chairs (mean 4.13)--all team members except the board coordinators were scored highly. Overall scoring reliability was good (0.79). CONCLUSIONS: The cMDT-MODe instrument can be reliably used to prospectively assess decision making in the multidisciplinary management of colorectal patients. By systematically quantifying the quality of a colorectal cancer tumor board, we can identify areas for improving practice so as to optimize decision making for cancer care.

Developing and testing TEAM (Team Evaluation and Assessment Measure), a self-assessment tool to improve cancer multidisciplinary teamwork.

BACKGROUND: Cancer multidisciplinary teams (MDTs) are well established worldwide and are an expensive resource yet no standardised tools exist to measure performance. We aimed to develop and test an MDT self-assessment tool underpinned by literature review and consensus from over 2000 UK MDT members about the "characteristics of an effective MDT." METHODS: Questionnaire items relating to all characteristics of MDTs (particularly Leadership and Chairing; Teamworking and Culture; Patient-centred care; Clinical decision-making process; and Organisation and administration during meetings) were developed by an expert panel. Acceptability, feasibility and psychometric properties were tested by online completion of the questionnaire by 23 MDTs from 4 UK NHS Trusts followed by interviews with 74 team members including members from all teams and nonresponders. 10 of the MDTs also completed questionnaires that directly translated each characteristic to an item (for the five domains above) to test content validity. RESULTS: A total of 47 items were created, each rated for agreement on a 5-point scale. A total of 329 (52 %) of 637 team members completed the questionnaire, including representation from medical, nursing and clerical MDT members. Responses correlated well with domain-specific questionnaires (r > 0.67, p = 0.01), most domain-scales had acceptable internal consistency (Cronbach alpha > 0.60), and good item discrimination (majority of items r < 0.20). Team members were positive about its value. CONCLUSIONS: Self-assessment of team performance using this tool may support MDT development.

Multidisciplinary team working across different tumour types: analysis of a national survey.

BACKGROUND: Using data from a national survey, this study aimed to address whether the current model for multidisciplinary team (MDT) working is appropriate for all tumour types. PATIENTS AND METHODS: Responses to the 2009 National Cancer Action Team national survey were analysed by tumour type. Differences indicate lack of consensus between MDT members in different tumour types. RESULTS: One thousand one hundred and forty-one respondents from breast, gynaecological, colorectal, upper gastrointestinal, urological, head and neck, haematological and lung MDTs were included. One hundred and sixteen of 136 statements demonstrated consensus between respondents in different tumour types. There were no differences regarding the infrastructure for meetings and team governance. Significant consensus was seen for team characteristics, and respondents disagreed regarding certain aspects of meeting organisations and logistics, and patient-centred decision making. Haematology MDT members were outliers in relation to the clinical decision-making process, and lung MDT members disagreed with other tumour types regarding treating patients with advanced disease. CONCLUSIONS: This analysis reveals strong consensus between MDT members from different tumour types, while also identifying areas that require a more tailored approach, such as the clinical decision-making process, and preparation for and the organisation of MDT meetings. Policymakers should remain sensitive to the needs of health care teams working in individual tumour types.

Observational Skill-based Clinical Assessment tool for Resuscitation (OSCAR): development and validation.

AIM: The aim of the study reported here was to address the need to assess and train teamwork and non-technical skills in the context of Resuscitation. Specifically, we sought to develop a tool that is feasible to use and psychometrically sound to assess team behaviours during cardiac arrest resuscitation attempts. METHODS: To ensure validity, reliability, and feasibility, the Observational Skill based Clinical Assessment tool for Resuscitation (OSCAR) was developed in 3 phases. A review of the literature leading to initial tool development was followed by an assessment of face and content validity, and finally a thorough reliability assessment, using Cronbach's α to assess internal consistency and intraclass correlation to assess inter-rater reliability. RESULTS: OSCAR was developed methodically, and tested for face and content validity. Cronbach's α results ranged from 0.736 to 0.965 demonstrating high internal consistency, and intraclass correlation results ranged from 0.652 to 0.911, all of which are strongly significant and indicate good inter-rater reliability. CONCLUSION: On the basis of our results, we conclude that OSCAR is psychometrically robust, scientifically sound, and clinically relevant. We have developed the Observational Skill-based Clinical Assessment tool for Resuscitation (OSCAR) for the assessment of non-technical skills in Resuscitation teams. We propose the use of this tool in simulation and real Cardiac Arrest Resuscitation attempts to assess, guide and train non-technical skills to team members, to improve patient safety and maximise the chances of successful resuscitation.