Methodological quality of 100 recent systematic reviews of health-related outcome measurement instruments: an overview of reviews.

PURPOSE: Systematic reviews evaluating and comparing the measurement properties of outcome measurement instruments (OMIs) play an important role in OMI selection. Earlier overviews of review quality (2007, 2014) evidenced substantial concerns with regards to alignment to scientific standards. This overview aimed to investigate whether the quality of recent systematic reviews of OMIs lives up to the current scientific standards. METHODS: One hundred systematic reviews of OMIs published from June 1, 2021 onwards were randomly selected through a systematic literature search performed on March 17, 2022 in MEDLINE and EMBASE. The quality of systematic reviews was appraised by two independent reviewers. An updated data extraction form was informed by the earlier studies, and results were compared to these earlier studies' findings. RESULTS: A quarter of the reviews had an unclear research question or aim, and in 22% of the reviews the search strategy did not match the aim. Half of the reviews had an incomprehensive search strategy, because relevant search terms were not included. In 63% of the reviews (compared to 41% in 2014 and 30% in 2007) a risk of bias assessment was conducted. In 73% of the reviews (some) measurement properties were evaluated (58% in 2014 and 55% in 2007). In 60% of the reviews the data were (partly) synthesized (42% in 2014 and 7% in 2007); evaluation of measurement properties and data syntheses was not conducted separately for subscales in the majority. Certainty assessments of the quality of the total body of evidence were conducted in only 33% of reviews (not assessed in 2014 and 2007). The majority (58%) did not make any recommendations on which OMI (not) to use. CONCLUSION: Despite clear improvements in risk of bias assessments, measurement property evaluation and data synthesis, specifying the research question, conducting the search strategy and performing a certainty assessment remain poor. To ensure that systematic reviews of OMIs meet current scientific standards, more consistent conduct and reporting of systematic reviews of OMIs is needed.

A Video-Based Procedure-Specific Competency Assessment Tool for Minimally Invasive Esophagectomy.

Importance: Minimally invasive esophagectomy (MIE) is a complex procedure with substantial learning curves. In other complex minimally invasive procedures, suboptimal surgical performance has convincingly been associated with less favorable patient outcomes as assessed by peer review of the surgical procedure. Objective: To develop and validate a procedure-specific competency assessment tool (CAT) for MIE. Design, Setting, and Participants: In this international quality improvement study, a procedure-specific MIE-CAT was developed and validated. The MIE-CAT contains 8 procedural phases, and 4 quality components per phase are scored with a Likert scale ranging from 1 to 4. For evaluation of the MIE-CAT, intraoperative MIE videos performed by a single surgical team in the Esophageal Center East Netherlands were peer reviewed by 18 independent international MIE experts (with more than 120 MIEs performed). Each video was assessed by 2 or 3 blinded experts to evaluate feasibility, content validity, reliability, and construct validity. MIE-CAT version 2 was composed with refined content aimed at improving interrater reliability. A total of 32 full-length MIE videos from patients who underwent MIE between 2011 and 2020 were analyzed. Data were analyzed from January 2021 to January 2023. Exposure: Performance assessment of transthoracic MIE with an intrathoracic anastomosis. Main Outcomes and Measures: Feasibility, content validity, interrater and intrarater reliability, and construct validity, including correlations with both experience of the surgical team and clinical parameters, of the developed MIE-CAT. Results: Experts found the MIE-CAT easy to understand and easy to use to grade surgical performance. The MIE-CAT demonstrated good intrarater reliability (range of intraclass correlation coefficients [ICCs], 0.807 [95% CI, 0.656 to 0.892] for quality component score to 0.898 [95% CI, 0.846 to 0.932] for phase score). Interrater reliability was moderate (range of ICCs, 0.536 [95% CI, -0.220 to 0.994] for total MIE-CAT score to 0.705 [95% CI, 0.473 to 0.846] for quality component score), and most discrepancies originated in the lymphadenectomy phases. Hypothesis testing for construct validity showed more than 75% of hypotheses correct: MIE-CAT performance scores correlated with experience of the surgical team (r = 0.288 to 0.622), blood loss (r = -0.034 to -0.545), operative time (r = -0.309 to -0.611), intraoperative complications (r = -0.052 to -0.319), and severe postoperative complications (r = -0.207 to -0.395). MIE-CAT version 2 increased usability. Interrater reliability improved but remained moderate (range of ICCs, 0.666 to 0.743), and most discrepancies between raters remained in the lymphadenectomy phases. Conclusions and Relevance: The MIE-CAT was developed and its feasibility, content validity, reliability, and construct validity were demonstrated. By providing insight into surgical performance of MIE, the MIE-CAT might be used for clinical, training, and research purposes.

Putting ICU triage guidelines into practice: A simulation study using observations and interviews.

BACKGROUND: The COVID-19 pandemic has prompted many countries to formulate guidelines on how to deal with a worst-case scenario in which the number of patients needing intensive care unit (ICU) care exceeds the number of available beds. This study aims to explore the experiences of triage teams when triaging fictitious patients with the Dutch triage guidelines. It provides an overview of the factors that influence decision-making when performing ICU triage with triage guidelines. METHODS: Eight triage teams from four hospitals were given files of fictitious patients needing intensive care and instructed to triage these patients. Sessions were observed and audio-recorded. Four focus group interviews with triage team members were held to reflect on the sessions and the Dutch guidelines. The results were analyzed by inductive content analysis. RESULTS: The Dutch triage guidelines were the main basis for making triage decisions. However, some teams also allowed their own considerations (outside of the guidelines) to play a role when making triage decisions, for example to help avoid using non-medical criteria such as prioritization based on age group. Group processes also played a role in decision-making: triage choices can be influenced by the triagists' opinion on the guidelines and the carefulness with which they are applied. Intensivists, being most experienced in prognostication of critical illness, often had the most decisive role during triage sessions. CONCLUSIONS: Using the Dutch triage guidelines is feasible, but there were some inconsistencies in prioritization between teams that may be undesirable. ICU triage guideline writers should consider which aspects of their criteria might, when applied in practice, lead to inconsistencies or ethically questionable prioritization of patients. Practical training of triage team members in applying the guidelines, including explanation of the rationale underlying the triage criteria, might improve the willingness and ability of triage teams to follow the guidelines closely.

Crowd-sourced and expert video assessment in minimally invasive esophagectomy.

BACKGROUND: Video-based assessment by experts may structurally measure surgical performance using procedure-specific competency assessment tools (CATs). A CAT for minimally invasive esophagectomy (MIE-CAT) was developed and validated previously. However, surgeon's time is scarce and video assessment is time-consuming and labor intensive. This study investigated non-procedure-specific assessment of MIE video clips by MIE experts and crowdsourcing, collective surgical performance evaluation by anonymous and untrained laypeople, to assist procedure-specific expert review. METHODS: Two surgical performance scoring frameworks were used to assess eight MIE videos. First, global performance was assessed with the non-procedure-specific Global Operative Assessment of Laparoscopic Skills (GOALS) of 64 procedural phase-based video clips < 10 min. Each clip was assessed by two MIE experts and > 30 crowd workers. Second, the same experts assessed procedure-specific performance with the MIE-CAT of the corresponding full-length video. Reliability and convergent validity of GOALS for MIE were investigated using hypothesis testing with correlations (experience, blood loss, operative time, and MIE-CAT). RESULTS: Less than 75% of hypothesized correlations between GOALS scores and experience of the surgical team (r < 0.3), blood loss (r = - 0.82 to 0.02), operative time (r = - 0.42 to 0.07), and the MIE-CAT scores (r = - 0.04 to 0.76) were met for both crowd workers and experts. Interestingly, experts' GOALS and MIE-CAT scores correlated strongly (r = 0.40 to 0.79), while crowd workers' GOALS and experts' MIE-CAT scores correlations were weak (r = - 0.04 to 0.49). Expert and crowd worker GOALS scores correlated poorly (ICC ≤ 0.42). CONCLUSION: GOALS assessments by crowd workers lacked convergent validity and showed poor reliability. It is likely that MIE is technically too difficult to assess for laypeople. Convergent validity of GOALS assessments by experts could also not be established. GOALS might not be comprehensive enough to assess detailed MIE performance. However, expert's GOALS and MIE-CAT scores strongly correlated indicating video clip (instead of full-length video) assessments could be useful to shorten assessment time.

Roles of general practitioners in shared decision-making for patients with cancer: A qualitative study.

OBJECTIVE: The shared decision-making (SDM) process for the treatment of pancreatic and oesophageal cancer primarily takes place with healthcare professionals (HCPs) in the hospital setting. This study aims to explore the perspectives of general practitioners (GPs) on their possible roles during this SDM process, their added value and their requirements for involvement in SDM. METHODS: Semi-structured interviews were conducted with 12 GPs about their views on SDM for patients with cancer. The interviews were analysed by two researchers using an inductive open coding approach. RESULTS: Five potential roles in SDM were described by the interviewed GPs, of which the role as 'coach' of the patient was mentioned by all. GPs see their main added value as their long-standing relationship with the patient. To be able to participate optimally in SDM, GPs indicated that they need to be kept up to date during the patient's care process and should receive enough medical information about treatment options and contextual information. CONCLUSION: GPs see different potential roles for themselves when involved in SDM. Hospital HCPs that want to facilitate GP involvement should take the initiative, provide the GPs with enough and timely information and must be easy to consult.

[Using pandemic guidelines in simulated triage sessions. Preliminary observations and lessons learned]. / Pandemiedraaiboeken in gesimuleerde triagesessies.

In Spring 2020, the Netherlands appeared to be poorly prepared for 'code black': a situation in which Intensive Care Unit (ICU) bed shortage would be a reality. Guidelines were developed with medical and non-medical considerations for ICU triage. In recent months, we organized eight simulated triage sessions in four hospitals, as part of a research project. We asked triage teams to prioritize cases of patients in need of ICU care, aided by the guidelines. Data are still being analyzed; here we share some preliminary observations and lessons learned. In our sessions, it was striking how much triage depends on the input of the intensivist, especially when it comes to crucial expertise regarding prognosis and length of stay. Differences in prioritization predominantly depended on different estimations of prognosis and length of stay. The guidelines provide direction, but prioritizing ICU patients can be mentally burdensome for triagists. This requires support and aftercare for triage teams.

Development of a patient-reported outcome measure for patients who have recovered from a subarachnoid hemorrhage: the "questionnaire for the screening of symptoms in aneurysmal subarachnoid hemorrhage" (SOS-SAH).

BACKGROUND: Patients who have been successfully treated for an aneurysmal subarachnoid hemorrhage (aSAH) often retain multiple health complaints, including mood disorders, cognitive complaints, fatigue, and problems with social participation. These problems are not always fully addressed during hospital visits or in current outcome measures, such as the modified Rankin score and the Glasgow Outcome Scale. Here, we present the development of the "Questionnaire for the Screening of Symptoms in aneurysmal Subarachnoid Hemorrhage" (SOS-SAH), which screens for the self-reported symptoms of patients with mild disabilities. METHODS: During the development of the SOS-SAH we adhered to the PROM-cycle framework for the selection and implementation of patient-reported outcome measures (PROMs). The SOS-SAH was developed in an iterative process informed by a literature study. Patients and healthcare professionals were involved in the development process through participating in a working group, interviews, and a cognitive validation study. RESULTS AND CONCLUSIONS: Relevant patient-reported outcomes (PROs) were identified for patients with aSAH. The SOS-SAH was developed primarily using domains and items from existing PROMs and, if necessary, by developing new items. The SOS-SAH consists of 40 items and covers 14 domains: cognitive abilities, hypersensitivity to stimuli, anxiety, depression, fatigue, social roles, personality change, language, vision, taste, smell, hearing, headache, and sexual function. It also includes a proxy measurement for use by family members to assess cognitive functioning and personality change.

Measurement properties of the Dutch-Flemish patient-reported outcomes measurement information system (PROMIS) physical function item bank and instruments: a systematic review.

BACKGROUND: Limitations in physical functioning are a big concern especially for patients with chronic or musculoskeletal diseases. Therefore, physical functioning is often used as a core outcome of treatments. The generic patient-reported outcomes information system (PROMIS) physical function (PF) item bank has shown potential to measure PF with better precision, interpretability and lower respondent burden compared with traditional patient-reported outcome measures. This study provides an overview of the current evidence on the quality of the measurement properties of the translated Dutch-Flemish PROMIS-PF item bank and its subdomains, and their derived short forms and computer adaptive tests (CATs). METHODS: PubMed was searched up to June 17th 2020 for validation studies of Dutch-Flemish PROMIS-PF in Dutch and Flemish adults. Quality assessment of the included studies was conducted using the COSMIN Risk of bias checklist. The COSMIN criteria for good measurement properties were used to judge the results of the studies, which were adjusted and added to where needed for this review, in the context of IRT instruments and item banks. The quality of evidence was summarized for each measurement property based on the Grading of Recommendation Assessment, Development, and Evaluation (GRADE) approach. RESULTS: Eleven studies were included, evaluating the PROMIS-PF item bank, the Upper Extremity (UE) subdomain, and/or their derived short forms and CATs in different clinical populations. There is evidence for sufficient structural validity, measurement precision, construct validity, and cross-cultural validity of the Dutch-Flemish PROMIS-PF item bank. The upper extremity subdomain item bank shows high quality evidence for structural validity and measurement precision. Content validity of these item banks has not been thoroughly demonstrated in a Dutch-Flemish population. Furthermore, the derived instruments have far less robust evidence: there are fewer validation studies available and none examined their performance as stand-alone administered instruments. CONCLUSIONS: The first studies into the Dutch-Flemish PROMIS-PF item bank and the UE subdomain show promising results, with especially high quality evidence for sufficient structural validity and measurement precision. However, more studies, and with higher methodological quality, are needed to study the instruments derived from these item banks. These studies should also evaluate content validity, reliability and responsiveness.

Translation and validation of two disease-specific patient-reported outcome measures (Bladder Cancer Index and FACT-Bl-Cys) in Dutch bladder cancer patients.

BACKGROUND: The Bladder Cancer Index (BCI) and Functional Assessment of Cancer Therapy-Bladder-Cystectomy (FACT-Bl-Cys) were developed to measure disease-specific health-related quality of life (HRQOL) in bladder cancer patients and patients treated with radical cystectomy, respectively. Both patient-reported outcome measures (PROMs) are frequently used in clinical practice, but are not yet validated according to the COSMIN criteria and not yet available in Dutch. Therefore, the aim of this study was to translate the BCI and FACT-Bl-Cys into Dutch and to evaluate their measurement properties according to the COSMIN criteria. METHODS: The BCI and FACT-Bl-Cys were translated into Dutch using a forward-backward method, and subsequently administered at baseline (pre-operatively) and 3 months post-operatively in bladder cancer patients who received a radical cystectomy. Validity (content and construct), reliability (internal consistency, test-retest reliability, and measurement error), floor and ceiling effects, and responsiveness were assessed according to the COSMIN criteria. RESULTS: Forward-backward translation encountered no particular linguistic problems. In total 260 patients completed the baseline measurement, while 182 patients completed the three-month measurement. Only a ceiling effect was identified for the BCI. Hypotheses testing for construct validity was satisfying, as 67% and 92% of the hypothesized correlations were confirmed. Structural validity was moderate for both measures, as confirmatory factor analyses showed limited fit. Reliability of both PROMs was good. The intraclass correlation coefficient (ICC) of the BCI domains ranged from 0.47 to 0.93, minimal value of Cronbach's α was 0.70, smallest detectable change on group level (SDC group) ranged from 1.9 to 8.6. The ICC of the FACT-Bl-Cys domains ranged from 0.43 to 0.83, minimal value of Cronbach's α was 0.77, SDC group was around 1. Only the FACT-Bl-Cys total score was found to be responsive to changes in generic quality of life. CONCLUSIONS: The Dutch versions of the BCI and FACT-Bl-Cys were shown to be reliable and have good content validity. Structural validity was limited for both measures. Only the FACT-Bl-Cys total score was responsive to changes in generic HRQOL. Despite some limitations, both PROMs seem suitable for use in clinical practice and research.

Does the Patient-Reported Apnea Questionnaire (PRAQ) increase patient-centredness in the daily practice of sleep centres? a mixed-methods study.

OBJECTIVES: The objective of this exploratory study was to see how the Patient-Reported Apnea Questionnaire (PRAQ) may impact the daily clinical practice of sleep centres, and why it may or may not work as expected. The hypotheses were tested that this patient-reported outcome measure makes patients more aware of which of their health complaints may be related to obstructive sleep apnoea (OSA), and that it improves patient-centredness of care by shifting the focus of care away from (only) medical problems towards the individual burden of disease and quality of life. DESIGN: Mixed methods. The quantitative study (surveys, patient records) was a before-and-after study. SETTING: Three sleep centres in The Netherlands (secondary care). PARTICIPANTS: 27 patients and 14 healthcare professionals were interviewed. 487 patients completed surveys pre-implementation, and 377 patients completed surveys post-implementation of the PRAQ. For the health records, 125 patients were included in the pre-implementation group, and 124 other patients in the post-implementation group. INTERVENTIONS: The PRAQ was used in clinical practice for six successive months. OUTCOME MEASURES: Scores on individual survey items, number of patients receiving non-medical treatment, adjustment of treatment at first follow-up, compliance with treatment. RESULTS: Patients were generally positive about the usefulness of the PRAQ before and during the consultation, as they felt more informed. Healthcare providers did not consider the PRAQ very useful, and they reported minor impact on their consultations. The surveys and health record study did not show an impact of the PRAQ on clinical practice. CONCLUSIONS: Implementing the PRAQ may be beneficial to patients, but this study does not show much impact with regard to patient-centredness of care. New Dutch guidelines for OSA care may lead to a greater emphasis on quality of life and value of care for patients, making its integration in clinical care potentially more useful.

Instrument completion and validation of the patient-reported apnea questionnaire (PRAQ).

BACKGROUND: We previously developed the preliminary version of the Patient-Reported Apnea Questionnaire (PRAQ), a questionnaire measuring health-related quality of life in patients with (suspected) obstructive sleep apnea (OSA). This questionnaire was developed for clinical practice, where it can potentially serve two goals: use on an individual patient level to improve patient care, and use on an aggregate level to measure outcomes for quality improvement at a sleep center. In this study we aim to finalize the PRAQ, make a subselection of items and domains specifically for outcome measurement, and assess the validity, reliability and responsiveness of the PRAQ. METHODS: Patients with suspected OSA were included and asked to complete the PRAQ and additional questionnaires one or more times. The collected data was used to perform the final item selection for clinical practice and for outcome measurement, create the domains for outcome measurement, and assess the measurement properties internal consistency, test-retest reliability, convergent validity and responsiveness. RESULTS: 180 patients were included in the study. The final version of the PRAQ for use in clinical practice contains 40 items and 10 domains. A subselection of 33 items in 5 domains was selected for optimal outcome measurement with the PRAQ. The results for the outcome measurement domains were: Cronbach's alpha 0.88-0.95, ICC 0.81-0.88, and > 75% of hypotheses correct for convergent validity and responsiveness. CONCLUSIONS: The PRAQ shows good measurement properties in patients with (suspected) OSA.

The development of a patient-reported outcome measure for patients with obstructive sleep apnea: the Patient-Reported Apnea Questionnaire (PRAQ).

BACKGROUND: Obstructive sleep apnea (OSA) is a chronic condition that can have a wide range of consequences for a patient's health-related quality of life. Monitoring aspects of quality of life in clinical practice has the potential to improve the patient-centeredness of care for patients with OSA. The aim of this article is to describe the development of the Patient-Reported Apnea Questionnaire (PRAQ), a patient-reported outcome measure (PROM) that is designed for use in clinical practice on an individual patient level, as well as subsequent outcome measurement on an aggregate level. METHODS: We used the items of available PROMs for OSA to create a new PROM with focus on its applicability in clinical practice. We used a tailored development process to come to a selection of domains and items. Patients and healthcare professionals were intensively involved in the development of the PRAQ via membership of the development team, online surveys and focus groups, as well as two rounds of cognitive validation. RESULTS: This first version of the PRAQ consists of 43 items and 10 preliminary domains, and covers the aspects of quality of life that healthcare professionals and patients wish to discuss in clinical practice. Patients indicate that PRAQ is comprehensive and that its length is acceptable. Comprehensive patient involvement has ensured good content validity for the PRAQ. CONCLUSIONS: This article shows how a PROM can be developed with a specific focus on its applicability in clinical practice.

Appraising convergent validity of patient-reported outcome measures in systematic reviews: constructing hypotheses and interpreting outcomes.

PURPOSE: Convergent validity is one type of validity that is commonly assessed for patient-reported outcome measures (PROMs). It is assessed by means of "hypothesis testing": determining whether the scores of the instrument under study correlate with other instruments in the way that one would expect. Authors of systematic reviews on measurement properties for PROMs may encounter validation articles which do not state hypotheses by which convergent validity can be tested. The information in these articles can therefore not be readily used to determine the adequacy of convergent validity. We suggest that in these cases, reviewers construct their own hypotheses. However, constructing hypotheses and interpreting outcomes is not always straightforward, and we wish to aid reviewers based on our own recent experiences with a systematic review on measurement properties. RECOMMENDATIONS: We have the following recommendations for authors of a systematic review on measurement properties who wish to construct hypotheses for convergent validity: take an active role in judging the suitability of the comparator instruments of validation articles; be transparent about which hypotheses were constructed, the underlying assumptions on which they are based, and whether they were constructed by the authors of the validation article or by the reviewer; discuss unmet hypotheses, especially if convergent validity is judged to be inadequate; and when synthesizing data, add up the results of all hypotheses for one instrument, rather than judging convergent validity per study.

Measurement properties of patient-reported outcome measures (PROMs) in adults with obstructive sleep apnea (OSA): A systematic review.

This systematic review summarizes the evidence regarding the quality of patient-reported outcome measures (PROMs) validated in patients with obstructive sleep apnea (OSA). We performed a systematic literature search of all PROMs validated in patients with OSA, and found 22 measures meeting our inclusion criteria. The quality of the studies was assessed using the consensus-based standards for the selection of health status measurement instruments (COSMIN) checklist. The results showed that most of the measurement properties of the PROMs were not, or not adequately, assessed. For many identified PROMs there was no involvement of patients with OSA during their development or before the PROM was tested in patients with OSA. Positive exceptions and the best current candidates for assessing health status in patients with OSA are the sleep apnea quality of life index (SAQLI), Maugeri obstructive sleep apnea syndrome (MOSAS) questionnaire, Quebec sleep questionnaire (QSQ) and the obstructive sleep apnea patient-oriented severity index (OSAPOSI). Even though there is not enough evidence to fully judge the quality of these PROMs as outcome measure, when interpreted with caution, they have the potential to add value to clinical research and clinical practice in evaluating aspects of health status that are important to patients.