Radiologists and Clinical Trials: Part 1 The Truth About Reader Disagreements.

The debate over human visual perception and how medical images should be interpreted have persisted since X-rays were the only imaging technique available. Concerns over rates of disagreement between expert image readers are associated with much of the clinical research and at times driven by the belief that any image endpoint variability is problematic. The deeper understanding of the reasons, value, and risk of disagreement are somewhat siloed, leading, at times, to costly and risky approaches, especially in clinical trials. Although artificial intelligence promises some relief from mistakes, its routine application for assessing tumors within cancer trials is still an aspiration. Our consortium of international experts in medical imaging for drug development research, the Pharma Imaging Network for Therapeutics and Diagnostics (PINTAD), tapped the collective knowledge of its members to ground expectations, summarize common reasons for reader discordance, identify what factors can be controlled and which actions are likely to be effective in reducing discordance. Reinforced by an exhaustive literature review, our work defines the forces that shape reader variability. This review article aims to produce a singular authoritative resource outlining reader performance's practical realities within cancer trials, whether they occur within a clinical or an independent central review.

Radiologists and Clinical Trials: Part 2: Practical Statistical Methods for Understanding and Monitoring Independent Reader Performance.

Though many clinical trials rely on medical image evaluations for primary or key secondary endpoints, the methods to monitor reader performance are all too often mired in the legacy use of adjudication rates. If misused, this simple metric can be misleading and sometimes entirely contradictory. Furthermore, attempts to overcome the limitations of adjudication rates using de novo or ad hoc methods often ignore well-established research conducted over the last half-century and can lead to inaccurate conclusions or variable interpretations. Underperforming readers can be missed, expert readers retrained, or worse, replaced. This paper aims to standardize reader performance evaluations using proven statistical methods. Additionally, these methods will describe how to discriminate between scenarios of concern and normal medical interpretation variability. Statistical methods are provided for inter-reader and intra-reader variability and bias, including the adjudicator's bias. Finally, we have compiled guidelines for calculating correct sample sizes, considerations for intra-reader memory recall, and applying alternative designs for independent readers.

Investigator and independent review committee exploratory assessment and verification of tumor response in a non-Hodgkin lymphoma study.

Interpretation of endpoints (e.g. overall response rate) in clinical trials depends on the accurate and reliable measurement and identification of tumors. Regulatory agencies recommend blinded reviews of imaging data by independent review committees (IRCs). Differences in response outcomes that arise between IRCs and site investigators raise regulatory/sponsor concerns. Here, we evaluate discrepant tumor response assessments by the IRC and unblinded investigators (complete versus partial response, respectively) occurring in 52 (13% of 393 IRC-assessed responders) of 447 enrolled patients with treatment-naïve non-Hodgkin lymphoma from a randomized study. The IRC and investigators were 'likely correct' in 73% and 25% of cases, respectively (p < .001). Investigators were more likely to make errors by misinterpreting lymph node data and not utilizing PET results. This post hoc finding suggests a possible role for post-training site evaluation/audit, with retraining as needed, and a specialized consensus committee for concurrent blinded review of site/central data.