The ABR 2021 Radiation Oncology Remote Examinations: Development, Administration, and Implications for the Future.

With the onset of the global coronavirus disease 2019 pandemic in early 2020, it became apparent that routine administration of the ABR Qualifying and Certifying Exams would be disrupted. Initial intent for postponement was later altered to a recognition that replacement of the existing delivery methodologies was essential. Herein, the authors describe the conceptualization, development, administration, and future implications of the new remote examination delivery platforms.

Snapshot of a Specialty: Results of the ABR 2016 Radiation Oncology Clinical Practice Analysis.

PURPOSE: Beginning in 2010, the ABR has administered triennial clinical practice analysis surveys to inform examination development volunteers and staff about the actual state of radiation oncology practice. METHODS AND MATERIALS: As reported here, the 2016 survey was designed to provide objective data regarding actual patient volumes of specific disease sites and subjective insight as to the importance and relevance of site-specific therapy to individual practices. RESULTS: The survey instrument was circulated to 4,075 radiation oncologists listed in the membership database of the American Society for Radiation Oncology, and responses were received from 690 (16.9%); a total of 287 (41.5%) self-identified as being in academic practice. Even in the academic setting, a majority (216 of 287, or 75.3%) indicated that they spend most of their time in clinical practice. CONCLUSIONS: Data from the survey are informative regarding changes in the practice of radiation oncology over the past 6 years.

ABR examinations: the why, what, and how.

The American Board of Radiology (ABR) has provided certification for diagnostic radiologists and other specialists and subspecialists for more than 75 years. The Board certification process is a tangible expression of the social contract between the profession and the public by which the profession enjoys the privilege of self-regulation and the public is assured that it can expect medical professionals to put patients' interests first, guarantees the competence of practitioners, and guards the public health. A primary tool used by the ABR in fulfilling this responsibility is the secure proctored examination. This article sets forth seven standards based on authoritative sources in the field of psychometrics (the science of mental measurements), and explains in each case how the ABR implements that standard. Readers are encouraged to understand that, despite the multiple opinions that may be held, these standards developed over decades by experts using the scientific method should be the central feature in any discussion or critique of examinations given for the privilege of professional practice and for safeguarding the public well-being.

ABR examinations: the why, what, and how.

The American Board of Radiology (ABR) has provided certification for diagnostic radiologists and other specialists and subspecialists for more than 75 years. The Board certification process is a tangible expression of the social contract between the profession and the public by which the profession enjoys the privilege of self-regulation and the public is assured that it can expect medical professionals to put patients' interests first, guarantees the competence of practitioners, and guards the public health. A primary tool used by the ABR in fulfilling this responsibility is the secure proctored examination. This article sets forth seven standards based on authoritative sources in the field of psychometrics (the science of mental measurements), and explains in each case how the ABR implements that standard. Readers are encouraged to understand that, despite the multiple opinions that may be held, these standards developed over decades by experts using the scientific method should be the central feature in any discussion or critique of examinations given for the privilege of professional practice and for safeguarding the public well-being.

Reliability of oral examinations: Radiation oncology certifying examination.

PURPOSE: Oral examinations are used in certifying examinations by many medical specialty boards. They represent daily clinical practice situations more realistically than do written tests or computer-based tests. However, there are repeated concerns in the literature regarding objectivity, fairness, and extraneous factors from interpersonal interactions, item bias, reliability, and validity. In this study, the reliability of oral examination on the radiation oncology certifying examination, which was administered in May of 2010, was analyzed. METHODS AND MATERIALS: One hundred fifty-two candidates rotated though 8 examination stations. Stations consisted of a hotel room equipped with a computer and software that exhibited images appropriate to the content areas. Each candidate had a 25-30 minute face-to-face encounter with an oral examiner who was a content expert in one of the following areas: gastrointestinal, gynecology, genitourinary, lymphoma/leukemia/transplant/myeloma, head/neck/skin, breast, central nervous system/pediatrics, or lung/sarcoma. This type of design is typically referred to as a repeated measures design or a subject by treatment design, although the oral examination was a routine event without any experimental manipulation. RESULTS: The reliability coefficient was obtained by applying Feldt and Charter's simple computational alternative to analysis of variance formulas that yielded KR-20, or Cronbach's coefficient alpha of 0.81. CONCLUSIONS: An experimental design to develop a blueprint in order to improve the consistency of evaluation is suggested.

Practice analysis: a basis for content validity for american board of radiology examinations in diagnostic radiology.

The ABR performs practice analysis every 3 years, according to its strategic plan, in an effort to strengthen the content validity of its qualifying and certifying examinations as well as its maintenance of certification examinations. A nationwide survey of diagnostic radiologists was conducted in July 2010 for the purpose of determining the critically important and frequently performed activities in 12 clinical categories. The survey instrument was distributed electronically to 17,721 members of the ACR, with a unique identification code for each individual. A 5-point scale was established for both frequency and importance variables. The frequency scale ranged from 1 to 5 as follows: 1 = not applicable, 2 = occasionally, 3 = monthly, 4 = weekly, and 5 = daily. The scale for importance also ranged from 1 to 5: 1 = not applicable, 2 = not important, 3 = somewhat important, 4 = important, and 5 = essential. A total of 2,909 diagnostic radiologists (19.32%) participated. Of these, 2,233 (76.76%) indicated that they spent ≥50% of their time in clinical practice. Because of its brevity of the list of the activities, results for the gastrointestinal category are presented in this article. The list of activities weighted according to importance and frequency is presented in this article and, as illustrated, could become the foundation for developing a more detailed blueprint for the gastrointestinal category certifying examinations in diagnostic radiology. Findings on demographic information are also presented.

American Board of Radiology radiation oncology initial qualifying ("written") examinations.

This report was prepared by those who are closely involved in the radiation oncology initial qualification examinations. The primary purpose of this article is to disseminate information concerning test preparation, test administration, scoring, and reporting processes of the ABR. The authors hope that the information contained in the article will be helpful to radiology residents, program directors, and other interested parties.

American Board of Radiology certifying examination: oral versus computer-based format.

OBJECTIVE: This pilot study of a computer-based examination for primary certification by the American Board of Radiology was designed to acquire comparative data on candidates that were measures of individual performance on the oral examination compared with the computer-based examination. MATERIALS AND METHODS: The pilot computer-based pediatric radiology examination was designed by experienced oral board examiners and the pediatric subspecialty trustees. Images were chosen from the examination repository of the American Board of Radiology. The 20-minute examination was designed to include 8-10 cases with 26-31 scorable units covering all aspects of pediatric radiology. RESULTS: Among the 1,317 candidates taking the oral board examination, 1,048 candidates (79.6%) participated in the voluntary pilot examination. The scores of the two examinations were subjected to statistical analysis. The sensitivity and specificity of the pilot examination were 94.5% and 45.7%. The overall accuracy was 92.8%. Seventy-five candidates (7.2%) who participated in this study received different verdicts on the pilot examination and the pediatric radiology category of the oral examination. Fifty-six of these candidates (5.3%) failed the pilot examination but passed in the oral pediatric radiology category; 19 of the candidates (1.8%) passed the pilot examination but failed the oral pediatric radiology test. Pilot examination scores were higher for candidates who passed the oral pediatric radiology category (median score, 80; interquartile range, 74.1-85.2) than for candidates who failed (median score, 65.4; interquartile range, 58.6-71.0) (p < 0.0001). CONCLUSION: The pediatric pilot examination was useful for differentiating passing candidates from failing candidates when the score in the pediatric radiology category of the oral examination was used as the reference standard. The overall accuracy was 92.8%.

American Board of Radiology diagnostic radiology initial qualifying (written) examinations.

The purpose of this article is to inform radiology residents, program directors, and other interested parties of the processes involved in developing, administering, and scoring the American Board of Radiology (ABR) diagnostic radiology qualifying (written) examinations. The residents, once certified, will have a lifelong professional relationship with the ABR. It is important for the ABR to be transparent about the processes it uses to ensure that its examinations are fair, valid, and reliable so that residents and their program directors have accurate information about these high-stakes examinations.

Evaluation of off-the-shelf displays for use in the American Board of Radiology maintenance of certification examination.

PURPOSE: To prospectively compare high-, mid-, and low-resolution off-the-shelf displays currently employed by commercial testing centers, in terms of visibility of lesion features needed to render a diagnostic decision when possible diagnoses are provided in a multiple-choice format during a maintenance of certification (MOC) examination. MATERIALS AND METHODS: The Psychometrics Division of the American Board of Radiology (ABR) approved the studies (human subjects and HIPAA compliant). One study compared 1280 x 1024 displays with 1024 x 768 displays; the second, 1600 x 1200 with 1280 x 1024 displays. Images from nine subspecialties were used. In each study, observers viewed images twice-once on each display. Diagnoses were provided, and observers rated visibility of diagnostic features. RESULTS: Of 7977 data pairs analyzed in study 1, the 1024 and 1280 displays received the same ratings for 5726 data pairs (72% of the time), with the 1024 display receiving a higher rating for 679 data pairs (9% of the time) and the 1280 receiving a higher rating for 1572 data pairs (19% of the time) (P < .0001). When rating differences existed, all subspecialties except nuclear medicine had significantly more high-visibility ratings with the 1280 display. Of 1090 data pairs analyzed in study 2, the 1280 and 1600 displays received the same ratings for 689 data pairs (63% of the time), with the 1280 receiving a higher rating for 162 data pairs (15% of the time) and the 1600 receiving a higher rating for 239 data pairs (22% of the time) (P = .0001). When rating differences existed, only cardiopulmonary and musculoskeletal images had significantly more high-visibility ratings with the 1600 display. CONCLUSION: For the ABR MOC examinations, 1280 x 1024 displays should be used, compared to 1024 x 768 displays; 1600 x 1200 displays may be necessary for some images. Good-quality images must be used on the examinations, so digital rather than digitized film images should be used to ensure high-quality images.