RESUMEN
PURPOSE: Perioperative decision making for large (> 2 cm) rectal polyps with ambiguous features is complex. The most common intraprocedural assessment is clinician judgement alone while radiological and endoscopic biopsy can provide periprocedural detail. Fluorescence-augmented machine learning (FA-ML) methods may optimise local treatment strategy. METHODS: Surgeons of varying grades, all performing colonoscopies independently, were asked to visually judge endoscopic videos of large benign and early-stage malignant (potentially suitable for local excision) rectal lesions on an interactive video platform (Mindstamp) with results compared with and between final pathology, radiology and a novel FA-ML classifier. Statistical analyses of data used Fleiss Multi-rater Kappa scoring, Spearman Coefficient and Frequency tables. RESULTS: Thirty-two surgeons judged 14 ambiguous polyp videos (7 benign, 7 malignant). In all cancers, initial endoscopic biopsy had yielded false-negative results. Five of each lesion type had had a pre-excision MRI with a 60% false-positive malignancy prediction in benign lesions and a 60% over-staging and 40% equivocal rate in cancers. Average clinical visual cancer judgement accuracy was 49% (with only 'fair' inter-rater agreement), many reporting uncertainty and higher reported decision confidence did not correspond to higher accuracy. This compared to 86% ML accuracy. Size was misjudged visually by a mean of 20% with polyp size underestimated in 4/6 and overestimated in 2/6. Subjective narratives regarding decision-making requested for 7/14 lesions revealed wide rationale variation between participants. CONCLUSION: Current available clinical means of ambiguous rectal lesion assessment is suboptimal with wide inter-observer variation. Fluorescence based AI augmentation may advance this field via objective, explainable ML methods.
Asunto(s)
Colonoscopía , Neoplasias del Recto , Humanos , Neoplasias del Recto/patología , Neoplasias del Recto/cirugía , Neoplasias del Recto/diagnóstico por imagen , Pólipos Intestinales/patología , Pólipos Intestinales/cirugía , Aprendizaje Automático , Masculino , Fluorescencia , Femenino , Variaciones Dependientes del ObservadorRESUMEN
CONTEXT: Objective Structured Clinical Examinations (OSCEs) are an increasingly popular evaluation modality for medical students. While the face-to-face interaction allows for more in-depth assessment, it may cause standardization problems. Methods to quantify, limit or adjust for examiner effects are needed. METHODS: Data originated from 3 OSCEs undergone by 900-student classes of 5th- and 6th-year medical students at Université Paris Cité in the 2022-2023 academic year. Sessions had five stations each, and one of the three sessions was scored by consensus by two raters (rather than one). We report OSCEs' longitudinal consistency for one of the classes and staff-related and student variability by session. We also propose a statistical method to adjust for inter-rater variability by deriving a statistical random student effect that accounts for staff-related and station random effects. RESULTS: From the four sessions, a total of 16,910 station scores were collected from 2615 student sessions, with two of the sessions undergone by the same students, and 36, 36, 35 and 20 distinct staff teams in each station for each session. Scores had staff-related heterogeneity (p<10-15), with staff-level standard errors approximately doubled compared to chance. With mixed models, staff-related heterogeneity explained respectively 11.4%, 11.6%, and 4.7% of station score variance (95% confidence intervals, 9.5-13.8, 9.7-14.1, and 3.9-5.8, respectively) with 1, 1 and 2 raters, suggesting a moderating effect of consensus grading. Student random effects explained a small proportion of variance, respectively 8.8%, 11.3%, and 9.6% (8.0-9.7, 10.3-12.4, and 8.7-10.5), and this low amount of signal resulted in student rankings being no more consistent over time with this metric, rather than with average scores (p=0.45). CONCLUSION: Staff variability impacts OSCE scores as much as student variability, and the former can be reduced with dual assessment or adjusted for with mixed models. Both are small compared to unmeasured sources of variability, making them difficult to capture consistently.
Asunto(s)
Competencia Clínica , Evaluación Educacional , Variaciones Dependientes del Observador , Estudiantes de Medicina , Humanos , Evaluación Educacional/métodos , Evaluación Educacional/normas , Competencia Clínica/normas , Educación de Pregrado en Medicina/normas , Paris , Reproducibilidad de los ResultadosRESUMEN
The clinical relevance of rapid eye movement sleep-related obstructive sleep apnea (REM OSA) is supported by its associated adverse health outcomes and impact on optimal treatment strategies. To date, no assessment of REM OSA phenotyping performance has been conducted for any type of sleep testing technology. The objective of this study was to assess this for polysomnography and peripheral arterial tone-based home sleep apnea testing (PAT HSAT). In a dataset comprising 261 participants, the sensitivity and specificity of the agreement on REM OSA phenotyping was assessed for two independent scorings of polysomnography and a synchronously administered PAT HSAT. The sensitivity and specificity of REM OSA phenotyping were 0.87 and 0.89, respectively, for the polysomnography inter-scorer comparison, and 0.68 and 0.97 for the PAT HSAT on a single-night basis, using the conventional minimum required rapid eye movement sleep time of 30 min. Polysomnography-based REM OSA phenotyping was found to be sensitive and specific even for a single-night testing protocol. Peripheral arterial tone-based REM OSA phenotyping showed a lower sensitivity but a slightly higher specificity compared to polysomnography. In order to increase performance and conclusiveness of peripheral arterial tone-based REM OSA phenotyping, a multi-night protocol of 2-5 nights could be considered. Finally, the minimum required rapid eye movement sleep time could be lowered from the conventional 30 min to 15 min without significantly lowering REM OSA phenotyping sensitivity and specificity, while increasing the level of phenotyping conclusiveness.
Asunto(s)
Apnea Obstructiva del Sueño , Humanos , Polisomnografía/métodos , Sensibilidad y Especificidad , Sueño , Apnea Obstructiva del Sueño/diagnóstico , Sueño REMRESUMEN
OBJECTIVE: Automated quantification of infratentorial multiple sclerosis lesions on magnetic resonance imaging is clinically relevant but challenging. To overcome some of these problems, we propose a fully automated lesion segmentation algorithm using 3D convolutional neural networks (CNNs). METHODS: The CNN was trained on a FLAIR image alone or on FLAIR and T1-weighted images from 1809 patients acquired on 156 different scanners. An additional training using an extra class for infratentorial lesions was implemented. Three experienced raters manually annotated three datasets from 123 MS patients from different scanners. RESULTS: The inter-rater sensitivity (SEN) was 80% for supratentorial lesions but only 62% for infratentorial lesions. There was no statistically significant difference between the inter-rater SEN and the SEN of the CNN with respect to the raters. For supratentorial lesions, the CNN featured an intra-rater intra-scanner SEN of 0.97 (R1 = 0.90, R2 = 0.84) and for infratentorial lesion a SEN of 0.93 (R1 = 0.61, R2 = 0.73). CONCLUSION: The performance of the CNN improved significantly for infratentorial lesions when specifically trained on infratentorial lesions using a T1 image as an additional input and matches the detection performance of experienced raters. Furthermore, for infratentorial lesions the CNN was more robust against repeated scans than experienced raters. KEY POINTS: ⢠A 3D convolutional neural network was trained on MRI data from 1809 patients (156 different scanners) for the quantification of supratentorial and infratentorial multiple sclerosis lesions. ⢠Inter-rater variability was higher for infratentorial lesions than for supratentorial lesions. The performance of the 3D convolutional neural network (CNN) improved significantly for infratentorial lesions when specifically trained on infratentorial lesions using a T1 image as an additional input. ⢠The detection performance of the CNN matches the detection performance of experienced raters.
Asunto(s)
Esclerosis Múltiple , Algoritmos , Humanos , Procesamiento de Imagen Asistido por Computador/métodos , Imagen por Resonancia Magnética/métodos , Esclerosis Múltiple/diagnóstico por imagen , Esclerosis Múltiple/patología , Redes Neurales de la ComputaciónRESUMEN
PURPOSE: To investigate and compare human judgment and machine learning tools for quality assessment of clinical MR spectra of brain tumors. METHODS: A very large set of 2574 single voxel spectra with short and long echo time from the eTUMOUR and INTERPRET databases were used for this analysis. Original human quality ratings from these studies as well as new human guidelines were used to train different machine learning algorithms for automatic quality control (AQC) based on various feature extraction methods and classification tools. The performance was compared with variance in human judgment. RESULTS: AQC built using the RUSBoost classifier that combats imbalanced training data performed best. When furnished with a large range of spectral and derived features where the most crucial ones had been selected by the TreeBagger algorithm it showed better specificity (98%) in judging spectra from an independent test-set than previously published methods. Optimal performance was reached with a virtual three-class ranking system. CONCLUSION: Our results suggest that feature space should be relatively large for the case of MR tumor spectra and that three-class labels may be beneficial for AQC. The best AQC algorithm showed a performance in rejecting spectra that was comparable to that of a panel of human expert spectroscopists. Magn Reson Med 79:2500-2510, 2018. © 2017 International Society for Magnetic Resonance in Medicine.
Asunto(s)
Neoplasias Encefálicas/diagnóstico por imagen , Interpretación de Imagen Asistida por Computador/métodos , Aprendizaje Automático , Imagen por Resonancia Magnética/métodos , Algoritmos , Encéfalo/diagnóstico por imagen , Humanos , Control de CalidadRESUMEN
PURPOSE: The aim of this study was to assess the inter-rater variability of the visual interpretation of 11C-PiB PET images regarding the positivity/negativity of amyloid deposition that were obtained in a multicenter clinical research project, Japanese Alzheimer's Disease Neuroimaging Initiative (J-ADNI). The results of visual interpretation were also compared with a semi-automatic quantitative analysis using mean cortical standardized uptake value ratio to the cerebellar cortex (mcSUVR). METHODS: A total of 162 11C-PiB PET scans, including 45 mild Alzheimer's disease, 60 mild cognitive impairment, and 57 normal cognitive control cases that had been acquired as J-ADNI baseline scans were analyzed. Based on visual interpretation by three independent raters followed by consensus read, each case was classified into positive, equivocal, and negative deposition (ternary criteria) and further dichotomized by merging the former two (binary criteria). RESULTS: Complete agreement of visual interpretation by the three raters was observed for 91.3% of the cases (Cohen κ = 0.88 on average) in ternary criteria and for 92.3% (κ = 0.89) in binary criteria. Cases that were interpreted as visually positive in the consensus read showed significantly higher mcSUVR than those visually negative (2.21 ± 0.37 vs. 1.27 ± 0.09, p < 0.001), and positive or negative decision by visual interpretation was dichotomized by a cut-off value of mcSUVR = 1.5. Significant positive/negative associations were observed between mcSUVR and the number of raters who evaluated as positive (ρ = 0.87, p < 0.0001) and negative (ρ = -0.85, p < 0.0001) interpretation. Cases of disagreement among raters showed generally low mcSUVR. CONCLUSIONS: Inter-rater agreement was almost perfect in 11C-PiB PET scans. Positive or negative decision by visual interpretation was dichotomized by a cut-off value of mcSUVR = 1.5. As some cases of disagreement among raters tended to show low mcSUVR, referring to quantitative method may facilitate correct diagnosis when evaluating images of low amyloid deposition.
Asunto(s)
Enfermedad de Alzheimer/diagnóstico por imagen , Benzotiazoles , Interpretación de Imagen Asistida por Computador/métodos , Neuroimagen , Tomografía de Emisión de Positrones , Compuestos de Anilina , Consenso , Femenino , Humanos , Masculino , Persona de Mediana Edad , Variaciones Dependientes del Observador , TiazolesRESUMEN
BACKGROUND: Poor inter-rater reliability in chest radiograph interpretation has been reported in the context of acute respiratory distress syndrome (ARDS), although not for the Berlin definition of ARDS. We sought to examine the effect of training material on the accuracy and consistency of intensivists' chest radiograph interpretations for ARDS diagnosis. METHODS: We conducted a rater agreement study in which 286 intensivists (residents 41.3%, junior attending physicians 35.3%, and senior attending physician 23.4%) independently reviewed the same 12 chest radiographs developed by the ARDS Definition Task Force ("the panel") before and after training. Radiographic diagnoses by the panel were classified into the consistent (n = 4), equivocal (n = 4), and inconsistent (n = 4) categories and were used as a reference. The 1.5-hour training course attended by all 286 intensivists included introduction of the diagnostic rationale, and a subsequent in-depth discussion to reach consensus for all 12 radiographs. RESULTS: Overall diagnostic accuracy, which was defined as the percentage of chest radiographs that were interpreted correctly, improved but remained poor after training (42.0 ± 14.8% before training vs. 55.3 ± 23.4% after training, p < 0.001). Diagnostic sensitivity and specificity improved after training for all diagnostic categories (p < 0.001), with the exception of specificity for the equivocal category (p = 0.883). Diagnostic accuracy was higher for the consistent category than for the inconsistent and equivocal categories (p < 0.001). Comparisons of pre-training and post-training results revealed that inter-rater agreement was poor and did not improve after training, as assessed by overall agreement (0.450 ± 0.406 vs. 0.461 ± 0.575, p = 0.792), Fleiss's kappa (0.133 ± 0.575 vs. 0.178 ± 0.710, p = 0.405), and intraclass correlation coefficient (ICC; 0.219 vs. 0.276, p = 0.470). CONCLUSIONS: The radiographic diagnostic accuracy and inter-rater agreement were poor when the Berlin radiographic definition was used, and were not significantly improved by the training set of chest radiographs developed by the ARDS Definition Task Force. TRIAL REGISTRATION: The study was registered at ClinicalTrials.gov (registration number NCT01704066 ) on 6 October 2012.
Asunto(s)
Competencia Clínica/normas , Radiografía Torácica/métodos , Síndrome de Dificultad Respiratoria/diagnóstico , Enseñanza/normas , Competencia Clínica/estadística & datos numéricos , Femenino , Humanos , Masculino , Variaciones Dependientes del Observador , Estudios Prospectivos , Radiografía Torácica/estadística & datos numéricos , Reproducibilidad de los Resultados , Síndrome de Dificultad Respiratoria/diagnóstico por imagen , Enseñanza/estadística & datos numéricosRESUMEN
Whenever multiple observers provide ratings, even of the same performance, inter-rater variation is prevalent. The resulting 'idiosyncratic rater variance' is considered to be unusable error of measurement in psychometric models and is a threat to the defensibility of our assessments. Prior studies of inter-rater variation in clinical assessments have used open response formats to gather raters' comments and justifications. This design choice allows participants to use idiosyncratic response styles that could result in a distorted representation of the underlying rater cognition and skew subsequent analyses. In this study we explored rater variability using the structured response format of Q methodology. Physician raters viewed video-recorded clinical performances and provided Mini Clinical Evaluation Exercise (Mini-CEX) assessment ratings through a web-based system. They then shared their assessment impressions by sorting statements that described the most salient aspects of the clinical performance onto a forced quasi-normal distribution ranging from "most consistent with my impression" to "most contrary to my impression". Analysis of the resulting Q-sorts revealed distinct points of view for each performance shared by multiple physicians. The points of view corresponded with the ratings physicians assigned to the performance. Each point of view emphasized different aspects of the performance with either rapport-building and/or medical expertise skills being most salient. It was rare for the points of view to diverge based on disagreements regarding the interpretation of a specific aspect of the performance. As a result, physicians' divergent points of view on a given clinical performance cannot be easily reconciled into a single coherent assessment judgment that is impacted by measurement error. If inter-rater variability does not wholly reflect error of measurement, it is problematic for our current measurement models and poses challenges for how we are to adequately analyze performance assessment ratings.
Asunto(s)
Educación Médica/métodos , Educación Médica/normas , Evaluación Educacional/métodos , Evaluación Educacional/normas , Competencia Clínica , Femenino , Humanos , Masculino , Variaciones Dependientes del Observador , Reproducibilidad de los Resultados , Grabación en VideoRESUMEN
The aim of this study was to assess the success of rhinoplasty by evaluating the inter-rater variability in the light of primary indication as functional or cosmetic. Subjective aesthetic perception was compared with objective facial analysis. 45 rhinoplasty patients were included in the study. 25 had cosmetic plus functional reasons with septal deviation (group 1) and 20 had pure cosmetic reasons without septal deviation (group 2). Preoperatively and 6 months postoperatively, four individuals (patient, surgeon, 2 independent surgeons) rated the aesthetic appearance of the nose with visual analogue scale. Facial photogrammetric analysis was applied. The patient's aesthetic perception score was significantly correlated with the two independent surgeons (p < 0.05) whereas not with the primary surgeons. Regarding the objective parameters, patient's aesthetic perception was significantly correlated with the dorsal alignment in both groups (p < 0.05). General satisfaction score was significantly correlated with the nasal breathing as well as with the aesthetic perception scores in both groups. This correlation was higher for aesthetic perception in group 1 and nasal breathing in group 2. Inter-rater variability of outcome perception was higher in cosmetic patients. Nasal dorsal alignment was the only objective parameter which was correlated with the patient's perception. Patient's perception of outcome has better represented the objective photogrammetric analysis rather than the primary surgeons. An interesting finding was the more significant correlation of general satisfaction with aesthetic perception in the functional group whereas nasal breathing in the cosmetic group.
Asunto(s)
Estética , Satisfacción del Paciente , Fotogrametría , Rinoplastia , Adulto , Femenino , Humanos , Masculino , Tabique Nasal/cirugía , Deformidades Adquiridas Nasales/cirugíaRESUMEN
OBJECTIVES: Sonography of muscle architecture provides physicians and researchers with information about muscle function and muscle-related disorders. Inter-rater reliability is a crucial parameter in daily clinical routines. The aim of this study was to assess the inter-rater reliability of sonographic muscle architecture assessments and quantification of errors that arise from inconsistent probe positioning and image interpretation. METHODS: The medial gastrocnemius muscle of 15 healthy participants was measured with sagittal B-mode ultrasound scans. The muscle thickness, fascicle length, superior pennation angle, and inferior pennation angle were assessed. The participants were examined by 2 investigators. A custom-made foam cast was used for standardized positioning of the probe. To analyze inter-rater reliability, the examinations of both raters were compared. The impact of probe positioning was assessed by comparison of foam cast and freehand scans. Error arising from picture interpretation was assessed by comparing the investigators' analyses of foam cast scans independently. Reliability was expressed as the intraclass correlation coefficient (ICC), inter-rater variability (IRV), Bland-Altman analysis (bias ± limits of agreement [LoA]), and standard error of measurement (SEM). RESULTS: Inter-rater reliability was good overall (ICC, 0.77-0.90; IRV, 9.0%-13.4%; bias ± LoA, 0.2 ± 0.2-1.7 ± 3.0). Superior and inferior pennation angles showed high systematic bias and LoA in all setups, ranging from 2.0° ± 2.2° to 3.4° ± 4.1°. The highest IRV was found for muscle thickness (13.4%). When the probe position was standardized, the SEM for muscle thickness decreased from 0.1 to 0.05 cm. CONCLUSIONS: Sonographic examination of muscle architecture of the medial gastrocnemius has good to high reliability. In contrast to pennation angle measurements, length measurements can be improved by standardization of the probe position.
Asunto(s)
Músculo Esquelético/anatomía & histología , Músculo Esquelético/diagnóstico por imagen , Competencia Profesional/estadística & datos numéricos , Ultrasonografía/estadística & datos numéricos , Adulto , Femenino , Humanos , Masculino , Variaciones Dependientes del Observador , Tamaño de los Órganos , Reproducibilidad de los Resultados , Sensibilidad y EspecificidadRESUMEN
OBJECTIVE: We published a list of "must-know" routine EEG (rEEG) findings for trainees based on expert opinion. Here, we studied the accuracy and inter-rater agreement (IRA) of these "must-know" rEEG findings among international experts. METHODS: A previously validated online rEEG examination was disseminated to EEG experts. It consisted of a survey and 30 multiple-choice questions predicated on the previously published "must-know" rEEG findings divided into four domains: normal, abnormal, normal variants, and artifacts. Questions contained de-identified 10-20-s epochs of EEG that were considered unequivocal examples by five EEG experts. RESULTS: The examination was completed by 258 international EEG experts. Overall mean accuracy and IRA (AC1) were 81% and substantial (0.632), respectively. The domain-specific mean accuracies and IRA were: 76%, moderate (0.558) (normal); 78%, moderate (0.575) (abnormal); 85%, substantial (0.678) (normal variants); 85%, substantial (0.740) (artifacts). Academic experts had a higher accuracy than private practice experts (82% vs. 77%; p = .035). Country-specific overall mean accuracies and IRA were: 92%, almost perfect (0.836) (U.S.); 86%, substantial (0.762) (Brazil); 79%, substantial (0.646) (Italy); and 72%, moderate (0.496) (India). In conclusion, collective expert accuracy and IRA of "must-know" rEEG findings are suboptimal and heterogeneous. SIGNIFICANCE: We recommend the development and implementation of pragmatic, accessible, country-specific ways to measure and improve the expert accuracy and IRA.
Asunto(s)
Electroencefalografía , Neurología , Adulto , Niño , Humanos , Variaciones Dependientes del Observador , Artefactos , ItaliaRESUMEN
BACKGROUND: Lung ultrasonography (LUS) is a non-invasive imaging method used to diagnose and monitor conditions such as pulmonary edema, pneumonia, and pneumothorax. It is precious where other imaging techniques like CT scan or chest X-rays are of limited access, especially in low- and middle-income countries with reduced resources. Furthermore, LUS reduces radiation exposure and its related blood cancer adverse events, which is particularly relevant in children and young subjects. The score obtained with LUS allows semi-quantification of regional loss of aeration, and it can provide a valuable and reliable assessment of the severity of most respiratory diseases. However, inter-observer reliability of the score has never been systematically assessed. This study aims to assess experienced LUS operators' agreement on a sample of video clips showing predefined findings. METHODS: Twenty-five anonymized video clips comprehensively depicting the different values of LUS score were shown to renowned LUS experts blinded to patients' clinical data and the study's aims using an online form. Clips were acquired from five different ultrasound machines. Fleiss-Cohen weighted kappa was used to evaluate experts' agreement. RESULTS: Over a period of 3 months, 20 experienced operators completed the assessment. Most worked in the ICU (10), ED (6), HDU (2), cardiology ward (1), or obstetric/gynecology department (1). The proportional LUS score mean was 15.3 (SD 1.6). Inter-rater agreement varied: 6 clips had full agreement, 3 had 19 out of 20 raters agreeing, and 3 had 18 agreeing, while the remaining 13 had 17 or fewer people agreeing on the assigned score. Scores 0 and score 3 were more reproducible than scores 1 and 2. Fleiss' Kappa for overall answers was 0.87 (95% CI 0.815-0.931, p < 0.001). CONCLUSIONS: The inter-rater agreement between experienced LUS operators is very high, although not perfect. The strong agreement and the small variance enable us to say that a 20% tolerance around a measured value of a LUS score is a reliable estimate of the patient's true LUS score, resulting in reduced variability in score interpretation and greater confidence in its clinical use.
RESUMEN
BACKGROUND: Neonatal pain assessment (NPA) represents a huge global problem of essential importance, as a timely and accurate assessment of neonatal pain is indispensable for implementing pain management. PURPOSE: To investigate the consistency of pain scores derived through video-based NPA (VB-NPA) and on-site NPA (OS-NPA), providing the scientific foundation and feasibility of adopting VB-NPA results in a real-world scenario as the gold standard for neonatal pain in clinical studies and labels for artificial intelligence (AI)-based NPA (AI-NPA) applications. SETTING: A total of 598 neonates were recruited from a pediatric hospital in China. METHODS: This observational study recorded 598 neonates who underwent one of 10 painful procedures, including arterial blood sampling, heel blood sampling, fingertip blood sampling, intravenous injection, subcutaneous injection, peripheral intravenous cannulation, nasopharyngeal suctioning, retention enema, adhesive removal, and wound dressing. Two experienced nurses performed OS-NPA and VB-NPA at a 10-day interval through double-blind scoring using the Neonatal Infant Pain Scale to evaluate the pain level of the neonates. Intra-rater and inter-rater reliability were calculated and analyzed, and a paired samples t-test was used to explore the bias and consistency of the assessors' pain scores derived through OS-NPA and VB-NPA. The impact of different label sources was evaluated using three state-of-the-art AI methods trained with labels given by OS-NPA and VB-NPA, respectively. RESULTS: The intra-rater reliability of the same assessor was 0.976-0.983 across different times, as measured by the intraclass correlation coefficient. The inter-rater reliability was 0.983 for single measures and 0.992 for average measures. No significant differences were observed between the OS-NPA scores and the assessment of an independent VB-NPA assessor. The different label sources only caused a limited accuracy loss of 0.022-0.044 for the three AI methods. CONCLUSION: VB-NPA in a real-world scenario is an effective way to assess neonatal pain due to its high intra-rater and inter-rater reliability compared to OS-NPA and could be used for the labeling of large-scale NPA video databases for clinical studies and AI training.
RESUMEN
Accurate segmentation of liver and tumor regions in medical imaging is crucial for the diagnosis, treatment, and monitoring of hepatocellular carcinoma (HCC) patients. However, manual segmentation is time-consuming and subject to inter- and intra-rater variability. Therefore, automated methods are necessary but require rigorous validation of high-quality segmentations based on a consensus of raters. To address the need for reliable and comprehensive data in this domain, we present LiverHccSeg, a dataset that provides liver and tumor segmentations on multiphasic contrast-enhanced magnetic resonance imaging from two board-approved abdominal radiologists, along with an analysis of inter-rater agreement. LiverHccSeg provides a curated resource for liver and HCC tumor segmentation tasks. The dataset includes a scientific reading and co-registered contrast-enhanced multiphasic magnetic resonance imaging (MRI) scans with corresponding manual segmentations by two board-approved abdominal radiologists and relevant metadata and offers researchers a comprehensive foundation for external validation, and benchmarking of liver and tumor segmentation algorithms. The dataset also provides an analysis of the agreement between the two sets of liver and tumor segmentations. Through the calculation of appropriate segmentation metrics, we provide insights into the consistency and variability in liver and tumor segmentations among the radiologists. A total of 17 cases were included for liver segmentation and 14 cases for HCC tumor segmentation. Liver segmentations demonstrates high segmentation agreement (mean Dice, 0.95 ± 0.01 [standard deviation]) and HCC tumor segmentations showed higher variation (mean Dice, 0.85 ± 0.16 [standard deviation]). The applications of LiverHccSeg can be manifold, ranging from testing machine learning algorithms on public external data to radiomic feature analyses. Leveraging the inter-rater agreement analysis within the dataset, researchers can investigate the impact of variability on segmentation performance and explore methods to enhance the accuracy and robustness of liver and tumor segmentation algorithms in HCC patients. By making this dataset publicly available, LiverHccSeg aims to foster collaborations, facilitate innovative solutions, and ultimately improve patient outcomes in the diagnosis and treatment of HCC.
RESUMEN
BACKGROUND: Despite its wide use in dementia diagnosis on the basis of cut-off points, the inter-rater variability of the Addenbrooke's Cognitive Examination-Third Edition (ACE-III) has been poorly studied. METHODS: Thirty-one healthcare professionals from an older adults' mental health team scored two ACE-III protocols based on mock patients in a computerised form. Scoring accuracy, as well as total and domain-specific scoring variability, were calculated; factors relevant to participants were obtained, including their level of experience and self-rated confidence administering the ACE-III. RESULTS: There was considerable inter-rater variability (up to 18 points for one of the cases), and one case's mean score was significantly higher (by nearly four points) than the true score. The Fluency, Visuospatial and Attention domains had greater levels of variability than Language and Memory. Higher scoring accuracy was not associated with either greater levels of experience or higher self-confidence in administering the ACE-III. CONCLUSIONS: The results suggest that the ACE-III is susceptible to scoring error and considerable inter-rater variability, which highlights the critical importance of initial, and continued, administration and scoring training.
RESUMEN
Purpose: An accurate zonal segmentation of the prostate is required for prostate cancer (PCa) management with MRI. Approach: The aim of this work is to present UFNet, a deep learning-based method for automatic zonal segmentation of the prostate from T2-weighted (T2w) MRI. It takes into account the image anisotropy, includes both spatial and channelwise attention mechanisms and uses loss functions to enforce prostate partition. The method was applied on a private multicentric three-dimensional T2w MRI dataset and on the public two-dimensional T2w MRI dataset ProstateX. To assess the model performance, the structures segmented by the algorithm on the private dataset were compared with those obtained by seven radiologists of various experience levels. Results: On the private dataset, we obtained a Dice score (DSC) of 93.90 ± 2.85 for the whole gland (WG), 91.00 ± 4.34 for the transition zone (TZ), and 79.08 ± 7.08 for the peripheral zone (PZ). Results were significantly better than other compared networks' ( p - value < 0.05 ). On ProstateX, we obtained a DSC of 90.90 ± 2.94 for WG, 86.84 ± 4.33 for TZ, and 78.40 ± 7.31 for PZ. These results are similar to state-of-the art results and, on the private dataset, are coherent with those obtained by radiologists. Zonal locations and sectorial positions of lesions annotated by radiologists were also preserved. Conclusions: Deep learning-based methods can provide an accurate zonal segmentation of the prostate leading to a consistent zonal location and sectorial position of lesions, and therefore can be used as a helping tool for PCa diagnosis.
RESUMEN
OBJECTIVE: To assess the inter-rater reliability of MScanFit MUNE using a "Round Robin" research design. METHODS: Twelve raters from different centres examined six healthy study participants over two days. Median, ulnar and common peroneal nerves were stimulated, and compound muscle action potential (CMAP)-scans were recorded from abductor pollicis brevis (APB), abductor digiti minimi (ADM) and anterior tibial (TA) muscles respectively. From this we calculated the Motor Unit Number Estimation (MUNE) and "A50", a motor unit size parameter. As statistical analysis we used the measures Limits of Agreement (LOA) and Coefficient of Variation (COV). Study participants scored their perception of pain from the examinations on a rating scale from 0 (no pain) to 10 (unbearable pain). RESULTS: Before this study, 41.6% of the raters had performed MScanFit less than five times. The mean MUNE-values were: 99.6 (APB), 131.4 (ADM) and 126.2 (TA), with LOA: 19.5 (APB), 29.8 (ADM) and 20.7 (TA), and COV: 13.4 (APB), 6.3 (ADM) and 5.6 (TA). MUNE-values correlated to CMAP max amplitudes (R2-values were: 0.463 (APB) (p<0.001), 0.421 (ADM) (p<0.001) and 0.645 (TA) (p<0.001)). The average perception of pain was 4. DISCUSSION: MScanFit indicates a high level of inter-rater reliability, even with only limited rater experience and is overall reasonably well tolerated by patients. These results may indicate MScanFit as a reliable MUNE method with potential as a biomarker in drug trials.
Asunto(s)
Esclerosis Amiotrófica Lateral , Neuronas Motoras , Potenciales de Acción/fisiología , Electromiografía/métodos , Humanos , Neuronas Motoras/fisiología , Músculo Esquelético/inervación , Dolor , Reproducibilidad de los ResultadosRESUMEN
PURPOSE: Frailty is a valuable predictor for outcome in elderly ICU patients, and has been suggested to be used in various decision-making processes prior to and during an ICU admission. There are many instruments developed to assess frailty, but few of them can be used in emergency situations. In this setting the clinical frailty scale (CFS) is frequently used. The present study is a sub-study within a larger outcome study of elderly ICU patients in Europe (the VIP-2 study) in order to document the reliability of the CFS. MATERIALS AND METHODS: From the VIP-2 study, 129 ICUs in 20 countries participated in this sub-study. The patients were acute admissions ≥ 80 years of age and frailty was assessed at admission by two independent observers using the CFS. Information was obtained from the patient, if not feasible, from the family/caregivers or from hospital files. The profession of the rater and source of data were recorded along with the score. Interrater variability was calculated using linear weighted kappa analysis. RESULTS: 1923 pairs of assessors were included and background data of patients were similar to the whole cohort (n = 3920). We found a very high inter-rater agreement (weighted kappa 0.86), also in subgroup analyses. The agreement when comparing information from family or hospital records was better than using only direct patient information, and pairs of raters from same profession performed better than from different professions. CONCLUSIONS: Overall, we documented a high reliability using CFS in this setting. This frailty score could be used more frequently in elderly ICU patients in order to create a more holistic and realistic impression of the patient´s condition prior to ICU admission.
RESUMEN
PURPOSE: Definitions of acute respiratory distress syndrome (ARDS) include radiographic criteria, but there are concerns about reliability and prognostic relevance. This study aimed to evaluate the independent relationship between chest imaging and mortality and examine the inter-rater variability of interpretations of chest radiographs (CXR) in pediatric ARDS (PARDS). METHODS: Prospective, international observational study in children meeting Pediatric Acute Lung Injury Consensus Conference (PALICC) criteria for PARDS, which requires new infiltrate(s) consistent with pulmonary parenchymal disease, without mandating bilateral infiltrates. Mortality analysis focused on the entire cohort, whereas inter-observer variability used a subset of patients with blinded, simultaneous interpretation of CXRs by intensivists and radiologists. RESULTS: Bilateral infiltrates and four quadrants of alveolar consolidation were associated with mortality on a univariable basis, using CXRs from 708 patients with PARDS. For patients on either invasive (IMV) or non-invasive ventilation (NIV) with PaO2/FiO2 (PF) ratios (or SpO2/FiO2 (SF) ratio equivalent) > 100, neither bilateral infiltrates (OR 1.3 (95% CI 0.68, 2.5), p = 0.43), nor 4 quadrants of alveolar consolidation (OR 1.6 (0.85, 3), p = 0.14) were associated with mortality. For patients with PF ≤ 100, bilateral infiltrates (OR 3.6 (1.4, 9.4), p = 0.01) and four quadrants of consolidation (OR 2.0 (1.14, 3.5), p = 0.02) were associated with higher mortality. A subset of 702 CXRs from 233 patients had simultaneous interpretations. Interobserver agreement for bilateral infiltrates and quadrants was "slight" (kappa 0.31 and 0.33). Subgroup analysis showed agreement did not differ when stratified by PARDS severity but was slightly higher for children with chronic respiratory support (kappa 0.62), NIV at PARDS diagnosis (kappa 0.53), age > 10 years (kappa 0.43) and fluid balance > 40 ml/kg (kappa 0.48). CONCLUSION: Bilateral infiltrates and quadrants of alveolar consolidation are associated with mortality only for those with PF ratio ≤ 100, although there is high- inter-rater variability in these chest-x ray parameters.
Asunto(s)
Síndrome de Dificultad Respiratoria , Niño , Humanos , Incidencia , Pronóstico , Estudios Prospectivos , Reproducibilidad de los Resultados , Síndrome de Dificultad Respiratoria/diagnóstico por imagenRESUMEN
OBJECTIVE: To present and test a production-matching method with external references, looking at the improvement of inter-rater variability of expert evaluations. METHOD: It consists of adjusting quality attribute levels of a synthetic vowel for a simultaneous matching with the natural patient vowel (NPV) attributes. In an initial experiment, seven speech-language pathology (SLP) experts performed this task with the new method and evaluated the same NPV with the standard method. Targets were twelve NPVs with a variety of quality attribute combinations. In a second experiment, we employed the proposed method to assess the evaluation performance of 65 SLP students. RESULTS: Expert evaluations show less dispersion for the proposed method than those obtained using the standard rating method. Student individual responses were compared with overall responses from their own group and were cross referenced with expert responses. A Kappa index is proposed as a measure of SLP students' performance. CONCLUSIONS: The proposed method was readily accepted by both SLP experts and students. Experts' consensus was improved. SLP students could benefit by quickly learning to discriminate complex attributes, which usually demands years of experience.