Test-retest and inter-rater reliability of two devices measuring tactile mechanical detection thresholds in healthy adults: Semmes-Weinstein monofilaments and the cutaneous mechanical stimulator.

INTRODUCTION/AIMS: Limitations exist in evaluating mechanical detection thresholds (MDTs) due to a lack of dependable electronic instruments designed to assess Aß fibers and measure MDTs across different body areas. This study aims to evaluate the test-retest and inter-rater reliability of the cutaneous mechanical stimulator (CMS), an electronic tactile stimulator, in quantifying MDTs. METHODS: Using a test-retest design, participants underwent assessments of MDTs using Semmes-Weinstein monofilaments (SWM) and the CMS. This study included 27 healthy volunteers (mean age 24.07 ± 3.76 years). Two raters assessed MDTs using SWM and the CMS at two stimulation sites (the left hand and foot) in two experimental sessions approximately 2 weeks apart. RESULTS: MDTs using SWM and the CMS showed excellent reliability on the hand (intraclass correlation coefficient [ICC] = .84) and foot (ICC = .90). A comparison of results obtained at the two sessions showed that MDTs on the hand displayed good reliability for both SWM (ICC = .63) and the CMS (ICC = .73), whereas MDTs on the foot displayed fair reliability for SWM (ICC = .50) and the CMS (ICC = .42). MDTs exhibited good inter-rater reliability with SWM (ICC = .66) and excellent inter-rater reliability with the CMS (ICC = .82) on the hand, as well as showing fair inter-rater reliability with SWM (ICC = .53) and good inter-rater reliability with the CMS (ICC = .60) on the foot. DISCUSSION: The CMS showed superior inter-rater reliability, indicating its potential as a valuable tool for assessing tactile sensitivity in research and clinical settings.

Reliability of single-lead electrocardiogram interpretation to detect atrial fibrillation: insights from the SAFER feasibility study.

AIMS: Single-lead electrocardiograms (ECGs) can be recorded using widely available devices such as smartwatches and handheld ECG recorders. Such devices have been approved for atrial fibrillation (AF) detection. However, little evidence exists on the reliability of single-lead ECG interpretation. We aimed to assess the level of agreement on detection of AF by independent cardiologists interpreting single-lead ECGs and to identify factors influencing agreement. METHODS AND RESULTS: In a population-based AF screening study, adults aged ≥65 years old recorded four single-lead ECGs per day for 1-4 weeks using a handheld ECG recorder. Electrocardiograms showing signs of possible AF were identified by a nurse, aided by an automated algorithm. These were reviewed by two independent cardiologists who assigned participant- and ECG-level diagnoses. Inter-rater reliability of AF diagnosis was calculated using linear weighted Cohen's kappa (κw). Out of 2141 participants and 162 515 ECGs, only 1843 ECGs from 185 participants were reviewed by both cardiologists. Agreement was moderate: κw = 0.48 (95% confidence interval, 0.37-0.58) at participant level and κw = 0.58 (0.53-0.62) at ECG level. At participant level, agreement was associated with the number of adequate-quality ECGs recorded, with higher agreement in participants who recorded at least 67 adequate-quality ECGs. At ECG level, agreement was associated with ECG quality and whether ECGs exhibited algorithm-identified possible AF. CONCLUSION: Inter-rater reliability of AF diagnosis from single-lead ECGs was found to be moderate in older adults. Strategies to improve reliability might include participant and cardiologist training and designing AF detection programmes to obtain sufficient ECGs for reliable diagnoses.

Inter-Rater and Intra-Rater Agreement in Scoring Severity of Rodent Cardiomyopathy and Relation to Artificial Intelligence-Based Scoring.

We previously developed a computer-assisted image analysis algorithm to detect and quantify the microscopic features of rodent progressive cardiomyopathy (PCM) in rat heart histologic sections and validated the results with a panel of five veterinary toxicologic pathologists using a multinomial logistic model. In this study, we assessed both the inter-rater and intra-rater agreement of the pathologists and compared pathologists' ratings to the artificial intelligence (AI)-predicted scores. Pathologists and the AI algorithm were presented with 500 slides of rodent heart. They quantified the amount of cardiomyopathy in each slide. A total of 200 of these slides were novel to this study, whereas 100 slides were intentionally selected for repetition from the previous study. After a washout period of more than six months, the repeated slides were examined to assess intra-rater agreement among pathologists. We found the intra-rater agreement to be substantial, with weighted Cohen's kappa values ranging from k = 0.64 to 0.80. Intra-rater variability is not a concern for the deterministic AI. The inter-rater agreement across pathologists was moderate (Cohen's kappa k = 0.56). These results demonstrate the utility of AI algorithms as a tool for pathologists to increase sensitivity and specificity for the histopathologic assessment of the heart in toxicology studies.

Assessment of laparoscopic fundoplication with endoscopy: room for improvement.

INTRODUCTION: Gastroesophageal reflux disease affects a significant portion of the Australian and world population. Minimally invasive laparoscopic fundoplication is a highly effective treatment in appropriately selected patients, with a 90% satisfaction rate. However, up to 5% will undergo revisional surgery. Endoscopy is an important investigation in the evaluation of persistent or new symptoms after fundoplication. Our study sought to evaluate the inter-rater reliability and variability in assessing fundoplication with endoscopy. METHODS: Upper gastrointestinal (UGI) surgeons and gastroenterologists were invited to join the cohort study through their professional membership with two societies based in Australia. Participants completed a two part 25-item multiple choice questionnaire, involving the analysis of ten static endoscopic images post-fundoplication. RESULTS: A total of 101 participants were included in the study (64 UGI surgeons and 37 gastroenterologists). Over 95% of participants were consultant level, working in non-rural tertiary hospitals. Total accuracy for all 10 cases combined was 76% for UGI surgeons and 69.9% for gastroenterologists. In three of the 10 cases, UGI surgeons performed significantly better than gastroenterologists (p < 0.05). When assessing performance across each of the 4 questions for each case, UGI surgeons were more accurate than gastroenterologists in describing the integrity of the wrap (p = 0.014). Inter-rater reliability was low across both groups for most domains (kappa < 1). CONCLUSION: Our study confirms low inter-rater reliability between endoscopists and large variations in reporting. UGI surgeons performed better than gastroenterologists in certain cases, usually when describing the integrity of the fundoplication. Our study provides further support for the use of a standardized reporting system in post-fundoplication patients.

The value of a multidisciplinary consensus meeting in achieving agreement on eating disorders diagnosis at a specialized referral center.

OBJECTIVE: This study aimed to evaluate the concordance of eating disorders (EDs) diagnoses within a multidisciplinary team in a specialized hospital unit dedicated to the medical care of ED. METHODS: The study analyzed data from 608 female patients who sought consultation at the Eating Disorders Referral Center between 2017 and 2021. The diagnoses were established according to the DSM-5 criteria by endocrinologists, psychiatrists, and finally confirmed or discussed within a monthly multidisciplinary consensus meeting (MCM). Fleiss' Kappa tests were conducted to assess inter-raters' agreement. RESULTS: Overall, substantial agreement was observed between endocrinologists and psychiatrists and the MCM. A more detailed analysis revealed variations in agreement across different disorders. Certain EDs demonstrated substantial agreement (e.g., anorexia nervosa restrictive subtype), while others approached near-perfect agreement (e.g., binge-eating disorder). In contrast, agreement was fair to poor for anorexia nervosa binge-purge subtype (ANBP) and slight for other specified feeding and ED. A period of temporary disagreement was noted for ANBP, partially attributed to practitioner turnover. An improvement in interdisciplinary agreement was observed for all ED diagnoses by the end of the study period. DISCUSSION: Variations or lower levels of inter-rater agreement may stem from atypical cases that fall on the border between two diagnoses or complex cases, as well as fluctuating symptoms. The progress observed throughout the study can be attributed in part to interdisciplinary learning, particularly facilitated by the MCM. The findings underscore the significance of striving for optimal concordance among different medical specialties to enhance patient care in ED treatment. PUBLIC SIGNIFICANCE: This study scrutinizes the agreement levels of ED diagnoses among endocrinologists and psychiatrists within a multidisciplinary team at an Eating Disorders Referral Center. While substantial overall agreement was achieved, disparities or lower agreement levels were evident for certain diagnoses such as anorexia nervosa binge-purge subtype. However, collaborative meetings led to a progressive enhancement in agreement over time. This research underscores the crucial role of a multidisciplinary team working collectively to ensure precise diagnoses and improved care for patients with EDs.

Surgeon assessment of significant rectal polyps using white light endoscopy alone and in comparison to fluorescence-augmented AI lesion classification.

PURPOSE: Perioperative decision making for large (> 2 cm) rectal polyps with ambiguous features is complex. The most common intraprocedural assessment is clinician judgement alone while radiological and endoscopic biopsy can provide periprocedural detail. Fluorescence-augmented machine learning (FA-ML) methods may optimise local treatment strategy. METHODS: Surgeons of varying grades, all performing colonoscopies independently, were asked to visually judge endoscopic videos of large benign and early-stage malignant (potentially suitable for local excision) rectal lesions on an interactive video platform (Mindstamp) with results compared with and between final pathology, radiology and a novel FA-ML classifier. Statistical analyses of data used Fleiss Multi-rater Kappa scoring, Spearman Coefficient and Frequency tables. RESULTS: Thirty-two surgeons judged 14 ambiguous polyp videos (7 benign, 7 malignant). In all cancers, initial endoscopic biopsy had yielded false-negative results. Five of each lesion type had had a pre-excision MRI with a 60% false-positive malignancy prediction in benign lesions and a 60% over-staging and 40% equivocal rate in cancers. Average clinical visual cancer judgement accuracy was 49% (with only 'fair' inter-rater agreement), many reporting uncertainty and higher reported decision confidence did not correspond to higher accuracy. This compared to 86% ML accuracy. Size was misjudged visually by a mean of 20% with polyp size underestimated in 4/6 and overestimated in 2/6. Subjective narratives regarding decision-making requested for 7/14 lesions revealed wide rationale variation between participants. CONCLUSION: Current available clinical means of ambiguous rectal lesion assessment is suboptimal with wide inter-observer variation. Fluorescence based AI augmentation may advance this field via objective, explainable ML methods.

Evaluating inter-rater reliability in the context of "Sysmex UN2000 detection of protein/creatinine ratio and of renal tubular epithelial cells can be used for screening lupus nephritis": a statistical examination.

BACKGROUND: The evaluation of inter-rater reliability (IRR) is integral to research designs involving the assessment of observational ratings by two raters. However, existing literature is often heterogeneous in reporting statistical procedures and the evaluation of IRR, although such information can impact subsequent hypothesis testing analyses. METHODS: This paper evaluates a recent publication by Chen et al., featured in BMC Nephrology, aiming to introduce an alternative statistical approach to assessing IRR and discuss its statistical properties. The study underscores the crucial need for selecting appropriate Kappa statistics, emphasizing the accurate computation, interpretation, and reporting of commonly used IRR statistics between two raters. RESULTS: The Cohen's Kappa statistic is typically used for two raters dealing with two categories or for unordered categorical variables having three or more categories. On the other hand, when assessing the concordance between two raters for ordered categorical variables with three or more categories, the commonly employed measure is the weighted Kappa. CONCLUSION: Chen and colleagues might have underestimated the agreement between AU5800 and UN2000. Although the statistical approach adopted in Chen et al.'s research did not alter their findings, it is important to underscore the importance of researchers being discerning in their choice of statistical techniques to address their specific research inquiries.

Preliminary evaluation of a questionnaire for assessing fidelity of early intervention for psychosis services.

PURPOSE: Fast, easy, and cost-effective methods are needed for fidelity assessment, quality improvement initiatives, and population-based studies in Early Intervention for Psychosis (EIP) services. Having an online questionnaire assessing the fidelity of EIP services, completed by staff self-reports, and having evidence of reliability and validity, could fill that gap. We assess the reliability and validity of the Early Intervention for Psychosis Services Fidelity Questionnaire (EIPS-FQ), developed in Part I of this set of papers. METHODS: A convenience sample of 10 EIP teams in England was used. Two staff members completed online questionnaires assessing recent and past fidelity. An external rater completed the same questionnaire for the two time periods, using a random sample of patient medical records, program documentation, and interviews with staff. The intra-class correlation coefficient (ICC) was calculated to assess inter-rater reliability. Validity was assessed using Bland-Altman plots, absolute mean differences, and the ICC. RESULTS: The fidelity score measuring recent fidelity ranged from 54.2 to 82.7, out of a possible 100. The ICC assessing reliability of the fidelity score was 0.40 (95% CI: 0.0-0.81). The ICCs for the fidelity sub-category scores ranged from 0 to 0.76. Two sub-categories, comprehensive assessments and family involvement and intervention, had low ICCs, regardless of period examined. CONCLUSIONS: This first attempt at validating the EIPS-FQ has demonstrated that the reliability of the EIPS-FQ is moderate/low, and therefore requires modification prior to use. The next iteration of the fidelity questionnaire will clarify or remove items which had very low reliability and add evidence-based components not identified in the Delphi exercise.

Signs of perceptual disorder during movement were reliably assessed in children with cerebral palsy in Sweden.

AIM: The aim of this Swedish study was to evaluate the assessment of clinical signs of perceptual disorder in children with cerebral palsy (CP). METHODS: Three experienced raters assessed 56 videos of 19 children from 1 to 18 years of age with bilateral spastic CP, which were recorded by colleagues at an Italian hospital. Six signs were evaluated for inter-rater reliability and criterion validity. Clinical applicability was evaluated by assessing inter-rater reliability between 47 Swedish clinicians, who examined 15 of the videos during face-to-face and online education seminars. There were 41 physiotherapists, two occupational therapists and four doctors, with 1-37 years of clinical experience and a median of 10 years. RESULTS: The experienced raters demonstrated moderate to almost perfect inter-rater reliability (kappa 0.54-0.81) and criterion validity (0.54-0.87) for startle reaction, upper limbs in startle position, averted eye gaze and eye blinking. The clinicians recognised these signs with at least moderate reliability (0.56-0.88). Grimacing and posture freezing were less reliable (0.22-0.35) and valid (0.09-0.50). CONCLUSION: Four of the six signs of perceptual disorder were reliably recognised by experienced raters and by clinicians after education seminars. Extended education and larger study samples are needed to recognise all the signs.

Inter-rater variability and repeatability in the assessment of the Tanner-Whitehouse classification of hand radiographs for the estimation of bone age.

OBJECTIVE: To determine which bones and which grades had the highest inter-rater variability when employing the Tanner-Whitehouse (T-W) method. MATERIALS AND METHODS: Twenty-four radiologists were recruited and trained in the T-W classification of skeletal development. The consistency and skill of the radiologists in determining bone development status were assessed using 20 pediatric hand radiographs of children aged 1 to 18 years old. Four radiologists had a poor concordance rate and were excluded. The remaining 20 radiologists undertook a repeat reading of the radiographs, and their results were analyzed by comparing them with the mean assessment of two senior experts as the reference standard. Concordance rate, scoring, and Kendall's W were calculated to evaluate accuracy and consistency. RESULTS: Both the radius, ulna, and short finger (RUS) system (Kendall's W = 0.833) and the carpal (C) system (Kendall's W = 0.944) had excellent consistency, with the RUS system outperforming the C system in terms of scores. The repeatability analysis showed that the second rating test, performed after 2 months of further bone age assessment (BAA) practice, was more consistent and accurate than the first. The capitate had the lowest average concordance rate and scoring, as well as the lowest overall concordance rate for its D classification. Moreover, the G classifications of the seven carpal bones all had a concordance rate less than 0.6. The bones with lower Kendall's W were likewise those with lower scores and concordance rates. CONCLUSION: The D grade of the capitate showed the highest variation, and the use of the Tanner-Whitehouse 3rd edition (T-W3) to determine bone age (BA) was frequently inconsistent. A more comprehensive description with a focus on inaccuracy bones or ratings and a modification to the T-W3 approach would significantly advance BAA.

Measuring and correcting staff variability in large-scale OSCEs.

CONTEXT: Objective Structured Clinical Examinations (OSCEs) are an increasingly popular evaluation modality for medical students. While the face-to-face interaction allows for more in-depth assessment, it may cause standardization problems. Methods to quantify, limit or adjust for examiner effects are needed. METHODS: Data originated from 3 OSCEs undergone by 900-student classes of 5th- and 6th-year medical students at Université Paris Cité in the 2022-2023 academic year. Sessions had five stations each, and one of the three sessions was scored by consensus by two raters (rather than one). We report OSCEs' longitudinal consistency for one of the classes and staff-related and student variability by session. We also propose a statistical method to adjust for inter-rater variability by deriving a statistical random student effect that accounts for staff-related and station random effects. RESULTS: From the four sessions, a total of 16,910 station scores were collected from 2615 student sessions, with two of the sessions undergone by the same students, and 36, 36, 35 and 20 distinct staff teams in each station for each session. Scores had staff-related heterogeneity (p<10-15), with staff-level standard errors approximately doubled compared to chance. With mixed models, staff-related heterogeneity explained respectively 11.4%, 11.6%, and 4.7% of station score variance (95% confidence intervals, 9.5-13.8, 9.7-14.1, and 3.9-5.8, respectively) with 1, 1 and 2 raters, suggesting a moderating effect of consensus grading. Student random effects explained a small proportion of variance, respectively 8.8%, 11.3%, and 9.6% (8.0-9.7, 10.3-12.4, and 8.7-10.5), and this low amount of signal resulted in student rankings being no more consistent over time with this metric, rather than with average scores (p=0.45). CONCLUSION: Staff variability impacts OSCE scores as much as student variability, and the former can be reduced with dual assessment or adjusted for with mixed models. Both are small compared to unmeasured sources of variability, making them difficult to capture consistently.

Nonaneurysmal perimesencephalic subarachnoid hemorrhage on noncontrast head CT: An accuracy, inter-rater, and intra-rater reliability study.

BACKGROUND AND PURPOSE: To evaluate the reliability and accuracy of nonaneurysmal perimesencephalic subarachnoid hemorrhage (NAPSAH) on Noncontrast Head CT (NCCT) between numerous raters. MATERIALS AND METHODS: 45 NCCT of adult patients with SAH who also had a catheter angiography (CA) were independently evaluated by 48 diverse raters; 45 raters performed a second assessment one month later. For each case, raters were asked: 1) whether they judged the bleeding pattern to be perimesencephalic; 2) whether there was blood anterior to brainstem; 3) complete filling of the anterior interhemispheric fissure (AIF); 4) extension to the lateral part of the sylvian fissure (LSF); 5) frank intraventricular hemorrhage; 6) whether in the hypothetical presence of a negative CT angiogram they would still recommend CA. An automatic NAPSAH diagnosis was also generated by combining responses to questions 2-5. Reliability was estimated using Gwet's AC1 (κG), and the relationship between the NCCT diagnosis of NAPSAH and the recommendation to perform CA using Cramer's V test. Multi-rater accuracy of NCCT in predicting negative CA was explored. RESULTS: Inter-rater reliability for the presence of NAPSAH was moderate (κG = 0.58; 95%CI: 0.47, 0.69), but improved to substantial when automatically generated (κG = 0.70; 95%CI: 0.59, 0.81). The most reliable criteria were the absence of AIF filling (κG = 0.79) and extension to LSF (κG = 0.79). Mean intra-rater reliability was substantial (κG = 0.65). NAPSAH weakly correlated with CA decision (V = 0.50). Mean sensitivity and specificity were 58% (95%CI: 44%, 71%) and 83 % (95%CI: 72 %, 94%), respectively. CONCLUSION: NAPSAH remains a diagnosis of exclusion. The NCCT diagnosis was moderately reliable and its impact on clinical decisions modest.

Interrater Agreement of CT Grading of Blunt Splenic Injuries: Does the AAST Grading Need to Be Reimagined?

Introduction: The Revised Organ Injury Scale (OIS) of the American Association for Surgery of Trauma (AAST) is the most widely accepted classification of splenic trauma. The objective of this study was to evaluate inter-rater agreement for CT grading of blunt splenic injuries. Methods: CT scans in adult patients with splenic injuries at a level 1 trauma centre were independently graded by 5 fellowship trained abdominal radiologists using the AAST OIS for splenic injuries - 2018 revision. The inter-rater agreement for AAST CT injury score, as well as low-grade (IIII) versus high-grade (IV-V) splenic injury was assessed. Disagreement in two key clinical scenarios (no injury versus injury, and high versus low grade) were qualitatively reviewed to identify possible sources of disagreement. Results: A total of 610 examinations were included. The inter-rater absolute agreement was low (Fleiss kappa statistic 0.38, P < 0.001), but improved when comparing agreement between low and high grade injuries (Fleiss kappa statistic of 0.77, P < .001). There were 34 cases (5.6%) of minimum two-rater disagreement about no injury vs injury (AAST grade ≥ I). There were 46 cases (7.5%) of minimum two-rater disagreement of low grade (AAST grade I-III) versus high grade (AAST grade IV-V) injuries. Likely sources of disagreement were interpretation of clefts versus lacerations, peri-splenic fluid versus subcapsular hematoma, application of adding multiple low grade injuries to higher grade injuries, and identification of subtle vascular injuries. Conclusion: There is low absolute agreement in grading of splenic injuries using the existing AAST OIS for splenic injuries.

Reliability of the Test of Gross Motor Development Third Edition Among Children with Developmental Coordination Disorder.

AIM: The Test of Gross Motor Development Third Edition (TGMD-3) is used to assess the development of fundamental movement skills in children from 3 to 10 years old. This study aimed to evaluate the intra-rater, inter-rater, and test-retest reliability and to determine the minimal detectable change (MDC) value of the TGMD-3 in children with developmental coordination disorder (DCD). METHODS: The TGMD-3 was administered to 20 children with DCD. The child's fundamental movement skills were recorded using a digital video camera. Reliability was assessed at two occasions by three raters using the generalizability theory. RESULTS: The TGMD-3 demonstrates good inter-rater reliability for the locomotor skills subscale, the ball skills subscale, and the total score (φ = 0.77 - 0.91), while the intra-rater reliability was even higher (φ = 0.94 - 0.97). Test-retest reliability was also shown to be good (φ = 0.79-0.93). The MDC95 was determined to be 10 points. CONCLUSION: This study provides evidence that the TGMD-3 is a reliable test when used to evaluate fundamental movement skills in children with DCD and suggests that an increase of 10 points represents a significant change in the motor function of a child with DCD.

Evaluating the Reliability of MyotonPro in Assessing Muscle Properties: A Systematic Review of Diagnostic Test Accuracy.

Background and Objectives: Muscle properties are critical for performance and injury risk, with changes occurring due to physical exertion, aging, and neurological conditions. The MyotonPro device offers a non-invasive method to comprehensively assess muscle biomechanical properties. This systematic review evaluates the reliability of MyotonPro across various muscles for diagnostic purposes. Materials and Methods: Following PRISMA guidelines, a comprehensive literature search was conducted in Medline (PubMed), Ovid (Med), Epistemonikos, Embase, Cochrane Library, Clinical trials.gov, and the WHO International Clinical Trials platform. Studies assessing the reliability of MyotonPro across different muscles were included. A methodological quality assessment was performed using established tools, and reviewers independently conducted data extraction. Statistical analysis involved summarizing intra-rater and inter-rater reliability measures across muscles. Results: A total of 48 studies assessing 31 muscles were included in the systematic review. The intra-rater and inter-rater reliability were consistently high for parameters such as frequency and stiffness in muscles of the lower and upper extremities, as well as other muscle groups. Despite methodological heterogeneity and limited data on specific parameters, MyotonPro demonstrated promising reliability for diagnostic purposes across diverse patient populations. Conclusions: The findings suggest the potential of MyotonPro in clinical assessments for accurate diagnosis, treatment planning, and monitoring of muscle properties. Further research is needed to address limitations and enhance the applicability of MyotonPro in clinical practice. Reliable muscle assessments are crucial for optimizing treatment outcomes and improving patient care in various healthcare settings.

Evaluation of inter-rater and test-retest reliability for near-infrared spectroscopy reactive hyperemia measures.

BACKGROUND: Near-infrared spectroscopy (NIRS) is a non-invasive tool used to measure blood flow in peripheral tissues. More information on inter-rater agreement and test-retest reliability of NIRS-based reperfusion assessments is needed. PURPOSE: To assess inter-rater agreement for NIRS based data analysis, and evaluate the measurement's reliability across days. METHODS: On three separate days (average days between visits 1 and 3: 19.4 ± 6.9 days), participants' (N = 15 males, 22 ± 2 yr.) post-occlusion reactive hyperemia (PORH) was measured in the left gastrocnemius muscle using Continuous-Wave NIRS (CW-NIRS). A blood pressure cuff was placed proximal to the knee and inflated to occlude lower leg blood flow for 5 min. The following CW-NIRS parameters were selected: (1) percent saturation in HbO2 (StO2%) at baseline; (2) the O2Hb range used to normalize the NIRS signal; (3) the time for the O2Hb signal to reach 50 % peak post-occlusion hyperemia (T1/2), and (4) the post peak hyperemic O2Hb recovery slope (O2REC-SLP). Absolute agreement between the two analysts was calculated using two-way random effects Intraclass Correlation Coefficients (ICC2,1). Consistency between analysts and across days was calculated using two-way mixed models (ICC3,1). Mean and 95 % confidence intervals (CI) of ICCs are reported. Coefficient of variation (CV) and standard error of the measurement (SEM) are reported. RESULTS: The ICC2,1 data indicated "adequate" to "excellent" absolute agreement between the two NIRS analysts. ICC2,3 data indicated "adequate" to "good" reliability across visits. The CV and SEM for rater 1 and rater 2 across visit were StO2 (CV: 3.79 % ± 2.71 % and 4.50 % ± 2.37 %; SEM: 3.42 and 3.82), O2Hb range (CV: 10.50 ± 5.93 and 12.79 ± 12.41; SEM: 3.26 and 4.71), T1/2 (CV: 11.15 % ± 5.52 % and 10.96 % ± 4.50; SEM: 1.22 and 1.11), and O2REC-SLP (CV: 19.49 % ± 9.99 % and 18.45 % ± 9.48 %; SEM: 0.04 and 0.04). CONCLUSION: It is concluded that NIRS parameters assessed show adequate reliability between analysts and across three visits. It is recommended, when feasible and because of the absence of 100 % reliability, that investigators employ more than one rater for scoring at least a portion of the data across each trial in a study's control condition in order to have the ability to estimate the magnitude of error attributable to imperfect reliability.

Far-field potential of the compound muscle action potential as a reliable marker in amyotrophic lateral sclerosis.

INTRODUCTION/AIMS: Reliable neurophysiological markers in amyotrophic lateral sclerosis (ALS) are of great interest. The compound muscle action potential (CMAP) amplitude has been a conventional marker, although it is greatly influenced by the electrode position. We propose the far-field potential of the CMAP (FFP-CMAP) as a new neurophysiological marker in ALS. METHODS: Patients with ALS and age-matched healthy controls were enrolled. We used a proximal reference (pref) in addition to the conventional distal reference (dref). Routine CMAP was recorded from the belly-dref lead and FFP-CMAP from the dref-pref lead for the ulnar and tibial nerves. Multiple point stimulation motor unit number estimation (MUNE) was also examined in the ulnar nerve. Inter-rater reproducibility was evaluated by two examiners, and some patients were followed up every 3 mo for 1 y. RESULTS: We tested 17 patients with ALS and 10 controls. The amplitudes of routine CMAP and FFP-CMAP in the ulnar and tibial nerves, and hypothenar MUNE value in the ulnar nerve were significantly decreased in ALS compared to controls. Ulnar FFP-CMAP achieved the highest inter-rater intraclass correlation coefficient (ICC) value (0.942) when compared with routine CMAP (0.880) and MUNE (0.839). The tibial FFP-CMAP had a higher ICC value (0.986) than the routine CMAP (0.697). In this way, the FFP-CMAP showed high inter-rater reproducibility because its shape was not much influenced by the electrode position. During 1-y follow-up, decline of CMAP, FFP, and MUNE showed significant correlations with the Amyotrophic Lateral Sclerosis Functional Rating Scale - Revised (ALSFRS-R). DISCUSSION: The FFP-CMAP shows promise as a reliable marker for ALS.

Detection of grey zones in inter-rater agreement studies.

BACKGROUND: In inter-rater agreement studies, the assessment behaviour of raters can be influenced by their experience, training levels, the degree of willingness to take risks, and the availability of clear guidelines for the assessment. When the assessment behaviour of raters differentiates for some levels of an ordinal classification, a grey zone occurs between the corresponding adjacent cells to these levels around the main diagonal of the table. A grey zone introduces a negative bias to the estimate of the agreement level between the raters. In that sense, it is crucial to detect the existence of a grey zone in an agreement table. METHODS: In this study, a framework composed of a metric and the corresponding threshold is developed to identify grey zones in an agreement table. The symmetry model and Cohen's kappa are used to define the metric, and the threshold is based on a nonlinear regression model. A numerical study is conducted to assess the accuracy of the developed framework. Real data examples are provided to illustrate the use of the metric and the impact of identifying a grey zone. RESULTS: The sensitivity and specificity of the proposed framework are shown to be very high under moderate, substantial, and near-perfect agreement levels for [Formula: see text] and [Formula: see text] tables and sample sizes greater than or equal to 100 and 50, respectively. Real data examples demonstrate that when a grey zone is detected in the table, it is possible to report a notably higher level of agreement in the studies. CONCLUSIONS: The accuracy of the proposed framework is sufficiently high; hence, it provides practitioners with a precise way to detect the grey zones in agreement tables.

Repeatability and reproducibility of various 4D Flow MRI postprocessing software programs in a multi-software and multi-vendor cross-over comparison study.

BACKGROUND: Different software programs are available for the evaluation of 4D Flow cardiovascular magnetic resonance (CMR). A good agreement of the results between programs is a prerequisite for the acceptance of the method. Therefore, the goal was to compare quantitative results from a cross-over comparison in individuals examined on two scanners of different vendors analyzed with four postprocessing software packages. METHODS: Eight healthy subjects (27 ± 3 years, 3 women) were each examined on two 3T CMR systems (Ingenia, Philips Healthcare; MAGNETOM Skyra, Siemens Healthineers) with a standardized 4D Flow CMR sequence. Six manually placed aortic contours were evaluated with Caas (Pie Medical Imaging, SW-A), cvi42 (Circle Cardiovascular Imaging, SW-B), GTFlow (GyroTools, SW-C), and MevisFlow (Fraunhofer Institute MEVIS, SW-D) to analyze seven clinically used parameters including stroke volume, peak flow, peak velocity, and area as well as typically scientifically used wall shear stress values. Statistical analysis of inter- and intrareader variability, inter-software and inter-scanner comparison included calculation of absolute and relative error (ER), intraclass correlation coefficient (ICC), Bland-Altman analysis, and equivalence testing based on the assumption that inter-software differences needed to be within 80% of the range of intrareader differences. RESULTS: SW-A and SW-C were the only software programs showing agreement for stroke volume (ICC = 0.96; ER = 3 ± 8%), peak flow (ICC: 0.97; ER = -1 ± 7%), and area (ICC = 0.81; ER = 2 ± 22%). Results from SW-A/D and SW-C/D were equivalent only for area and peak flow. Other software pairs did not yield equivalent results for routinely used clinical parameters. Especially peak maximum velocity yielded poor agreement (ICC ≤ 0.4) between all software packages except SW-A/D that showed good agreement (ICC = 0.80). Inter- and intrareader consistency for clinically used parameters was best for SW-A and SW-D (ICC = 0.56-97) and worst for SW-B (ICC = -0.01-0.71). Of note, inter-scanner differences per individual tended to be smaller than inter-software differences. CONCLUSIONS: Of all tested software programs, only SW-A and SW-C can be used equivalently for determination of stroke volume, peak flow, and vessel area. Irrespective of the applied software and scanner, high intra- and interreader variability for all parameters have to be taken into account before introducing 4D Flow CMR in clinical routine. Especially in multicenter clinical trials a single image evaluation software should be applied.

Challenges in Inter-rater Agreement on Lamina Propria Fibrosis in Esophageal Biopsies.

BACKGROUND: Mucosal biopsies in eosinophilic esophagitis (EoE) can exhibit lamina propria (LP) fibrosis, which may portend stenotic complications; however, the histologic diagnosis of LP fibrosis is subjective. We sought to assess and improve the consistency of LP fibrosis diagnosis among our pathologist group. METHODS: At a large pediatric hospital, 25 esophageal biopsy slides from 19 patients (16 with EoE) exhibiting a wide spectrum of LP area, artifacts, and fibrosis severity were scanned into whole-slide images. Staff pediatric pathologists (n = 8) separate from the authors classified each biopsy by LP adequacy and fibrosis severity 1 month before and after completion of an educational tutorial. Consensus was defined as >70% agreement. RESULTS: At baseline, 16/25 (64%) cases reached consensus for no fibrosis (n = 3), fibrosis (n = 7), or inadequate LP (n = 6); agreement was fair (α = 0.34). Post-tutorial, 13/25 (52%) cases reached consensus for no fibrosis (n = 2), fibrosis (n = 7), or inadequate LP (n = 4); agreement was again fair (α = 0.33). There was moderate agreement in grading of fibrosis severity (α = 0.54). CONCLUSION: We document only fair-to-moderate agreement in the diagnosis of esophageal LP fibrosis and adequacy in a large pediatric pathologist group despite targeted education, highlighting a challenge in incorporating this feature into EoE research and clinical decision-making.