Validation of automated Alberta Stroke Program Early CT Score (ASPECTS) software for detection of early ischemic changes on non-contrast brain CT scans.

PURPOSE: In ASPECTS, 10 brain regions are scored visually for presence of acute ischemic stroke damage. We evaluated automated ASPECTS in comparison to expert readers. METHODS: Consecutive, baseline non-contrast CT-scans (5-mm slice thickness) from the prospective MR CLEAN trial (n = 459, MR CLEAN Netherlands Trial Registry number: NTR1804) were evaluated. A two-observer consensus for ASPECTS regions (normal/abnormal) was used as reference standard for training and testing (0.2/0.8 division). Two other observers provided individual ASPECTS-region scores. The Automated ASPECTS software was applied. A region score specificity of ≥ 90% was used to determine the software threshold for detection of an affected region based on relative density difference between affected and contralateral region. Sensitivity, specificity, and receiver-operating characteristic curves were calculated. Additionally, we assessed intraclass correlation coefficients (ICCs) for automated ASPECTS and observers in comparison to the reference standard in the test set. RESULTS: In the training set (n = 104), with software thresholds for a specificity of ≥ 90%, we found a sensitivity of 33-49% and an area under the curve (AUC) of 0.741-0.785 for detection of an affected ASPECTS region. In the test set (n = 355), the results for the found software thresholds were 89-89% (specificity), 41-57% (sensitivity), and 0.750-0.795 (AUC). Comparison of automated ASPECTS with the reference standard resulted in an ICC of 0.526. Comparison of observers with the reference standard resulted in an ICC of 0.383-0.464. CONCLUSION: The performance of automated ASPECTS is comparable to expert readers and could support readers in the detection of early ischemic changes.

Automated neurosurgical stereotactic planning for intraoperative use: a comprehensive review of the literature and perspectives.

The creation of intracranial stereotactic trajectories, from entry point to target point, is still mostly done manually by the neurosurgeon. The development of automated stereotactic planning tools has been described in the literature. This systematic review aims to assess the effectiveness of stereotactic planning procedure automation and develop tools for patients undergoing neurosurgical stereotactic procedures. PubMed/MEDLINE, EMBASE, Google Scholar, CINAHL, PsycINFO, and Cochrane Register of Controlled Trials databases were searched from inception to September 1, 2019, at the exception of Google Scholar (from 1 January 2010 to September 1, 2019) in French and English. Eligible studies included all studies proposing automated stereotactic planning. A total of 1543 studies were screened. Forty-two studies were included in the systematic review, including 18 (42.9%) conference papers. The surgical procedures planned automatically were mainly deep brain stimulation (n = 14, 33.3%), stereoelectroencephalography (n = 12, 28.6%), and not specified (n = 10, 23.8%). The most frequently used surgical constraints to plan the trajectory were blood vessels (n = 32, 76.2%), cerebral sulci (n = 27, 64.3%), and cerebral ventricles (n = 23, 54.8%). The distance from blood vessels ranged from 1.96 to 4.78 mm for manual trajectories and from 2.47 to 7.0 mm for automated trajectories. At least one neurosurgeon was involved in 36 studies (85.7%). The automated stereotactic trajectory was preferred in 75.4% of the studied cases (range 30-92.9). Only 3 (7.1%) studies were multicentric. No study reported prospective use of the planning software. Stereotactic planning automation is a promising tool to provide valuable stereotactic trajectories for clinical applications.

Diffusion gradient nonlinearity bias correction reduces bias of breast cancer bone metastasis ADC values.

CONTRACT GRANT SPONSOR: Health Research Fund of Central Denmark Region. BACKGROUND: Diffusion gradient nonlinearity (DGNL) bias causes apparent diffusion coefficient (ADC) values to drop with increasing superior-inferior (SI) isocenter offset. This is a concern when performing quantitative diffusion-weighted imaging (DWI). PURPOSE/HYPOTHESIS: To investigate if DGNL ADC bias can be corrected in breast cancer bone metastases using a clinical DWI protocol and an online correction algorithm. STUDY TYPE: Prospective. SUBJECTS/PHANTOM: A diffusion phantom (Model 128, High Precision Devices, Boulder, CO) was used for in vitro validation. Twenty-three women with bone-metastasizing breast cancer were enrolled to assess DGNL correction in vivo. FIELD STRENGTH/SEQUENCE: DWI was performed on a 1.5T MRI system as single-shot, spin-echo, echo-planar imaging with short-tau inversion recovery (STIR) fat-saturation. ADC maps with and without DGNL correction were created from the b50 and b800 images. ASSESSMENT: Uncorrected and DGNL-corrected ADC values were measured in phantom and bone metastases by placing regions of interest on b800 images and copying them to the ADC map. The SI offset was recorded. STATISTICAL TESTS: In all, 79 bone metastases were assessed. ADC values with and without DGNL correction were compared at 14 cm SI offset using a two-tailed t-test. RESULTS: In the diffusion phantom, DGNL correction increased SI offset, where ADC bias was lower than 5%, from 7.3-13.8 cm. Of the 23 patients examined, six had no metastases in the covered regions. In the remaining patients, bias of uncorrected bone metastasis ADC values was 19.1% (95% confidence interval [CI]: 15.4-22.9%) at 14 cm SI offset. After DGNL correction, ADC bias was significantly reduced to 3.5% (95% CI: 0.7-6.3%, P < 0.001), thus reducing bias due to DGNL by 82%. DATA CONCLUSION: Online DGNL correction corrects DGNL ADC value bias and allows increased station lengths in the SI direction. LEVEL OF EVIDENCE: 2 Technical Efficacy: Stage 2 J. Magn. Reson. Imaging 2020;51:904-911.

Feasibility and accuracy of a novel automated three-dimensional ultrasonographic analysis system for abdominal aortic aneurysm: comparison with two-dimensional ultrasonography and computed tomography.

BACKGROUND: Accurate measurement of the maximum aortic diameter (Dmax) is crucial for patients with abdominal aortic aneurysm (AAA). Aortic computed tomography (CT) provides accurate Dmax values by three-dimensional (3-D) reconstruction but may cause nephrotoxicity because of contrast use and radiation hazard. We aimed to evaluate the accuracy of a novel semi-automated 3-D ultrasonography (3-D US) system compared with that of CT as a reference. METHODS: Patients with AAA (n = 59) or individuals with normal aorta (n = 18) were prospectively recruited in an outpatient setting. Two-dimensional ultrasonography (2-D US) and 3-D US images were acquired with a single-sweep volumetric transducer. The analysis was performed offline with a software. Dmax and the vessel area of the Dmax slice were measured with 2-D US, 3-D US, and CT. The lumen and thrombus areas of the Dmax slice were also measured in 40 patients with intraluminal thrombus. Vessel and thrombus volumes were measured using 3-D US and CT. RESULTS: The Dmax values from 3-D US demonstrated better agreement (R2 = 0.984) with the CT values than with the 2-D US values (R2 = 0.938). Overall, 2-D US underestimated Dmax compared with 3-D US (32.3 ± 12.1 mm vs. 35.1 ± 12.0 mm). The Bland-Altman analysis of the 3-D US values, revealed better agreement with the CT values (2 standard deviations [SD], 2.9 mm) than with the 2-D US values (2 SD, 5.4 mm). The vessel, lumen, and thrombus areas all demonstrated better agreement with CT than with 2-D US (R2 = 0.986 vs. 0.960 for the vessel, R2 = 0.891 vs. 0.837 for the lumen, and R2 = 0.977 vs. 0.872 for the thrombus). The thrombus volume assessed with 3-D US showed good correlation with the CT value (R2 = 0.981 and 2 SD in the Bland-Altman analysis: 13.6 cm3). CONCLUSIONS: Our novel semi-automated 3-D US analysis system provides more accurate Dmax values than 2-D US and provides precise volumetric data, which were not evaluable with 2-D US. The application of the semi-automated 3-D US analysis system in abdominal aorta assessment is easy and accurate.

QardioArm Blood Pressure Monitoring in a Population With Type 2 Diabetes: Validation Study.

BACKGROUND: Home blood pressure monitoring has many benefits, even more so, in populations prone to high blood pressure, such as persons with diabetes. OBJECTIVE: The purpose of this research was to validate the QardioArm mobile device in a sample of individuals with noninsulin-dependent type 2 diabetes in accordance with the guidelines of the second International Protocol of the European Society of Hypertension. METHODS: The sample consisted of 33 patients with type 2 diabetes. To evaluate the validity of QardioArm by comparing its data with that obtained with a digital sphygmomanometer (Omron M3 Intellisense), two nurses collected diastolic blood pressure, systolic blood pressure, and heart rate with both devices. RESULTS: The analysis indicated that the test device QardioArm met all the validation requirements using a sample population with type 2 diabetes. CONCLUSIONS: This paper reports the first validation of QardioArm in a population of individuals with noninsulin-dependent type 2 diabetes. QardioArm for home monitoring of blood pressure and heart rate met the requirements of the second International Protocol of the European Society of Hypertension.

Automated versus manual imaging assessment of early ischemic changes in acute stroke: comparison of two software packages and expert consensus.

AIM: The purpose of our study was to compare the agreement of both the total Alberta Stroke Program Early CT Score (ASPECTS) and region-based scores from two automated ASPECTS software packages and an expert consensus (EC) reading with final ASPECTS in a selected cohort of patients who had prompt reperfusion from endovascular thrombectomy (EVT). METHODS: ASPECTS were retrospectively and blindly assessed by two software packages and EC on baseline non-contrast-enhanced computed tomography (NCCT) images. All patients had multimodal CT imaging including NCCT, CT angiography, and CT perfusion which demonstrated an acute anterior circulation ischemic stroke with a large vessel occlusion. Final ASPECTS on follow-up scans in patients who had EVT and achieved complete reperfusion within 100 min from NCCT served as ground truth and were compared to total and region-based scores. RESULTS: Fifty-two patients met our study criteria. Moderate agreement was obtained between both software packages and EC for total ASPECTS and there was no significant difference in overall performance. However, the software packages differed with respect to regional contribution. In this cohort, the majority of infarcted regions were deep structures. Package A was more sensitive in cortical areas than the other methods, but at a cost of specificity. EC and software package B had greater sensitivity, but lower specificity for deep brain structures. CONCLUSION: In this cohort, using the final ASPECTS as ground truth, no clinically significant difference was observed for total ASPECT score between human or automated packages, but there were differences in the characteristics of the regions scored. KEY POINTS: • Some national stroke guidelines have incorporated ASPECTS in their recommendations for selecting patients for endovascular therapy. • Computer-aided diagnosis is a promising tool to aid the evaluation of early ischemic changes identified on CT. • Software packages for automated ASPECTS assessment differed significantly with respect to regional contribution without any significant difference in the overall ASPECT score.

A New Methodology to Determine Apposition, Dilatation, and Position of Endografts in the Descending Thoracic Aorta After Thoracic Endovascular Aortic Repair.

Purpose: To validate computed tomography angiography (CTA)-applied software to assess apposition, dilatation, and position of endografts in the proximal and distal landing zones after thoracic endovascular aortic repair (TEVAR) of thoracic aortic aneurysm. Materials and Methods: Twenty-two patients (median age 75.5 years; 11 men) with a degenerative descending thoracic aortic aneurysm treated with TEVAR with at least one postoperative CTA were selected from a single center's database. New CTA-applied software was used to determine the available apposition surface in the proximal and distal landing zones, apposition of the endograft fabric with the aortic wall, shortest apposition length, endograft inflow and outflow diameters, shortest distance between the left subclavian artery and the proximal endograft fabric, and shortest distance between the celiac trunk and the distal endograft fabric on each CTA. Interobserver variability for these parameters was assessed with the repeatability coefficient and the intraclass correlation coefficient. Results: Excellent interobserver agreement was found for all measurements. Interobserver variability of surface and shortest apposition length calculations was larger for the distal site compared with the proximal site, with a mean difference of 10% vs 2% of the mean available apposition surface, 12% vs 5% of the endograft apposition surface, and 16% vs 8% of the shortest apposition length, respectively. Inflow and outflow diameters of the endograft showed low variability, with a mean difference of 0.1 mm with 95% of the interobserver difference within 1.8 mm. Mean interobserver differences of the proximal and distal shortest fabric distances were 1.0 and 0.9 mm (both 2% of the mean lengths). Conclusion: Assessment of apposition, dilatation, and position of the proximal and distal parts of an endograft in the descending thoracic aorta is feasible after TEVAR with the new software. Interobserver agreement for all measured parameters was excellent for the proximal and distal landing zones. The new method allows detection of subtle changes during follow-up. However, a larger study is needed to quantify how parameters change over time in complicated and uncomplicated TEVAR cases and to define the real added value of the new methodology.

Comparability of semiautomatic tortuosity measurements in the carotid artery.

PURPOSE: Increased arterial tortuosity has been suggested as a predisposing factor for carotid artery dissection, which is an important risk factor for development of extracranial carotid artery aneurysms (ECAA). Prior to comparison with non-ECAA controls, the optimal measurement technique should be defined. This study describes the difference between software packages in terms of reproducibility and absolute outcome of arterial tortuosity measurements in ECAA patients. METHODS: CT-angiography analysis was performed on 12 ECAA patients selected from our registry, using four software packages: 3mensio Vascular, TeraRecon, Vital Images, and Aycan OsiriX PRO. The tortuosity index (TI) was calculated from the skull base until the carotid bifurcation and aortic arch, and was defined as the centerline's true length divided by the straight line distance. Intraclass correlation coefficients (ICC) with 95% confidence intervals were calculated to quantify inter- and intra-observer variability within one software package, and differences in measured TI between packages. RESULTS: Inter-observer agreement was nearly perfect for 3mensio, excellent for Vital Images and OsiriX, and substantial for TeraRecon, with ICC 0.99 (0.96-1.0), 0.90 (0.69-0.97), 0.84 (0.53-0.95), and 0.72 (0.28-0.91), respectively. Intra-observer agreement ranged from ICC 1.0 for 3mensio to 0.91 for TeraRecon. Agreements in TI ranged from ICC 0.99 (0.98-1.0) for 3mensio vs. OsiriX, to 0.95 (0.82-0.98) for 3mensio vs. TeraRecon. Median time needed to complete one round of measurements was highest for OsiriX (p = 0.013). CONCLUSIONS: Carotid artery tortuosity measurements are reproducible and comparable between current commercially available software packages, with high intra-observer agreement. Although the reproducibility differed per software packages, all packages scored an acceptable inter-observer agreement.

Standardization of hippocampus volumetry using automated brain structure volumetry tool for an initial Alzheimer's disease imaging biomarker.

BACKGROUND: One significant barrier to incorporate Alzheimer's disease (AD) imaging biomarkers into diagnostic criteria is the lack of standardized methods for biomarker quantification. The European Alzheimer's Disease Consortium-Alzheimer's Disease Neuroimaging Initiative (EADC-ADNI) Harmonization Protocol project provides the most authoritative guideline for hippocampal definition and has produced a manually segmented reference dataset for validation of automated methods. PURPOSE: To validate automated hippocampal volumetry using AccuBrain™, against the EADC-ADNI dataset, and assess its diagnostic performance for differentiating AD and normal aging in an independent cohort. MATERIAL AND METHODS: The EADC-ADNI reference dataset comprise of manually segmented hippocampal labels from 135 volumetric T1-weighted scans from various scanners. Dice similarity coefficient (DSC), intraclass correlation coefficient (ICC), and Pearson's r were obtained for AccuBrain™ and FreeSurfer. The magnetic resonance imaging (MRI) of a separate cohort of 299 individuals (150 normal controls, 149 with AD) were obtained from the ADNI database and processed with AccuBrain™ to assess its diagnostic accuracy. Area under the curve (AUC) for total hippocampal volumes (HV) and hippocampal fraction (HF) were determined. RESULTS: Compared with EADC-ADNI dataset ground truths, AccuBrain™ had a mean DSC of 0.89/0.89/0.89, ICC of 0.94/0.96/0.95, and r of 0.95/0.96/0.95 for right/left/total HV. AccuBrain™ HV and HF had AUC of 0.76 and 0.80, respectively. Thresholds of ≤ 5.71 mL and ≤ 0.38% afforded 80% sensitivity for AD detection. CONCLUSION: AccuBrain™ provides accurate automated hippocampus segmentation in accordance with the EADC-ADNI standard, with great potential value in assisting clinical diagnosis of AD.

QardioArm Upper Arm Blood Pressure Monitor Against Omron M3 Upper Arm Blood Pressure Monitor in Patients With Chronic Kidney Disease: A Validation Study According to the European Society of Hypertension International Protocol Revision 2010.

BACKGROUND: Hypertension is considered as a main risk factor for chronic kidney disease development and progression. Thus, the control and evaluation of this disease with new software and devices are especially important in patients who suffer from chronic kidney disease. OBJECTIVE: This study aimed to validate the QardioArm mobile device, which is used for blood pressure (BP) self-measurement in patients who suffer from chronic kidney disease, by following the European Society of Hypertension International Protocol 2 (ESH-IP2) guidelines. METHODS: A validation study was carried out by following the ESH-IP2 guidelines. A sample of 33 patients with chronic kidney disease self-measured their BP by using the QardioArm and Omron M3 Intellisense devices. Heart rate (HR), diastolic BP, and systolic BP were measured. RESULTS: The QardioArm fulfilled the ESH-IP2 validation criteria in patients who suffered from chronic kidney disease. CONCLUSIONS: Thus, this study is considered as the first validation using a wireless upper arm oscillometric device connected to an app to measure BP and HR meeting the ESH-IP2 requirements in patients who suffer from chronic kidney disease. New validation studies following the ESH-IP2 guidelines should be carried out using different BP devices in patients with specific diseases.

VAMPIRE® fundus image analysis algorithms: Validation and diagnostic relevance in hypertensive cats.

OBJECTIVES: To validate a retinal imaging software named VAMPIRE® (Vascular Assay and Measurement Platform for Images of the Retina) in feline patients and test the clinical utility in hypertensive cats. ANIMALS STUDIED: One hundred and five healthy cats were enrolled. They represented the normal dataset used in the validation (group 1). Forty-three hypertensive cats with no noticeable retinal abnormalities were enrolled for the clinical validity of the software (group 2). PROCEDURES: Eleven points (4 veins, 4 arteries, and 3 arterial bifurcations) were measured for each digital image. Repeatability and reproducibility of measurements were assessed using two independent operators. Data were statistically analyzed by the Mann-Whiney and Tukey box plot. Significance was considered when P < 0.05. RESULTS: Two hundred and ten retinal images were analyzed for a total of 2310 measurements. Total mean was 9.1 and 6.1 pixels for veins and arteries, respectively. First, second, and third arteriolar bifurcations angles were 73.6°, 76.9°, and 85.4°, respectively. A comparison between groups 1 and 2 showed a statistically significant reduction in arteriolar diameter (mean 3.3 pixels) and branch angle (55°, 47.8° and 59.9°) associated with increasing vein diameter (mean 24.15 pixels). CONCLUSIONS: Current image analysis techniques used in human medicine were investigated in terms of extending their use to veterinary medicine. The VAMPIRE® algorithm proved useful for an objective diagnosis of retinal vasculature changes secondary to systemic hypertension in cats, and could be an additional diagnostic test for feline systemic hypertension.

Using AutoCAD software to measure venous leg ulcers: a reproducibility assessment study.

OBJECTIVE: To assess the reproducibility of using AutoCAD software to measure the area of venous leg ulcers (VLUs). METHOD: Data from patients with VLUs were collected between March and July 2015, using data collection forms and photographing the different ulcers. A researcher and five nurses collected the data. The wounds were measured using AutoCAD software. Data were analysed using intraclass correlation coefficient (ICC), concordance correlation coefficient (CCC) and Bland-Altman analysis. RESULTS: A total of 21 patients with 36 VLUs were included in the study. A statistically significant difference (p<0.05) was observed between the areas of VLUs measured by the researcher and the evaluators. There was an excellent agreement when analysing the ICC [p=0.98; 95% CI (0.97-0.99); p <0.05] and the CCC [CCC=CI 0.97; 95% (0.95-0.98)]. There was no difference between the measurements of VLUs ≤10 cm2 (p=0.49) and those with an area >10cm2 (p=0.22). CONCLUSION: The use of AutoCAD software is appropriate for measuring VLUs and appears to be more accurate when used to measure VLUs with an area >10cm2.

[Legal framework for E­health products on the European market]. / Medizinprodukterechtliche Rahmenbedingungen für E­Health-Produkte im europäischen Wirtschaftsraum.

Legitimately categorizing software in the healthcare sector is complex. According to European medical devices law, software can be considered a medical device. The decisive factor is whether the software is used for directly controlling a therapeutic or diagnostic medical device or as stand-alone software that achieves specific medical purposes, as they are described in the legal definition of a medical device. A contribution of the software to diagnosis or therapy can be sufficient for the categorization; it does not have to provide the complete diagnosis or the complete therapy itself.This principle will continue to be the same with the new Regulation on Medical Devices (EU) 2017/745, even though the classification rules have been more closely and more specifically tailored to software and more stringent requirements for essential safety and performance requirements have to be met.

Myocardial segmentation based on coronary anatomy using coronary computed tomography angiography: Development and validation in a pig model.

OBJECTIVES: To validate a method for performing myocardial segmentation based on coronary anatomy using coronary CT angiography (CCTA). METHODS: Coronary artery-based myocardial segmentation (CAMS) was developed for use with CCTA. To validate and compare this method with the conventional American Heart Association (AHA) classification, a single coronary occlusion model was prepared and validated using six pigs. The unstained occluded coronary territories of the specimens and corresponding arterial territories from CAMS and AHA segmentations were compared using slice-by-slice matching and 100 virtual myocardial columns. RESULTS: CAMS more precisely predicted ischaemic area than the AHA method, as indicated by 95% versus 76% (p < 0.001) of the percentage of matched columns (defined as percentage of matched columns of segmentation method divided by number of unstained columns in the specimen). According to the subgroup analyses, CAMS demonstrated a higher percentage of matched columns than the AHA method in the left anterior descending artery (100% vs. 77%; p < 0.001) and mid- (99% vs. 83%; p = 0.046) and apical-level territories of the left ventricle (90% vs. 52%; p = 0.011). CONCLUSIONS: CAMS is a feasible method for identifying the corresponding myocardial territories of the coronary arteries using CCTA. KEY POINTS: • CAMS is a feasible method for identifying corresponding coronary territory using CTA • CAMS is more accurate in predicting coronary territory than the AHA method • The AHA method may underestimate the ischaemic territory of LAD stenosis.

Framework for the quality assurance of 'omics technologies considering GLP requirements.

'Omics technologies are gaining importance to support regulatory toxicity studies. Prerequisites for performing 'omics studies considering GLP principles were discussed at the European Centre for Ecotoxicology and Toxicology of Chemicals (ECETOC) Workshop Applying 'omics technologies in Chemical Risk Assessment. A GLP environment comprises a standard operating procedure system, proper pre-planning and documentation, and inspections of independent quality assurance staff. To prevent uncontrolled data changes, the raw data obtained in the respective 'omics data recording systems have to be specifically defined. Further requirements include transparent and reproducible data processing steps, and safe data storage and archiving procedures. The software for data recording and processing should be validated, and data changes should be traceable or disabled. GLP-compliant quality assurance of 'omics technologies appears feasible for many GLP requirements. However, challenges include (i) defining, storing, and archiving the raw data; (ii) transparent descriptions of data processing steps; (iii) software validation; and (iv) ensuring complete reproducibility of final results with respect to raw data. Nevertheless, 'omics studies can be supported by quality measures (e.g., GLP principles) to ensure quality control, reproducibility and traceability of experiments. This enables regulators to use 'omics data in a fit-for-purpose context, which enhances their applicability for risk assessment.

OPEN-SOURCE SOFTWARE IN DENTISTRY: A SYSTEMATIC REVIEW.

OBJECTIVES: Technological development and the need for electronic health records management resulted in the need for a computer with dedicated, commercial software in daily dental practice. The alternative for commercial software may be open-source solutions. Therefore, this study reviewed the current literature on the availability and use of open-source software (OSS) in dentistry. METHODS: A comprehensive database search was performed on February 1, 2017. Only articles published in peer-reviewed journals with a focus on the use or description of OSS were retrieved. The level of evidence, according to Oxford EBM Centre Levels of Evidence Scale was classified for all studies. Experimental studies underwent additional quality reporting assessment. RESULTS: The screening and evaluation process resulted in twenty-one studies from 1,940 articles found, with 10 of them being experimental studies. None of the articles provided level 1 evidence, and only one study was considered high quality following quality assessment. Twenty-six different OSS programs were described in the included studies of which ten were used for image visualization, five were used for healthcare records management, four were used for educations processes, one was used for remote consultation and simulation, and six were used for general purposes. CONCLUSIONS: Our analysis revealed that the dental literature on OSS consists of scarce, incomplete, and methodologically low quality information.

Ease of adoption of clinical natural language processing software: An evaluation of five systems.

OBJECTIVE: In recognition of potential barriers that may inhibit the widespread adoption of biomedical software, the 2014 i2b2 Challenge introduced a special track, Track 3 - Software Usability Assessment, in order to develop a better understanding of the adoption issues that might be associated with the state-of-the-art clinical NLP systems. This paper reports the ease of adoption assessment methods we developed for this track, and the results of evaluating five clinical NLP system submissions. MATERIALS AND METHODS: A team of human evaluators performed a series of scripted adoptability test tasks with each of the participating systems. The evaluation team consisted of four "expert evaluators" with training in computer science, and eight "end user evaluators" with mixed backgrounds in medicine, nursing, pharmacy, and health informatics. We assessed how easy it is to adopt the submitted systems along the following three dimensions: communication effectiveness (i.e., how effective a system is in communicating its designed objectives to intended audience), effort required to install, and effort required to use. We used a formal software usability testing tool, TURF, to record the evaluators' interactions with the systems and 'think-aloud' data revealing their thought processes when installing and using the systems and when resolving unexpected issues. RESULTS: Overall, the ease of adoption ratings that the five systems received are unsatisfactory. Installation of some of the systems proved to be rather difficult, and some systems failed to adequately communicate their designed objectives to intended adopters. Further, the average ratings provided by the end user evaluators on ease of use and ease of interpreting output are -0.35 and -0.53, respectively, indicating that this group of users generally deemed the systems extremely difficult to work with. While the ratings provided by the expert evaluators are higher, 0.6 and 0.45, respectively, these ratings are still low indicating that they also experienced considerable struggles. DISCUSSION: The results of the Track 3 evaluation show that the adoptability of the five participating clinical NLP systems has a great margin for improvement. Remedy strategies suggested by the evaluators included (1) more detailed and operation system specific use instructions; (2) provision of more pertinent onscreen feedback for easier diagnosis of problems; (3) including screen walk-throughs in use instructions so users know what to expect and what might have gone wrong; (4) avoiding jargon and acronyms in materials intended for end users; and (5) packaging prerequisites required within software distributions so that prospective adopters of the software do not have to obtain each of the third-party components on their own.

Using a 3D virtual supermarket to measure food purchase behavior: a validation study.

BACKGROUND: There is increasing recognition that supermarkets are an important environment for health-promoting interventions such as fiscal food policies or front-of-pack nutrition labeling. However, due to the complexities of undertaking such research in the real world, well-designed randomized controlled trials on these kinds of interventions are lacking. The Virtual Supermarket is a 3-dimensional computerized research environment designed to enable experimental studies in a supermarket setting without the complexity or costs normally associated with undertaking such research. OBJECTIVE: The primary objective was to validate the Virtual Supermarket by comparing virtual and real-life food purchasing behavior. A secondary objective was to obtain participant feedback on perceived sense of "presence" (the subjective experience of being in one place or environment even if physically located in another) in the Virtual Supermarket. METHODS: Eligible main household shoppers (New Zealand adults aged ≥18 years) were asked to conduct 3 shopping occasions in the Virtual Supermarket over 3 consecutive weeks, complete the validated Presence Questionnaire Items Stems, and collect their real supermarket grocery till receipts for that same period. Proportional expenditure (NZ$) and the proportion of products purchased over 18 major food groups were compared between the virtual and real supermarkets. Data were analyzed using repeated measures mixed models. RESULTS: A total of 123 participants consented to take part in the study. In total, 69.9% (86/123) completed 1 shop in the Virtual Supermarket, 64.2% (79/123) completed 2 shops, 60.2% (74/123) completed 3 shops, and 48.8% (60/123) returned their real supermarket till receipts. The 4 food groups with the highest relative expenditures were the same for the virtual and real supermarkets: fresh fruit and vegetables (virtual estimate: 14.3%; real: 17.4%), bread and bakery (virtual: 10.0%; real: 8.2%), dairy (virtual: 19.1%; real: 12.6%), and meat and fish (virtual: 16.5%; real: 16.8%). Significant differences in proportional expenditures were observed for 6 food groups, with largest differences (virtual - real) for dairy (in expenditure 6.5%, P<.001; in items 2.2%, P=.04) and fresh fruit and vegetables (in expenditure: -3.1%, P=.04; in items: 5.9%, P=.002). There was no trend of overspending in the Virtual Supermarket and participants experienced a medium-to-high presence (88%, 73/83 scored medium; 8%, 7/83 scored high). CONCLUSIONS: Shopping patterns in the Virtual Supermarket were comparable to those in real life. Overall, the Virtual Supermarket is a valid tool to measure food purchasing behavior. Nevertheless, it is important to improve the functionality of some food categories, in particular fruit and vegetables and dairy. The results of this validation will assist in making further improvements to the software and with optimization of the internal and external validity of this innovative methodology.

Comparison of consistency in acute stroke: Automated ASPECT score software tools and manual consensus scores.

PURPOSE: The Alberta Stroke Program Early Computed Tomography Score (ASPECTS) is a standardized semi-quantitative scoring system. It is widely used to assess the extent of acute ischemic stroke (AIS). We evaluated the consistency of three automatic software packages with the overall and region-specific manual ASPECTS scores of AIS patients. METHODS: Retrospectively, we gathered patients who presented with stroke symptoms between February 2019 and June 2022, and 174 cases were eventually included in the trial. Two radiologists reviewed the NCCT images independently; After four weeks, the same two radiologists began randomly reviewing the DWI images, discussed different scores and give consistent results as ground truth. RESULTS: Median ASPECTS of the expert consensus reading was 7 (5-9). Good to excellent correlation of ASPECTS total scores among the three software tools (0.70, 0.74 and 0.83). Correlation among ground truth and Rapid-ASPECTS, RealNow-ASPECTS, ShuKun-ASPECTS (ICC = 0.51, Cronbach's α = 0.53), (ICC = 0.60, Cronbach's α = 0.70) and (ICC = 0.52, Cronbach's α = 0.64) respectively. The AUCs for Rapid, RealNow and ShuKun were 0.61, 0.67, and 0.62 respectively. The region-specific results showed a poor to good correlation. The correlations between the non-dominant and dominant cerebral hemispheres and the ground truth were statistically different (P < 0.05). CONCLUSIONS: Overall, the scoring consistency between the three automated scoring software and the ground truth is comparable, with RealNow-ASPECTS being no less consistent and effective than Rapid-ASPECTS and ShuKun-ASPECTS, and even better than both. But the consistency grade that still is developable.

Effect of emphysema on AI software and human reader performance in lung nodule detection from low-dose chest CT.

BACKGROUND: Emphysema influences the appearance of lung tissue in computed tomography (CT). We evaluated whether this affects lung nodule detection by artificial intelligence (AI) and human readers (HR). METHODS: Individuals were selected from the "Lifelines" cohort who had undergone low-dose chest CT. Nodules in individuals without emphysema were matched to similar-sized nodules in individuals with at least moderate emphysema. AI results for nodular findings of 30-100 mm3 and 101-300 mm3 were compared to those of HR; two expert radiologists blindly reviewed discrepancies. Sensitivity and false positives (FPs)/scan were compared for emphysema and non-emphysema groups. RESULTS: Thirty-nine participants with and 82 without emphysema were included (n = 121, aged 61 ± 8 years (mean ± standard deviation), 58/121 males (47.9%)). AI and HR detected 196 and 206 nodular findings, respectively, yielding 109 concordant nodules and 184 discrepancies, including 118 true nodules. For AI, sensitivity was 0.68 (95% confidence interval 0.57-0.77) in emphysema versus 0.71 (0.62-0.78) in non-emphysema, with FPs/scan 0.51 and 0.22, respectively (p = 0.028). For HR, sensitivity was 0.76 (0.65-0.84) and 0.80 (0.72-0.86), with FPs/scan of 0.15 and 0.27 (p = 0.230). Overall sensitivity was slightly higher for HR than for AI, but this difference disappeared after the exclusion of benign lymph nodes. FPs/scan were higher for AI in emphysema than in non-emphysema (p = 0.028), while FPs/scan for HR were higher than AI for 30-100 mm3 nodules in non-emphysema (p = 0.009). CONCLUSIONS: AI resulted in more FPs/scan in emphysema compared to non-emphysema, a difference not observed for HR. RELEVANCE STATEMENT: In the creation of a benchmark dataset to validate AI software for lung nodule detection, the inclusion of emphysema cases is important due to the additional number of FPs. KEY POINTS: • The sensitivity of nodule detection by AI was similar in emphysema and non-emphysema. • AI had more FPs/scan in emphysema compared to non-emphysema. • Sensitivity and FPs/scan by the human reader were comparable for emphysema and non-emphysema. • Emphysema and non-emphysema representation in benchmark dataset is important for validating AI.