Full-colour 3D holographic augmented-reality displays with metasurface waveguides.

Emerging spatial computing systems seamlessly superimpose digital information on the physical environment observed by a user, enabling transformative experiences across various domains, such as entertainment, education, communication and training1-3. However, the widespread adoption of augmented-reality (AR) displays has been limited due to the bulky projection optics of their light engines and their inability to accurately portray three-dimensional (3D) depth cues for virtual content, among other factors4,5. Here we introduce a holographic AR system that overcomes these challenges using a unique combination of inverse-designed full-colour metasurface gratings, a compact dispersion-compensating waveguide geometry and artificial-intelligence-driven holography algorithms. These elements are co-designed to eliminate the need for bulky collimation optics between the spatial light modulator and the waveguide and to present vibrant, full-colour, 3D AR content in a compact device form factor. To deliver unprecedented visual quality with our prototype, we develop an innovative image formation model that combines a physically accurate waveguide model with learned components that are automatically calibrated using camera feedback. Our unique co-design of a nanophotonic metasurface waveguide and artificial-intelligence-driven holographic algorithms represents a significant advancement in creating visually compelling 3D AR experiences in a compact wearable device.

Evaluating GPT-V4 (GPT-4 with Vision) on Detection of Radiologic Findings on Chest Radiographs.

Background Generating radiologic findings from chest radiographs is pivotal in medical image analysis. The emergence of OpenAI's generative pretrained transformer, GPT-4 with vision (GPT-4V), has opened new perspectives on the potential for automated image-text pair generation. However, the application of GPT-4V to real-world chest radiography is yet to be thoroughly examined. Purpose To investigate the capability of GPT-4V to generate radiologic findings from real-world chest radiographs. Materials and Methods In this retrospective study, 100 chest radiographs with free-text radiology reports were annotated by a cohort of radiologists, two attending physicians and three residents, to establish a reference standard. Of 100 chest radiographs, 50 were randomly selected from the National Institutes of Health (NIH) chest radiographic data set, and 50 were randomly selected from the Medical Imaging and Data Resource Center (MIDRC). The performance of GPT-4V at detecting imaging findings from each chest radiograph was assessed in the zero-shot setting (where it operates without prior examples) and few-shot setting (where it operates with two examples). Its outcomes were compared with the reference standard with regards to clinical conditions and their corresponding codes in the International Statistical Classification of Diseases, Tenth Revision (ICD-10), including the anatomic location (hereafter, laterality). Results In the zero-shot setting, in the task of detecting ICD-10 codes alone, GPT-4V attained an average positive predictive value (PPV) of 12.3%, average true-positive rate (TPR) of 5.8%, and average F1 score of 7.3% on the NIH data set, and an average PPV of 25.0%, average TPR of 16.8%, and average F1 score of 18.2% on the MIDRC data set. When both the ICD-10 codes and their corresponding laterality were considered, GPT-4V produced an average PPV of 7.8%, average TPR of 3.5%, and average F1 score of 4.5% on the NIH data set, and an average PPV of 10.9%, average TPR of 4.9%, and average F1 score of 6.4% on the MIDRC data set. With few-shot learning, GPT-4V showed improved performance on both data sets. When contrasting zero-shot and few-shot learning, there were improved average TPRs and F1 scores in the few-shot setting, but there was not a substantial increase in the average PPV. Conclusion Although GPT-4V has shown promise in understanding natural images, it had limited effectiveness in interpreting real-world chest radiographs. © RSNA, 2024 Supplemental material is available for this article.

Error-compensation network for ringing artifact reduction in holographic displays.

Recent advances in learning-based computer-generated holography (CGH) have unlocked novel possibilities for crafting phase-only holograms. However, existing approaches primarily focus on the learning ability of network modules, often neglecting the impact of diffraction propagation models. The resulting ringing artifacts, emanating from the Gibbs phenomenon in the propagation model, can degrade the quality of reconstructed holographic images. To this end, we explore a diffraction propagation error-compensation network that can be easily integrated into existing CGH methods. This network is designed to correct propagation errors by predicting residual values, thereby aligning the diffraction process closely with an ideal state and easing the learning burden of the network. Simulations and optical experiments demonstrate that our method, when applied to state-of-the-art HoloNet and CCNN, achieves PSNRs of up to 32.47 dB and 29.53 dB, respectively, surpassing baseline methods by 3.89 dB and 0.62 dB. Additionally, real-world experiments have confirmed a significant reduction in ringing artifacts. We envision this approach being applied to a variety of CGH algorithms, paving the way for improved holographic displays.

The expression and clinical significance of CFAP65 in colon cancer.

BACKGROUND: CFAP65 (cilia and flagella associated protein 65) is a fundamental protein in the development and formation of ciliated flagella, but few studies have focused on its role in cancer. This study aimed to investigate the prognostic significance of CFAP65 in colon cancer. METHODS: The functionally enriched genes related to CFAP65 were analyzed through the Gene Ontology (GO) database. Subsequently, CFAP65 expression levels in colon cancer were evaluated by reverse transcription and quantitative polymerase chain reaction (RT-qPCR) and immunoblotting in 20 pairs of frozen samples, including tumors and their matched paratumor tissue. Furthermore, protein expression of CFAP65 in 189 colon cancer patients were assessed via immunohistochemical staining. The correlations between CFAP65 expression and clinical features as well as long-term survival were statistically analyzed. RESULTS: CFAP65-related genes are significantly enriched on cellular processes of cell motility, ion channels, and GTPase-associated signaling. The expression of CFAP65 was significantly higher in colon cancer tissue compared to paratumor tissue. The proportion of high expression and low expression of CFAP65 in the clinical samples of colon cancer were 61.9% and 38.1%, respectively, and its expression level was not associated with the clinical parameters including gender, age, tumor location, histological differentiation, tumor stage, vascular invasion and mismatch repair deficiency. The five-year disease-free survival rate of the patients with CFAP65 low expression tumors was significantly lower than that those with high expression tumors (56.9% vs. 72.6%, P = 0.03), but the overall survival rate has no significant difference (69% vs. 78.6%, P = 0.171). The cox hazard regression analysis model showed that CFAP65 expression, tumor stage and tumor location were independent prognostic factors. CONCLUSIONS: In conclusion, we demonstrate CFAP65 is a potential predictive marker for tumor progression in colon cancer.

Development and Implementation of a Multicenter Registry for Resuscitation-Focused Transesophageal Echocardiography.

STUDY OBJECTIVE: To evaluate the clinical effect, safety, and clinical outcomes of focused transesophageal echocardiography (TEE) in the evaluation of critically ill patients in the emergency department (ED) and ICUs. METHODS: We established a prospective, multicenter, observational registry involving adult critically ill patients in whom focused TEE was performed for evaluation of out-of-hospital cardiac arrest (OHCA), inhospital cardiac arrest, evaluation of undifferentiated shock, hemodynamic monitoring, and/or procedural guidance in the ED, ICU, or operating room setting. The primary objective of the current investigation was to evaluate the clinical influence and safety of focused, point-of-care TEE in critically ill patients. Data elements included patient and procedure characteristics, laboratory values, timing of interventions, clinical outcomes, and TEE video images. RESULTS: A total of 1,045 focused TEE studies were collected among 916 patients from 28 hospitals, including 585 (64%) intraarrest and postarrest OHCA and inhospital cardiac arrest, 267 (29%) initial evaluation of undifferentiated shock, 101 (11%) procedural guidance, and 92 (10%) hemodynamic monitoring. TEE changed management in 85% of patients with undifferentiated shock, 71% of patients with inhospital cardiac arrest, and 62% of patients with OHCA. There were no reported esophageal perforations or oropharyngeal injuries, and other procedural complications were rare. CONCLUSIONS: A prospective, multicenter, and multidisciplinary TEE registry was successfully implemented, and demonstrated that focused TEE is safe and clinically impactful across multiple critical care applications. Further studies from this research network will accelerate the development of outcome-oriented research and knowledge translation on the use of TEE in emergency and critical care settings.

A survey of recent methods for addressing AI fairness and bias in biomedicine.

OBJECTIVES: Artificial intelligence (AI) systems have the potential to revolutionize clinical practices, including improving diagnostic accuracy and surgical decision-making, while also reducing costs and manpower. However, it is important to recognize that these systems may perpetuate social inequities or demonstrate biases, such as those based on race or gender. Such biases can occur before, during, or after the development of AI models, making it critical to understand and address potential biases to enable the accurate and reliable application of AI models in clinical settings. To mitigate bias concerns during model development, we surveyed recent publications on different debiasing methods in the fields of biomedical natural language processing (NLP) or computer vision (CV). Then we discussed the methods, such as data perturbation and adversarial learning, that have been applied in the biomedical domain to address bias. METHODS: We performed our literature search on PubMed, ACM digital library, and IEEE Xplore of relevant articles published between January 2018 and December 2023 using multiple combinations of keywords. We then filtered the result of 10,041 articles automatically with loose constraints, and manually inspected the abstracts of the remaining 890 articles to identify the 55 articles included in this review. Additional articles in the references are also included in this review. We discuss each method and compare its strengths and weaknesses. Finally, we review other potential methods from the general domain that could be applied to biomedicine to address bias and improve fairness. RESULTS: The bias of AIs in biomedicine can originate from multiple sources such as insufficient data, sampling bias and the use of health-irrelevant features or race-adjusted algorithms. Existing debiasing methods that focus on algorithms can be categorized into distributional or algorithmic. Distributional methods include data augmentation, data perturbation, data reweighting methods, and federated learning. Algorithmic approaches include unsupervised representation learning, adversarial learning, disentangled representation learning, loss-based methods and causality-based methods.

Leveraging generative AI for clinical evidence synthesis needs to ensure trustworthiness.

Evidence-based medicine promises to improve the quality of healthcare by empowering medical decisions and practices with the best available evidence. The rapid growth of medical evidence, which can be obtained from various sources, poses a challenge in collecting, appraising, and synthesizing the evidential information. Recent advancements in generative AI, exemplified by large language models, hold promise in facilitating the arduous task. However, developing accountable, fair, and inclusive models remains a complicated undertaking. In this perspective, we discuss the trustworthiness of generative AI in the context of automated summarization of medical evidence.

Identifying social determinants of health from clinical narratives: A study of performance, documentation ratio, and potential bias.

OBJECTIVE: To develop a natural language processing (NLP) package to extract social determinants of health (SDoH) from clinical narratives, examine the bias among race and gender groups, test the generalizability of extracting SDoH for different disease groups, and examine population-level extraction ratio. METHODS: We developed SDoH corpora using clinical notes identified at the University of Florida (UF) Health. We systematically compared 7 transformer-based large language models (LLMs) and developed an open-source package - SODA (i.e., SOcial DeterminAnts) to facilitate SDoH extraction from clinical narratives. We examined the performance and potential bias of SODA for different race and gender groups, tested the generalizability of SODA using two disease domains including cancer and opioid use, and explored strategies for improvement. We applied SODA to extract 19 categories of SDoH from the breast (n = 7,971), lung (n = 11,804), and colorectal cancer (n = 6,240) cohorts to assess patient-level extraction ratio and examine the differences among race and gender groups. RESULTS: We developed an SDoH corpus using 629 clinical notes of cancer patients with annotations of 13,193 SDoH concepts/attributes from 19 categories of SDoH, and another cross-disease validation corpus using 200 notes from opioid use patients with 4,342 SDoH concepts/attributes. We compared 7 transformer models and the GatorTron model achieved the best mean average strict/lenient F1 scores of 0.9122 and 0.9367 for SDoH concept extraction and 0.9584 and 0.9593 for linking attributes to SDoH concepts. There is a small performance gap (∼4%) between Males and Females, but a large performance gap (>16 %) among race groups. The performance dropped when we applied the cancer SDoH model to the opioid cohort; fine-tuning using a smaller opioid SDoH corpus improved the performance. The extraction ratio varied in the three cancer cohorts, in which 10 SDoH could be extracted from over 70 % of cancer patients, but 9 SDoH could be extracted from less than 70 % of cancer patients. Individuals from the White and Black groups have a higher extraction ratio than other minority race groups. CONCLUSIONS: Our SODA package achieved good performance in extracting 19 categories of SDoH from clinical narratives. The SODA package with pre-trained transformer models is available at https://github.com/uf-hobi-informatics-lab/SODA_Docker.

Neoadjuvant chemotherapy with capecitabine combined with oxaliplatin for mid-low locally advanced rectal cancer with negative mesorectal fascia: Long-term outcomes of a prospective trial (PKUCH-R03 trial).

Objective: To evaluate the safety and efficacy of neoadjuvant chemotherapy (NCT) in mid-low locally advanced rectal cancer with negative mesorectal fascia (MRF). Methods: This prospective, single-arm phase II trial was designed and conducted at Peking University Cancer Hospital. The patients who provided consent received 3 months of NCT (capecitabine and oxaliplatin, CapOX) followed by total mesorectal excision (TME). The primary endpoint was the rate of pathological complete response (pCR). Results: From January 2019 through December 2021, a total of 53 patients were enrolled, 7.5% of whom experienced grade 3-4 adverse events during NCT. The pCR rate was 17.0% for the entire cohort, and the overall rate of postoperative complications was 37.7% (1.9% of grade IIIa patients). The 3-year disease-free survival rate was 91.4%, and 23.5% (12/51) of the patients suffered from major low anterior resection syndrome (LARS). Postoperative complications were independently associated with major LARS. Conclusions: For patients with mid-low rectal cancer with negative MRF, 3 months of NCT were found to yield a favorable tumor response with acceptable toxicity. With fair long-term survival, the NCT regimen could be associated with low rates of perioperative complications as well as acceptable anal function.

Exploring angular-steering illumination-based eyebox expansion for holographic displays.

Holography represents an enabling technology for next-generation virtual and augmented reality systems. However, it remains challenging to achieve both wide field of view and large eyebox at the same time for holographic near-eye displays, mainly due to the essential étendue limitation of existing hardware. In this work, we present an approach to expanding the eyebox for holographic displays without compromising their underlying field of view. This is achieved by utilizing a compact 2D steering mirror to deliver angular-steering illumination beams onto the spatial light modulator in alignment with the viewer's eye movements. To facilitate the same image for the virtual objects perceived by the viewer when the eye moves, we explore an off-axis computational hologram generation scheme. Two bench-top holographic near-eye display prototypes with the proposed angular-steering scheme are developed, and they successfully showcase an expanded eyebox up to 8 mm × 8 mm for both VR- and AR-modes, as well as the capability of representing multi-depth holographic images.

Investigating learning-empowered hologram generation for holographic displays with ill-tuned hardware.

Existing computational holographic displays often suffer from limited reconstruction image quality mainly due to ill-conditioned optics hardware and hologram generation software. In this Letter, we develop an end-to-end hardware-in-the-loop approach toward high-quality hologram generation for holographic displays. Unlike other hologram generation methods using ideal wave propagation, ours can reduce artifacts introduced by both the light propagation model and the hardware setup, in particular non-uniform illumination. Experimental results reveal that, compared with classical computer-generated hologram algorithm counterparts, better quality of holographic images can be delivered without a strict requirement on both the fine assembly of optical components and the good uniformity of laser sources.

Comparison of State-of-the-Art Neural Network Survival Models with the Pooled Cohort Equations for Cardiovascular Disease Risk Prediction.

BACKGROUND: The Pooled Cohort Equations (PCEs) are race- and sex-specific Cox proportional hazards (PH)-based models used for 10-year atherosclerotic cardiovascular disease (ASCVD) risk prediction with acceptable discrimination. In recent years, neural network models have gained increasing popularity with their success in image recognition and text classification. Various survival neural network models have been proposed by combining survival analysis and neural network architecture to take advantage of the strengths from both. However, the performance of these survival neural network models compared to each other and to PCEs in ASCVD prediction is unknown. METHODS: In this study, we used 6 cohorts from the Lifetime Risk Pooling Project (with 5 cohorts as training/internal validation and one cohort as external validation) and compared the performance of the PCEs in 10-year ASCVD risk prediction with an all two-way interactions Cox PH model (Cox PH-TWI) and three state-of-the-art neural network survival models including Nnet-survival, Deepsurv, and Cox-nnet. For all the models, we used the same 7 covariates as used in the PCEs. We fitted each of the aforementioned models in white females, white males, black females, and black males, respectively. We evaluated models' internal and external discrimination power and calibration. RESULTS: The training/internal validation sample comprised 23216 individuals. The average age at baseline was 57.8 years old (SD = 9.6); 16% developed ASCVD during average follow-up of 10.50 (SD = 3.02) years. Based on 10 × 10 cross-validation, the method that had the highest C-statistics was Deepsurv (0.7371) for white males, Deepsurv and Cox PH-TWI (0.7972) for white females, PCE (0.6981) for black males, and Deepsurv (0.7886) for black females. In the external validation dataset, Deepsurv (0.7032), Cox-nnet (0.7282), PCE (0.6811), and Deepsurv (0.7316) had the highest C-statistics for white male, white female, black male, and black female population, respectively. Calibration plots showed that in 10 × 10 validation, all models had good calibration in all race and sex groups. In external validation, all models overestimated the risk for 10-year ASCVD. CONCLUSIONS: We demonstrated the use of the state-of-the-art neural network survival models in ASCVD risk prediction. Neural network survival models had similar if not superior discrimination and calibration compared to PCEs.

A scoping review on multimodal deep learning in biomedical images and texts.

OBJECTIVE: Computer-assisted diagnostic and prognostic systems of the future should be capable of simultaneously processing multimodal data. Multimodal deep learning (MDL), which involves the integration of multiple sources of data, such as images and text, has the potential to revolutionize the analysis and interpretation of biomedical data. However, it only caught researchers' attention recently. To this end, there is a critical need to conduct a systematic review on this topic, identify the limitations of current work, and explore future directions. METHODS: In this scoping review, we aim to provide a comprehensive overview of the current state of the field and identify key concepts, types of studies, and research gaps with a focus on biomedical images and texts joint learning, mainly because these two were the most commonly available data types in MDL research. RESULT: This study reviewed the current uses of multimodal deep learning on five tasks: (1) Report generation, (2) Visual question answering, (3) Cross-modal retrieval, (4) Computer-aided diagnosis, and (5) Semantic segmentation. CONCLUSION: Our results highlight the diverse applications and potential of MDL and suggest directions for future research in the field. We hope our review will facilitate the collaboration of natural language processing (NLP) and medical imaging communities and support the next generation of decision-making and computer-assisted diagnostic system development.

Representing and utilizing clinical textual data for real world studies: An OHDSI approach.

Clinical documentation in electronic health records contains crucial narratives and details about patients and their care. Natural language processing (NLP) can unlock the information conveyed in clinical notes and reports, and thus plays a critical role in real-world studies. The NLP Working Group at the Observational Health Data Sciences and Informatics (OHDSI) consortium was established to develop methods and tools to promote the use of textual data and NLP in real-world observational studies. In this paper, we describe a framework for representing and utilizing textual data in real-world evidence generation, including representations of information from clinical text in the Observational Medical Outcomes Partnership (OMOP) Common Data Model (CDM), the workflow and tools that were developed to extract, transform and load (ETL) data from clinical notes into tables in OMOP CDM, as well as current applications and specific use cases of the proposed OHDSI NLP solution at large consortia and individual institutions with English textual data. Challenges faced and lessons learned during the process are also discussed to provide valuable insights for researchers who are planning to implement NLP solutions in real-world studies.

Investigating deep optics model representation in affecting resolved all-in-focus image quality and depth estimation fidelity.

The end-to-end (E2E) optimization of optics and image processing, dubbed deep optics, has renewed the state-of-the-art in various computer vision tasks. However, specifying the proper model representation or parameterization of the optical elements remains elusive. This article comprehensibly investigates three modeling hypotheses of the phase coded-aperture imaging under a representative context of deep optics, joint all-in-focus (AiF) imaging and monocular depth estimation (MDE). Specifically, we analyze the respective trade-off of these models and provide insights into relevant domain-specific requirements, explore the connection between the spatial feature of the point spread function (PSF) and the performance trade-off between the AiF and MDE tasks, and discuss the model sensitivity to possible fabrication errors. This study provides new prospects for future deep optics designs, particularly those aiming for AiF and/or MDE.

Trustworthy assertion classification through prompting.

Accurate identification of the presence, absence or possibility of relevant entities in clinical notes is important for healthcare professionals to quickly understand crucial clinical information. This introduces the task of assertion classification - to correctly identify the assertion status of an entity in the unstructured clinical notes. Recent rule-based and machine-learning approaches suffer from labor-intensive pattern engineering and severe class bias toward majority classes. To solve this problem, in this study, we propose a prompt-based learning approach, which treats the assertion classification task as a masked language auto-completion problem. We evaluated the model on six datasets. Our prompt-based method achieved a micro-averaged F-1 of 0.954 on the i2b2 2010 assertion dataset, with ∼1.8% improvements over previous works. In particular, our model showed excellence in detecting classes with few instances (few-shot). Evaluations on five external datasets showcase the outstanding generalizability of the prompt-based method to unseen data. To examine the rationality of our model, we further introduced two rationale faithfulness metrics: comprehensiveness and sufficiency. The results reveal that compared to the "pre-train, fine-tune" procedure, our prompt-based model has a stronger capability of identifying the comprehensive (∼63.93%) and sufficient (∼11.75%) linguistic features from free text. We further evaluated the model-agnostic explanations using LIME. The results imply a better rationale agreement between our model and human beings (∼71.93% in average F-1), which demonstrates the superior trustworthiness of our model.

All-day thin-lens computational imaging with scene-specific learning recovery.

Modern imaging optics ensures high-quality photography at the cost of a complex optical form factor that deviates from the portability. The drastic development of image processing algorithms, especially advanced neural networks, shows great promise to use thin optics but still faces the challenges of residual artifacts and chromatic aberration. In this work, we investigate photorealistic thin-lens imaging that paves the way to actual applications by exploring several fine-tunes. Notably, to meet all-day photography demands, we develop a scene-specific generative-adversarial-network-based learning strategy and develop an integral automatic acquisition and processing pipeline. Color fringe artifacts are reduced by implementing a chromatic aberration pre-correction trick. Our method outperforms existing thin-lens imaging work with better visual perception and excels in both normal-light and low-light scenarios.

Deep learning multi-shot 3D localization microscopy using hybrid optical-electronic computing.

Current 3D localization microscopy approaches are fundamentally limited in their ability to image thick, densely labeled specimens. Here, we introduce a hybrid optical-electronic computing approach that jointly optimizes an optical encoder (a set of multiple, simultaneously imaged 3D point spread functions) and an electronic decoder (a neural-network-based localization algorithm) to optimize 3D localization performance under these conditions. With extensive simulations and biological experiments, we demonstrate that our deep-learning-based microscope achieves significantly higher 3D localization accuracy than existing approaches, especially in challenging scenarios with high molecular density over large depth ranges.

Unfiltered holography: optimizing high diffraction orders without optical filtering for compact holographic displays.

Computer-generated holography suffers from high diffraction orders (HDOs) created from pixelated spatial light modulators, which must be optically filtered using bulky optics. Here, we develop an algorithmic framework for optimizing HDOs without optical filtering to enable compact holographic displays. We devise a wave propagation model of HDOs and use it to optimize phase patterns, which allows HDOs to contribute to forming the image instead of creating artifacts. The proposed method significantly outperforms previous algorithms in an unfiltered holographic display prototype.

Foveated light-field display and real-time rendering for virtual reality.

Glasses-free light field displays have significantly progressed due to advances in high-resolution microdisplays and high-end graphics processing units (GPUs). However, for near-eye light-field displays requiring portability, the fundamental trade-off regarding achieved spatial resolution remains: retinal blur quality must be degraded; otherwise, computational consumption increases. This has prevented synthesizing the high-quality light field from being fast. By integrating off-the-shelf gaze tracking modules into near-eye light-field displays, we present wearable virtual reality prototypes supporting human visual system-oriented focus cues. An optimized, foveated light field is delivered to each eye subject to the gaze point, providing more natural visual experiences than state-of-the-art solutions. Importantly, the factorization runtime can be immensely reduced, since the image resolution is only high within the gaze cone. In addition, we demonstrate significant improvements in computation and retinal blur quality over counterpart near-eye displays.