EXSCLAIM!: Harnessing materials science literature for self-labeled microscopy datasets.

This work introduces the EXSCLAIM! toolkit for the automatic extraction, separation, and caption-based natural language annotation of images from scientific literature. EXSCLAIM! is used to show how rule-based natural language processing and image recognition can be leveraged to construct an electron microscopy dataset containing thousands of keyword-annotated nanostructure images. Moreover, it is demonstrated how a combination of statistical topic modeling and semantic word similarity comparisons can be used to increase the number and variety of keyword annotations on top of the standard annotations from EXSCLAIM! With large-scale imaging datasets constructed from scientific literature, users are well positioned to train neural networks for classification and recognition tasks specific to microscopy-tasks often otherwise inhibited by a lack of sufficient annotated training data.

Design and simulation of a snapshot multi-focal interferometric microscope.

Realizing both high temporal and spatial resolution across a large volume is a key challenge for 3D fluorescent imaging. Towards achieving this objective, we introduce an interferometric multifocus microscopy (iMFM) system, a combination of multifocus microscopy (MFM) with two opposing objective lenses. We show that the proposed iMFM is capable of simultaneously producing multiple focal plane interferometry that provides axial super-resolution and hence isotropic 3D resolution with a single exposure. We design and simulate the iMFM microscope by employing two special diffractive optical elements. The point spread function of this new iMFM microscope is simulated and the image formation model is given. For reconstruction, we use the Richardson-Lucy deconvolution algorithm with total variation regularization for 3D extended object recovery, and a maximum likelihood estimator (MLE) for single molecule tracking. A method for determining an initial axial position of the molecule is also proposed to improve the convergence of the MLE. We demonstrate both theoretically and numerically that isotropic 3D nanoscopic localization accuracy is achievable with an axial imaging range of 2um when tracking a fluorescent molecule in three dimensions and that the diffraction limited axial resolution can be improved by 3-4 times in the single shot wide-field 3D extended object recovery. We believe that iMFM will be a useful tool in 3D dynamic event imaging that requires both high temporal and spatial resolution.

Computational multifocal microscopy.

Despite recent advances, high performance single-shot 3D microscopy remains an elusive task. By introducing designed diffractive optical elements (DOEs), one is capable of converting a microscope into a 3D "kaleidoscope," in which case the snapshot image consists of an array of tiles and each tile focuses on different depths. However, the acquired multifocal microscopic (MFM) image suffers from multiple sources of degradation, which prevents MFM from further applications. We propose a unifying computational framework which simplifies the imaging system and achieves 3D reconstruction via computation. Our optical configuration omits optical elements for correcting chromatic aberrations and redesigns the multifocal grating to enlarge the tracking area. Our proposed setup features only one single grating in addition to a regular microscope. The aberration correction, along with Poisson and background denoising, are incorporated in our deconvolution-based fully-automated algorithm, which requires no empirical parameter-tuning. In experiments, we achieve spatial resolutions of 0.35um (lateral) and 0.5um (axial), which are comparable to the resolution that can be achieved with confocal deconvolution microscopy. We demonstrate a 3D video of moving bacteria recorded at 25 frames per second using our proposed computational multifocal microscopy technique.

Reduced electron exposure for energy-dispersive spectroscopy using dynamic sampling.

Analytical electron microscopy and spectroscopy of biological specimens, polymers, and other beam sensitive materials has been a challenging area due to irradiation damage. There is a pressing need to develop novel imaging and spectroscopic imaging methods that will minimize such sample damage as well as reduce the data acquisition time. The latter is useful for high-throughput analysis of materials structure and chemistry. In this work, we present a novel machine learning based method for dynamic sparse sampling of EDS data using a scanning electron microscope. Our method, based on the supervised learning approach for dynamic sampling algorithm and neural networks based classification of EDS data, allows a dramatic reduction in the total sampling of up to 90%, while maintaining the fidelity of the reconstructed elemental maps and spectroscopic data. We believe this approach will enable imaging and elemental mapping of materials that would otherwise be inaccessible to these analysis techniques.

Improved deconvolution of very weak confocal signals.

Deconvolution is typically used to sharpen fluorescence images, but when the signal-to-noise ratio is low, the primary benefit is reduced noise and a smoother appearance of the fluorescent structures. 3D time-lapse (4D) confocal image sets can be improved by deconvolution. However, when the confocal signals are very weak, the popular Huygens deconvolution software erases fluorescent structures that are clearly visible in the raw data. We find that this problem can be avoided by prefiltering the optical sections with a Gaussian blur. Analysis of real and simulated data indicates that the Gaussian blur prefilter preserves meaningful signals while enabling removal of background noise. This approach is very simple, and it allows Huygens to be used with 4D imaging conditions that minimize photodamage.

Quantitative Three-Dimensional Characterization of Block Copolymer Directed Self-Assembly on Combined Chemical and Topographical Prepatterned Templates.

Characterization of the three-dimensional (3D) structure in directed self-assembly (DSA) of block copolymers is crucial for understanding the complex relationships between the guiding template and the resulting polymer structure so DSA could be successfully implemented for advanced lithography applications. Here, we combined scanning transmission electron microscopy (STEM) tomography and coarse-grain simulations to probe the 3D structure of P2VP-b-PS-b-P2VP assembled on prepatterned templates using solvent vapor annealing. The templates consisted of nonpreferential background and raised guiding stripes that had PS-preferential top surfaces and P2VP-preferential sidewalls. The full 3D characterization allowed us to quantify the shape of the polymer domains and the interface between domains as a function of depth in the film and template geometry and offered important insights that were not accessible with 2D metrology. Sidewall guiding was advantageous in promoting the alignment and lowering the roughness of the P2VP domains over the sidewalls, but incommensurate confinement from the increased topography could cause roughness and intermittent dislocations in domains over the background region at the bottom of the film. The 3D characterization of bridge structures between domains over the background and breaks within domains on guiding lines sheds light on possible origins of common DSA defects. The positional fluctuations of the PS/P2VP interface between domains showed a depth-dependent behavior, with high levels of fluctuations near both the free surface of the film and the substrate and lower fluctuation levels in the middle of the film. This research demonstrates how 3D characterization offers a better understanding of DSA processes, leading to better design and fabrication of directing templates.

In Situ 3D Imaging of Catalysis Induced Strain in Gold Nanoparticles.

Multielectron transfer processes are crucially important in energy and biological science but require favorable catalysts to achieve fast kinetics. Nanostructuring catalysts can dramatically improve their properties, which can be difficult to understand due to strain- and size-dependent thermodynamics, the influence of defects, and substrate-dependent activities. Here, we report three-dimensional (3D) imaging of single gold nanoparticles during catalysis of ascorbic acid decomposition using Bragg coherent diffractive imaging (BCDI). Local strains were measured in single nanoparticles and modeled using reactive molecular dynamics (RMD) simulations and finite element analysis (FEA) simulations. RMD reveals the pathway for local strain generation in the gold lattice: chemisorption of hydroxyl ions. FEA reveals that the RMD results are transferable to the nanocrystal sizes studied in the experiment. Our study probes the strain-activity connection and opens a powerful avenue for theoretical and experimental studies of nanocrystal catalysis.

Effect of image compression and resolution on retinal vascular caliber.

PURPOSE: Changes in retinal vascular caliber measured from digital color fundus photographs have been independently associated with systemic outcomes in epidemiologic studies, but the effect of image resolution and compression on vascular measurements has not been previously evaluated. METHODS: To explore image compression, 40 natively digital fundus images were selected with good photo quality, high spatial resolution, and no previous image compression. Using Adobe Photoshop, these images were compressed at progressively higher levels up to 147:1, and then retinal vascular caliber was measured at each level using semiautomated software. To examine resolution, 40 fundus photographs acquired on high-resolution film were scanned with settings corresponding to 10, 7, 5, 3, and 1 megapixel fundus cameras. After adjusting for scale factor, vascular caliber was measured at each level of resolution. Data were analyzed by comparing the calculated central retinal arteriole equivalent (CRAE) and the central retinal venular equivalent (CRVE) of the original and altered images, using repeated measures ANOVA. RESULTS: CRAE became significantly wider with increasing levels of compression at the 25:1 threshold (~1 µm wider, P < 0.001) and was ~5 µm wider with 147:1 compression. CRVE also increased, but less than CRAE. Using 7 (megapixel)-MP resolution as the standard, CRVE was significantly narrower at the 5-MP simulation (~2 µm, P < 0.001) and was ~12 µm narrower at the 1-MP simulation. CRAE also decreased, but less than CRVE. CONCLUSIONS: Increasing digital image file compression and decreasing fundus image spatial resolution led to skewed measurements of the retinal vascular caliber.

Suboptimal image focus broadens retinal vessel caliber measurement.

PURPOSE: Studies have used central retinal arteriolar (CRAE) and central retinal venular (CRVE) calibers, measured from images produced with computerized image analysis, to detect risk factors for systemic diseases. The authors explored suboptimal image focus as a possible contributing factor to artificially larger vascular caliber measurements. METHODS: From the reading center image collections, 30 digital retinal images were selected for optimum quality. Image analysis software was used to derive nine progressively blurred versions of the originals. IVAN measurement software was used to measure CRAE and CRVE in the original and the blurred series derived from them. To check the adequacy of the simulation, progressively defocused series of images were taken of several volunteers. RESULTS: For CRAE, each level of simulated blurring produced a statically significant increase in apparent vessel caliber from the original (P<0.01, Wilcoxon signed rank test). For an average CRAE of 160 µm, mean broadening with minimal/moderate/severe blurring was 3 µm/12 µm/33 µm. For CRVE, every blurring level beyond the first was found to be significant (P<0.01). From an average CRVE of 200 µm, mean broadening ranged from 0 to 11 µm with minimal to severe blurring. Analysis of the progressively defocused series taken of volunteers yielded similar results overall. CONCLUSIONS: Suboptimal focus can result in erroneously larger vessel caliber measurements. Slight blurring has a minimal effect, but more severe blurring has a progressively greater effect.

Digital versus film Fundus photography for research grading of diabetic retinopathy severity.

PURPOSE: To assess agreement between digital and film photography for research classification of diabetic retinopathy severity. METHODS: Digital and film photographs from a 152-eye cohort with a full spectrum of Early Treatment Diabetic Retinopathy Study (ETDRS) severity levels were assessed for repeatability of grading within each image medium and for agreement on ETDRS discrete severity levels, ascending severity thresholds, and presence or absence of diabetic retinopathy index lesions, between digital and 35-mm slides (film). Digital photographs were color balanced to match film. RESULTS: There was substantial agreement (κ = 0.61, κ(w) [linear weighted] = 0.87) in classification of ETDRS diabetic retinopathy severity levels between digital images and film. Marginal homogeneity analyses found no significant difference in frequency distributions on the severity scale (P = 0.21, Bhapkar test). The κ results ranged from 0.72 to 0.95 for presence or absence of eight ascending diabetic retinopathy severity thresholds. Repeatability of grading between readers viewing digital images was equal to or better than that obtained with film (pair-wise interreader κ for digital images ranged from 0.47 to 0.57 and for film from 0.43 to 0.57. The κ results for identifying diabetic retinopathy lesions ranged from moderate to almost perfect. Moderate agreement of intraretinal microvascular abnormalities and venous beading between digital images and film accounted for slightly lower concordance for severity thresholds ≥47 and for slightly lower interreader agreement within digital and film images at severity thresholds ≥43 and ≥47. CONCLUSIONS: Under controlled circumstances, digital photography can equal the reliability of 35-mm slides for research classification of ETDRS severity level.

Combinatorial generation and replication-directed assembly of complex and varied geometries with thin films of diblock copolymers.

Directed assembly of fine-scale, very complex patterns with a variety of features, including terminations, jogs, disclinations, acute and obtuse bends, and sharp radii of curvature, was achieved with a symmetric poly(styrene-block-methylmethacrylate) (PS-b-PMMA) copolymer. The complex pattern was generated spontaneously by spin coating and annealing a thin film of a lamellae-forming block copolymer on a chemically neutral surface. The resulting "fingerprint" pattern had a domain spacing of 47.5 nm. Oxygen plasma treatment of the block copolymer converted it into an insoluble chemical nanopattern that was quantified by XPS, goniometry, and the wetting behavior of the block copolymer. Spin coating a second thin film of the block copolymer and annealing resulted in directed assembly that replicated the fingerprint pattern, including the most complicated defect structures. A computer vision algorithm was developed and implemented to compare the patterns quantitatively, taking into account inherent differences in image contrast, scale, rotation, and translation.

Comparison of retinal vessel measurements in digital vs film images.

PURPOSE: To compare central retinal artery equivalents (CRAE) and central retinal vein equivalents (CRVE) from vessels measured from images captured with a digital camera and digitized images taken with a film camera. DESIGN: Comparison of techniques. SETTING: Clinic. PATIENTS: A total of 75 eyes of 48 adults. OBSERVATION PROCEDURES: Gradings of retinal vessel diameters from images produced by using two photographic techniques: directly from a 45 degrees digital camera and digitized by scanning from a 30 degrees film-based camera. MAIN OUTCOME MEASURE: Differences in CRAE and CRVE from two types of images. RESULTS: CRAE and CRVE were highly comparable between the two techniques with correlation coefficients of 0.88 and 0.87, respectively. CONCLUSION: Estimates of CRAE and CRVE from measurements taken from digitally captured and digitized film-based images were highly comparable. It is likely that inferences about factors associated with retinal vessel diameters will be unbiased by the method of imaging.

Detecting progression of nuclear sclerosis by using human grading versus semiautomated computer grading.

PURPOSE: To assess indices of nuclear sclerosis derived from digitized images made from color (slide) photographs. METHODS: Film-based slit lamp images taken at baseline and at 5- and 10-year follow-up examinations of the Beaver Dam Eye Study cohort were digitized, and optical traces were taken along an axis through the center of the cornea and lens. Four indices of the severity of sclerosis were calculated based on the optical densities. The associations of the original Beaver Dam grades and these indices to age, vision, and change in severity of sclerosis over two subsequent visits were compared. RESULTS: At baseline photographs, the Spearman correlation between age and severity was 0.65 for the original film-based grading (n = 4518 right eyes) and varied between 0.46 and 0.71 for the measures from digitized images. Correlations of the indices to visual acuity were 0.38 for the film-based grading and ranged from 0.32 to 0.38 for the other indices. The authors assume that nuclear sclerosis does not regress and the percentage of regression is a reflection of error in grading. The percentage of regression and progression of sclerosis over 5- and 10-year intervals was determined for each index. After 5 years, 48.2% progressed and 4.9% regressed, using the Beaver Dam grades; progression occurred in 4.9% to 9.9%, and regression occurred in 4.5% to 7.0% for the other indices. After 10 years, 61.9% progressed and 3.2% regressed using the Beaver Dam grades; progression occurred in 8.0% to 19.7%, and regression occurred in 2.6% to 9.7% for the other indices. CONCLUSIONS: Semiautomated grading of the digitized images can be used to process thousands of images with little oversight by a trained grader. Indices of sclerosis that closely parallel human grading in their relationships to age and visual acuity can be easily computed. However, the indices appear to identify significantly less progression of nuclear sclerosis than does human grading. Further development to define a useful metric for identifying severity and progression of nuclear sclerosis is needed.

A digital video system for the automated measurement of repetitive joint motion.

Automated measurement and analysis of human motion during performance of workplace tasks are desirable for ergonomic studies. While numerous technologies exist for accurate measurement of biomechanical data, their use is often not feasible in the workplace environment. We present a digital-video based system suitable for measuring human motion of repetitive workplace tasks. Due to practical considerations, a single-camera solution is exploited by adding some control over the environment. We present an analysis of experiments demonstrating the accuracy of our system.

Repetitive motion analysis: segmentation and event classification.

Acquisition, analysis, and classification of repetitive human motion for the assessment of postural stress is of central importance to ergonomics practitioners. We present a two-threshold, multidimensional segmentation algorithm to automatically decompose a complex motion into a sequence of simple linear dynamic models. No a priori assumptions were made about the number of models that comprise the full motion or about the duration of the task cycle. A compact motion representation is obtained for each segment using parameters of a damped harmonic dynamic model. Event classification was performed using cluster analysis with the model parameters as input. Experiments demonstrate the technique on complex motion.

Epitaxial self-assembly of block copolymers on lithographically defined nanopatterned substrates.

Parallel processes for patterning densely packed nanometre-scale structures are critical for many diverse areas of nanotechnology. Thin films of diblock copolymers can self-assemble into ordered periodic structures at the molecular scale (approximately 5 to 50 nm), and have been used as templates to fabricate quantum dots, nanowires, magnetic storage media, nanopores and silicon capacitors. Unfortunately, perfect periodic domain ordering can only be achieved over micrometre-scale areas at best and defects exist at the edges of grain boundaries. These limitations preclude the use of block-copolymer lithography for many advanced applications. Graphoepitaxy, in-plane electric fields, temperature gradients, and directional solidification have also been demonstrated to induce orientation or long-range order with varying degrees of success. Here we demonstrate the integration of thin films of block copolymer with advanced lithographic techniques to induce epitaxial self-assembly of domains. The resulting patterns are defect-free, are oriented and registered with the underlying substrate and can be created over arbitrarily large areas. These structures are determined by the size and quality of the lithographically defined surface pattern rather than by the inherent limitations of the self-assembly process. Our results illustrate how hybrid strategies to nanofabrication allow for molecular level control in existing manufacturing processes.

Automated analysis of repetitive joint motion.

Automated measurement, analysis, and comparison of human motion during performance of workplace tasks or exercise therapy are core competencies required to realize many telemedicine applications. Ergonomic studies and telemonitoring of patients performing rehabilitation exercises are examples of applications that would benefit from a representation of complex human motion in a form amenable to comparison. We present a representation of joint motion suitable for the analysis of multidimensional angular joint motion time series data. Complex motion is reduced to a concatenation motion segments, where simple dynamic models approximate the observed motion on each segment. This compact representation still enables measurement of statistics familiar to ergonomics practitioners such as cycle length and task duration. An algorithm to obtain this representation from observed motion data (time series) is given. We introduce a metric, based on a kinetic energy-like measure, to compare motions. Experiments are presented to demonstrate the representation, its relationship to previous measures and the applicability of the kinetic energy metric for motion comparison.