Digitizing translucent object appearance by validating computed optical properties.

The optical properties available for an object are most often fragmented and insufficient for photorealistic rendering of the object. We propose a procedure for digitizing a translucent object with sufficient information for predictive rendering of its appearance. Based on object material descriptions, we compute optical properties and validate or adjust this object appearance model based on comparison of simulation with spectrophotometric measurements of the bidirectional scattering-surface reflectance distribution function (BSSRDF). To ease this type of comparison, we provide an efficient simulation tool that computes the BSSRDF for a particular light-view configuration. Even with just a few configurations, the localized lighting in BSSRDF measurements is useful for assessing the appropriateness of computed or otherwise acquired optical properties. To validate an object appearance model in a more common lighting environment, we render the appearance of the obtained digital twin and assess the photorealism of our renderings through pixel-by-pixel comparison with photographs of the physical object.

Surface Reconstruction from Structured Light Images Using Differentiable Rendering.

When 3D scanning objects, the objective is usually to obtain a continuous surface. However, most surface scanning methods, such as structured light scanning, yield a point cloud. Obtaining a continuous surface from a point cloud requires a subsequent surface reconstruction step, which is directly affected by any error from the computation of the point cloud. In this work, we propose a one-step approach in which we compute the surface directly from structured light images. Our method minimizes the least-squares error between photographs and renderings of a triangle mesh, where the vertex positions of the mesh are the parameters of the minimization problem. To ensure fast iterations during optimization, we use differentiable rendering, which computes images and gradients in a single pass. We present simulation experiments demonstrating that our method for computing a triangle mesh has several advantages over approaches that rely on an intermediate point cloud. Our method can produce accurate reconstructions when initializing the optimization from a sphere. We also show that our method is good at reconstructing sharp edges and that it is robust with respect to image noise. In addition, our method can improve the output from other reconstruction algorithms if we use these for initialization.

Alignment of rendered images with photographs for testing appearance models.

We propose a method for direct comparison of rendered images with a corresponding photograph in order to analyze the optical properties of physical objects and test the appropriateness of appearance models. To this end, we provide a practical method for aligning a known object and a point-like light source with the configuration observed in a photograph. Our method is based on projective transformation of object edges and silhouette matching in the image plane. To improve the similarity between rendered and photographed objects, we introduce models for spatially varying roughness and a model where the distribution of light transmitted by a rough surface influences direction-dependent subsurface scattering. Our goal is to support development toward progressive refinement of appearance models through quantitative validation.

Multi-modal deep learning for joint prediction of otitis media and diagnostic difficulty.

Objectives: In this study, we propose a diagnostic model for automatic detection of otitis media based on combined input of otoscopy images and wideband tympanometry measurements. Methods: We present a neural network-based model for the joint prediction of otitis media and diagnostic difficulty. We use the subclassifications acute otitis media and otitis media with effusion. The proposed approach is based on deep metric learning, and we compare this with the performance of a standard multi-task network. Results: The proposed deep metric approach shows good performance on both tasks, and we show that the multi-modal input increases the performance for both classification and difficulty estimation compared to the models trained on the modalities separately. An accuracy of 86.5% is achieved for the classification task, and a Kendall rank correlation coefficient of 0.45 is achieved for difficulty estimation, corresponding to a correct ranking of 72.6% of the cases. Conclusion: This study demonstrates the strengths of a multi-modal diagnostic tool using both otoscopy images and wideband tympanometry measurements for the diagnosis of otitis media. Furthermore, we show that deep metric learning improves the performance of the models.