A smartphone application for personalized facial aesthetic monitoring.

BACKGROUND: Methods available at home for capturing facial images to track changes in skin quality and evaluate skincare treatments are limited. In this study, we developed a smartphone camera application (app) for personalized facial aesthetic monitoring. MATERIALS AND METHODS: A face alignment indicators (FAIN) system utilizing facial landmark detection, an artificial intelligence technique, to estimate key facial parts, was implemented into the app to maintain a consistent facial appearance during image capture. The FAIN system is composed of a fixed target indicator and an alignment indicator that dynamically changes its shape according to the user's face position, size, and orientation. Users align their faces to match the alignment indicator with the fixed target indicator, and the image is automatically captured when alignment is achieved. RESULTS: We investigated the app's effectiveness in ensuring a consistent facial appearance by analyzing both geometric and colorimetric data. Geometric information from captured faces and colorimetric data from stickers applied to the faces were utilized. The coefficients of variation (CVs) for the L*, a*, and b* values of the stickers were higher compared to those measured by a colorimeter, with CVs of 14.9 times, 8.14 times, and 4.41 times for L*, a*, and b*, respectively. To assess the feasibility of the app for facial aesthetic monitoring, we tracked changes in pseudo-skin color on the cheek of a participant using skin-colored stickers. As a result, we observed the smallest color difference ∆Eab of 1.901, which can be considered as the experimentally validated detection limit using images acquired by the app. CONCLUSION: While the current monitoring method is a relative quantification approach, it contributes to evidence-based evaluations of skincare treatments.

Influence of Noise in Computer-Vision-Based Measurements on Parameter Identification in Structural Dynamics.

Nowadays, consumer electronics offer computer-vision-based (CV) measurements of dynamic displacements with some trade-offs between sampling frequency, resolution and low cost of the device. This study considers a consumer-grade smartphone camera based on complementary metal-oxide semiconductor (CMOS) technology and investigates the influence of its hardware limitations on the estimation of dynamic displacements, modal parameters and stiffness parameters of bolted connections in a laboratory structure. An algorithm that maximizes the zero-normalized cross-correlation function is employed to extract the dynamic displacements. The modal parameters are identified with the stochastic subspace identification method. The stiffness parameters are identified using a model-updating technique based on modal sensitivities. The results are compared with the corresponding data obtained with accelerometers and a laser distance sensor. The CV measurement allows lower-order vibration modes to be identified with a systematic (bias) error that is nearly proportional to the vibration frequency: from 2% for the first mode (9.4 Hz) to 10% for the third mode (71.4 Hz). However, the measurement errors introduced by the smartphone camera have a significantly lower influence on the values of the identified stiffness parameters than the numbers of modes and parameters taken into account. This is due to the bias-variance trade-off. The results show that consumer-grade electronics can be used as a low-cost and easy-to-use measurement tool if lower-order modes are required.

Digital Single-Image Smartphone Assessment of Total Body Fat and Abdominal Fat Using Machine Learning.

Background: Obesity is chronic health problem. Screening for the obesity phenotype is limited by the availability of practical methods. Methods: We determined the reproducibility and accuracy of an automated machine-learning method using smartphone camera-enabled capture and analysis of single, two-dimensional (2D) standing lateral digital images to estimate fat mass (FM) compared to dual X-ray absorptiometry (DXA) in females and males. We also report the first model to predict abdominal FM using 2D digital images. Results: Gender-specific 2D estimates of FM were significantly correlated (p < 0.001) with DXA FM values and not different (p > 0.05). Reproducibility of FM estimates was very high (R2 = 0.99) with high concordance (R2 = 0.99) and low absolute pure error (0.114 to 0.116 kg) and percent error (1.3 and 3%). Bland−Altman plots revealed no proportional bias with limits of agreement of 4.9 to −4.3 kg and 3.9 to −4.9 kg for females and males, respectively. A novel 2D model to estimate abdominal (lumbar 2−5) FM produced high correlations (R2 = 0.99) and concordance (R2 = 0.99) compared to DXA abdominal FM values. Conclusions: A smartphone camera trained with machine learning and automated processing of 2D lateral standing digital images is an objective and valid method to estimate FM and, with proof of concept, to determine abdominal FM. It can facilitate practical identification of the obesity phenotype in adults.

A demonstration of automated visual evaluation of cervical images taken with a smartphone camera.

We examined whether automated visual evaluation (AVE), a deep learning computer application for cervical cancer screening, can be used on cervix images taken by a contemporary smartphone camera. A large number of cervix images acquired by the commercial MobileODT EVA system were filtered for acceptable visual quality and then 7587 filtered images from 3221 women were annotated by a group of gynecologic oncologists (so the gold standard is an expert impression, not histopathology). We tested and analyzed on multiple random splits of the images using two deep learning, object detection networks. For all the receiver operating characteristics curves, the area under the curve values for the discrimination of the most likely precancer cases from least likely cases (most likely controls) were above 0.90. These results showed that AVE can classify cervix images with confidence scores that are strongly associated with expert evaluations of severity for the same images. The results on a small subset of images that have histopathologic diagnoses further supported the capability of AVE for predicting cervical precancer. We examined the associations of AVE severity score with gynecologic oncologist impression at all regions where we had a sufficient number of cases and controls, and the influence of a woman's age. The method was found generally resilient to regional variation in the appearance of the cervix. This work suggests that using AVE on smartphones could be a useful adjunct to health-worker visual assessment with acetic acid, a cervical cancer screening method commonly used in low- and middle-resource settings.

Smartphone-assisted tele-gynepathology: A pilot study.

INTRODUCTION: Traditional telepathology techniques like whole slide imaging require expensive equipment and are currently out of reach of the developing countries. However, the improvements in smartphone camera resolution and availability of faster internet have made smartphone-assisted telepathology possible. METHODS: A total of 186 cases pertaining to gynecologic pathology reported by single consultant (NT) were retrieved from the records of the histopathology department. A trained histopathologist then photographed representative areas of each case by using the smartphone camera. After a wash off period of 6 months, the images along with the clinical details were sent by Whatsapp Messenger to the same reporting pathologist. The reporting pathologist replied with the diagnosis of each case by using Whatsapp. RESULTS: The smartphone diagnosis was concordant in 179/186 (96.2%) cases. The intraobserver concordance rates varied with the organ involved - it was highest for endometrial and myometrial pathology (123/126, 97.6%) lowest for ovarian lesions (08/10, 80%). For cervical pathology, it was 97.2% (35/36) and for fallopian tube pathology it was 92.9% (13/14). CONCLUSION: Although the initial results of this pilot study are encouraging, there is a long way to go before smartphone-assisted telepathology can be put to routine use for the second opinion. More experience of the pathologists with this technique and faster internet and better smartphone cameras will further improve the concordance of smartphone-assisted telepathology diagnosis with conventional microscopy diagnosis.

Assessing the Influence of Temperature Changes on the Geometric Stability of Smartphone- and Raspberry Pi Cameras.

Knowledge about the interior and exterior camera orientation parameters is required to establish the relationship between 2D image content and 3D object data. Camera calibration is used to determine the interior orientation parameters, which are valid as long as the camera remains stable. However, information about the temporal stability of low-cost cameras due to the physical impact of temperature changes, such as those in smartphones, is still missing. This study investigates on the one hand the influence of heat dissipating smartphone components at the geometric integrity of implemented cameras and on the other hand the impact of ambient temperature changes at the geometry of uncoupled low-cost cameras considering a Raspberry Pi camera module that is exposed to controlled thermal radiation changes. If these impacts are neglected, transferring image measurements into object space will lead to wrong measurements due to high correlations between temperature and camera's geometric stability. Monte-Carlo simulation is used to simulate temperature-related variations of the interior orientation parameters to assess the extent of potential errors in the 3D data ranging from a few millimetres up to five centimetres on a target in X- and Y- direction. The target is positioned at a distance of 10 m to the camera and the Z-axis is aligned with camera's depth direction.

A DNAzyme assay coupled with effective magnetic separation and rolling circle amplification for detection of lead cations with a smartphone camera.

Rapid, portable, and efficient detection of lead cations (Pb2+) is of great significance for monitoring and evaluating environmental toxicity and human healthcare. In this work, we developed a simple and low-cost homogenous fluorescence DNAzyme assay for Pb2+ determination based on Pb2+-dependent cleaving and rolling circle amplification (RCA). DNAzyme and its substrate (5'-biotin) formed double-stranded hybrids in solution which could thoroughly react with Pb2+ in aqueous phase. Then, the unreacted DNAzyme/substrate hybrids as well as cleaved parts with biotin labeling of substrate strand would be captured by magnetic beads through biotin-streptavidin interactions and removed from the reaction solution. Meanwhile, the other parts of the substrate strand remained in solution and subsequently acted as the primer and triggered RCA. The concentration of the cleaved substrate strand correlated to that of Pb2+ and non-specific amplification was effectively minimized through biotin-streptavidin isolation. With a smartphone camera, the fluorescence intensity was recorded and quantified after 30-90 min amplification, making it a portable method with minimum instrumentation. A dynamic range of 1-100 nM of Pb2+ was achieved under optimized conditions and it was successfully employed for Pb2+ detection in spiked lake water. Graphical abstract.

Estimation of PM10 Levels and Sources in Air Quality Networks by Digital Analysis of Smartphone Camera Images Taken from Samples Deposited on Filters.

This paper explores the performance of smartphone cameras as low-cost and easily accessible tools to provide information about the levels and origin of particulate matter (PM) in ambient air. We tested the concept by digital analysis of the images of daily PM10 (particles with diameters 10 µm and smaller) samples captured on glass fibre filters by high-volume aerosol samplers at urban and rural locations belonging to the air quality monitoring network of Extremadura (Spain) for one year. The images were taken by placing the filters inside a box designed to maintain controlled and reproducible light conditions. Digital image analysis was carried out by a mobile colour-sensing application using red, green, blue/hue, saturation, value/hue, saturation, luminance (RGB/HSV/HSL) parameters, that were processed through statistical procedures, directly or transformed to greyscale. The results of the study show that digital image analysis of the filters can roughly estimate the concentration of PM10 within an air quality network, based on a significant linear correlation between the concentration of PM10 measured by an official gravimetric method and the colour parameters of the filters' images, with better results in the case of the saturation parameter (SHSV). The methodology based on digital analysis can discriminate urban and rural sampling locations affected by different local particle-emitting sources and is also able to identify the presence of remote sources such as Saharan dust outbreaks in both urban and rural locations. The proposed methodology can be considered as a useful complement to the aerosol sampling equipment of air quality network field units for a quick estimation of PM10 in the ambient air, through a simple, accessible and low-cost procedure, with further miniaturization potential.

Hybrid Indoor Localization Using IMU Sensors and Smartphone Camera.

Smartphone camera or inertial measurement unit (IMU) sensor-based systems can be independently used to provide accurate indoor positioning results. However, the accuracy of an IMU-based localization system depends on the magnitude of sensor errors that are caused by external electromagnetic noise or sensor drifts. Smartphone camera based positioning systems depend on the experimental floor map and the camera poses. The challenge in smartphone camera-based localization is that accuracy depends on the rapidness of changes in the user's direction. In order to minimize the positioning errors in both the smartphone camera and IMU-based localization systems, we propose hybrid systems that combine both the camera-based and IMU sensor-based approaches for indoor localization. In this paper, an indoor experiment scenario is designed to analyse the performance of the IMU-based localization system, smartphone camera-based localization system and the proposed hybrid indoor localization system. The experiment results demonstrate the effectiveness of the proposed hybrid system and the results show that the proposed hybrid system exhibits significant position accuracy when compared to the IMU and smartphone camera-based localization systems. The performance of the proposed hybrid system is analysed in terms of average localization error and probability distributions of localization errors. The experiment results show that the proposed oriented fast rotated binary robust independent elementary features (BRIEF)-simultaneous localization and mapping (ORB-SLAM) with the IMU sensor hybrid system shows a mean localization error of 0.1398 m and the proposed simultaneous localization and mapping by fusion of keypoints and squared planar markers (UcoSLAM) with IMU sensor-based hybrid system has a 0.0690 m mean localization error and are compared with the individual localization systems in terms of mean error, maximum error, minimum error and standard deviation of error.

Artificial Intelligence-Based Grading Quality of Bovine Blastocyst Digital Images: Direct Capture with Juxtaposed Lenses of Smartphone Camera and Stereomicroscope Ocular Lens.

In this study, we developed an online graphical and intuitive interface connected to a server aiming to facilitate professional access worldwide to those facing problems with bovine blastocysts classification. The interface Blasto3Q, where 3Q refers to the three qualities of the blastocyst grading, contains a description of 24 variables that were extracted from the image of the blastocyst and analyzed by three Artificial Neural Networks (ANNs) that classify the same loaded image. The same embryo (i.e., the biological specimen) was submitted to digital image capture by the control group (inverted microscope with 40× magnification) and the experimental group (stereomicroscope with maximum of magnification plus 4× zoom from the cell phone camera). The images obtained from the control and experimental groups were uploaded on Blasto3Q. Each image from both sources was evaluated for segmentation and submitted (only if it could be properly or partially segmented) for automatic quality grade classification by the three ANNs of the Blasto3Q program. Adjustments on the software program through the use of scaling algorithm software were performed to ensure the proper search and segmentation of the embryo in the raw images when they were captured by the smartphone, since this source produced small embryo images compared with those from the inverted microscope. With this new program, 77.8% of the images from smartphones were successfully segmented and from those, 85.7% were evaluated by the Blasto3Q in agreement with the control group.

Smartphone-Based Escalator Recognition for the Visually Impaired.

It is difficult for visually impaired individuals to recognize escalators in everyday environments. If the individuals ride on escalators in the wrong direction, they will stumble on the steps. This paper proposes a novel method to assist visually impaired individuals in finding available escalators by the use of smartphone cameras. Escalators are recognized by analyzing optical flows in video frames captured by the cameras, and auditory feedback is provided to the individuals. The proposed method was implemented on an Android smartphone and applied to actual escalator scenes. The experimental results demonstrate that the proposed method is promising for helping visually impaired individuals use escalators.

A Smartphone Camera-Based Indoor Positioning Algorithm of Crowded Scenarios with the Assistance of Deep CNN.

Considering the installation cost and coverage, the received signal strength indicator (RSSI)-based indoor positioning system is widely used across the world. However, the indoor positioning performance, due to the interference of wireless signals that are caused by the complex indoor environment that includes a crowded population, cannot achieve the demands of indoor location-based services. In this paper, we focus on increasing the signal strength estimation accuracy considering the population density, which is different to the other RSSI-based indoor positioning methods. Therefore, we propose a new wireless signal compensation model considering the population density, distance, and frequency. First of all, the number of individuals in an indoor crowded scenario can be calculated by our convolutional neural network (CNN)-based human detection approach. Then, the relationship between the population density and the signal attenuation is described in our model. Finally, we use the trilateral positioning principle to realize the pedestrian location. According to the simulation and tests in the crowded scenarios, the proposed model increases the accuracy of the signal strength estimation by 1.53 times compared to that without considering the human body. Therefore, the localization accuracy is less than 1.37 m, which indicates that our algorithm can improve the indoor positioning performance and is superior to other RSSI models.

Employing an Incentive Spirometer to Calibrate Tidal Volumes Estimated from a Smartphone Camera.

A smartphone-based tidal volume (V(T)) estimator was recently introduced by our research group, where an Android application provides a chest movement signal whose peak-to-peak amplitude is highly correlated with reference V(T) measured by a spirometer. We found a Normalized Root Mean Squared Error (NRMSE) of 14.998% ± 5.171% (mean ± SD) when the smartphone measures were calibrated using spirometer data. However, the availability of a spirometer device for calibration is not realistic outside clinical or research environments. In order to be used by the general population on a daily basis, a simple calibration procedure not relying on specialized devices is required. In this study, we propose taking advantage of the linear correlation between smartphone measurements and V(T) to obtain a calibration model using information computed while the subject breathes through a commercially-available incentive spirometer (IS). Experiments were performed on twelve (N = 12) healthy subjects. In addition to corroborating findings from our previous study using a spirometer for calibration, we found that the calibration procedure using an IS resulted in a fixed bias of -0.051 L and a RMSE of 0.189 ± 0.074 L corresponding to 18.559% ± 6.579% when normalized. Although it has a small underestimation and slightly increased error, the proposed calibration procedure using an IS has the advantages of being simple, fast, and affordable. This study supports the feasibility of developing a portable smartphone-based breathing status monitor that provides information about breathing depth, in addition to the more commonly estimated respiratory rate, on a daily basis.

AI-smartphone markerless motion capturing of hip, knee, and ankle joint kinematics during countermovement jumps.

Recently, AI-driven skeleton reconstruction tools that use multistage computer vision pipelines were designed to estimate 3D kinematics from 2D video sequences. In the present study, we validated a novel markerless, smartphone video-based artificial intelligence (AI) motion capture system for hip, knee, and ankle angles during countermovement jumps (CMJs). Eleven participants performed six CMJs. We used 2D videos created by a smartphone (Apple iPhone X, 4K, 60 fps) to create 24 different keypoints, which together built a full skeleton including joints and their connections. Body parts and skeletal keypoints were localized by calculating confidence maps using a multilevel convolutional neural network that integrated both spatial and temporal features. We calculated hip, knee, and ankle angles in the sagittal plane and compared it with the angles measured by a VICON system. We calculated the correlation between both method's angular progressions, mean squared error (MSE), mean average error (MAE), and the maximum and minimum angular error and run statistical parametric mapping (SPM) analysis. Pearson correlation coefficients (r) for hip, knee, and ankle angular progressions in the sagittal plane during the entire movement were 0.96, 0.99, and 0.87, respectively. SPM group-analysis revealed some significant differences only for ankle angular progression. MSE was below 5.7°, MAE was below 4.5°, and error for maximum amplitudes was below 3.2°. The smartphone AI motion capture system with the trained multistage computer vision pipeline was able to detect, especially hip and knee angles in the sagittal plane during CMJs with high precision from a frontal view only.

Feasibility of images acquired using smartphone camera for marginal and internal fit of fixed dental prosthesis: comparison and correlation study.

This study aimed to measure marginal and internal fit using images captured with both an optical microscope and a smartphone camera, comparing the fit measurement performance of these devices and analyzing their correlation. Working casts (with 10 posterior and 10 anterior teeth) created to fabricate fixed dental prostheses were used. These working casts were scanned using a desktop scanner (E1) to design an interim crown, and the designed interim crown was fabricated using a three-dimensional (3D) printer. Utilizing the silicone replica technique, the fabricated interim crown replicated the fit, which was then captured using both an optical microscope and a smartphone camera. The captured images were used to measure the marginal and internal fit according to the imaging device. Intraclass correlation coefficients (ICC) were used for reliability analysis according to the imaging device. Furthermore, the Wilcoxon signed-rank test was adopted for the comparative evaluation of the marginal and internal fit between the imaging devices (α = 0.05). The measurement results of the marginal and internal fit according to the optical microscope and smartphone camera did exhibit a significant difference (P < 0.05). The ICC between the two devices showed an "excellent" agreement of over 0.9 at all measurement points (P < 0.001). A smartphone camera could be used to obtain images for evaluating the marginal and internal fit.

Extraction-free analysis in cosmetics by digital image colorimetry, illustrated by the quantification of urea.

An extraction-free methodology is proposed for quantifying urea in cosmetics, which relies on urea-mediated decrease of methyl red decoloration by sodium hypochlorite. The method is applied directly to the cosmetic formulation and the resulting color intensity is captured by a smartphone camera. We demonstrate a linear relationship between color intensity and urea concentration in O/W emulsions and a shampoo. This quantification methodology is fully validated by determining its technical characteristics in an O/W cosmetic emulsion: The standard curve is linear over 2.5-30.0 % w/w urea (R2 ≥ 0.985). The coefficient of variation (CV %) on all quality control levels is ≤ 12.54 % for intermediate precision, indicating acceptable precision. Bias is up to ±4.93 % in the emulsion, indicating acceptable accuracy and a countable matrix effect. The proposed analysis setup in combination with a standard addition methodology is applied to verify urea content in purpose-made emulsions: bias is ≤±10.9 %, even in the presence of interfering ammonia. We finally demonstrate that the camera-captured color intensity of an O/W emulsion is proportional to different colorant concentrations in the formulation. This opens the route for further applications of the proposed setup to other ingredients capable of generating a colored product upon suitable reaction inside the formulation matrix.

Medical Students' Perspectives Regarding the Use of a Slit-Lamp Smartphone Adapter for Clinical Slit-Lamp Photography.

Background This study aimed to investigate medical students' perspectives regarding the ease and utility of smartphone slit-lamp photography. Methodology In this prospective experimental study, fourth and fifth-year medical students who were in or had finished ophthalmology rotation were included to attempt slit-lamp smartphone anterior segment photography on adult patients after a brief hands-on instruction course. Each medical student attempted to record five supervised slit-lamp videos of the anterior segment of five patients using the described adapter and their own smartphone. The time taken until photography was calculated for each attempt. After the fifth attempt, each medical student rated the ease of the use of this method of slit-lamp photography as well as their perspective regarding its utility as a potential means of medical education and telemedical consultations on a five-point Likert scale. Results A total of 33 medical students participated, with each successfully recording five slit-lamp examinations using their smartphones. The time used for the application of the adapter until the image capture ranged from 6 to 278 seconds (average = 39.51 ± 34.7 seconds) and markedly improved by the fifth attempt (30.5 ± 25.7 seconds) compared to the first attempt (67.3 ± 49.3 seconds). Learning this skill was perceived to be relatively easy (2.2 ± 1), with high potential in clinical education (4.6 ± 0.75) and teleconsultations (4.7 ± 0.65). Conclusions Smartphone slit-lamp photography is a relatively easy process. It can be quickly acquired by medical students and has the potential to enhance their medical education and telemedical consultation capabilities.

Optical Design of a Miniaturized 10× Periscope Zoom Lens for Smartphones.

The size of the optical zoom system is important in smartphone camera design, especially as it governs the thickness of the smartphone. We present the optical design of a miniaturized 10× periscope zoom lens for smartphones. To achieve the desired level of miniaturization, the conventional zoom lens can be replaced with a periscope zoom lens. In addition to this change in the optical design, the quality of the optical glass, which also affects the performance of the lens, must be considered. With advancements in the optical glass manufacturing process, aspheric lenses are becoming more widely used. In this study, aspheric lenses are incorporated into a design for a 10× optical zoom lens with a lens thickness of less than 6.5 mm and an eight-megapixel image sensor. Furthermore, tolerance analysis is carried out to prove its manufacturability.

Phone2SAS: 3D scanning by smartphone aids the realization of small-angle scattering.

Small-angle scattering (SAS) is a powerful tool for the detailed structural analysis of objects at the nanometer scale. In contrast to techniques such as electron microscopy, SAS data are presented as reciprocal space information, which hinders the intuitive interpretation of SAS data. This study presents a workflow: (1) creating objects, (2) 3D scanning, (3) the representation of the object as point clouds on a laptop, (4) computation of a distance distribution function, and (5) computation of SAS, executed via the computer program Phone2SAS. This enables us to realize SAS and perform the interactive modeling of SAS of the object of interest. Because 3D scanning is easily accessible through smartphones, this workflow driven by Phone2SAS contributes to the widespread use of SAS. The application of Phone2SAS for the structural assignment of SAS to Y-shaped antibodies is reported in this study.

COVID-19 Detection Using a 3D-Printed Micropipette Tip and a Smartphone.

The COVID-19 pandemic has caused over 7 million deaths worldwide and over 1 million deaths in the US as of October 15, 2022. Virus testing lags behind the level or availability necessary for pandemic events like COVID-19, especially in resource-limited settings. Here, we report a low cost, mix-and-read COVID-19 assay using a synthetic SARS-CoV-2 sensor, imaged and processed using a smartphone. The assay was optimized for saliva and employs 3D-printed micropipette tips with a layer of monoclonal anti-SARS-CoV-2 inside the tip. A polymeric sensor for SARS-CoV-2 spike (S) protein (COVRs) synthesized as a thin film on silica nanoparticles provides 3,3',5-5'-tetramethylbenzidine responsive color detection using streptavidin-poly-horseradish peroxidase (ST-poly-HRP) with 400 HRP labels per molecule. COVRs were engineered with an NHS-PEG4-biotin coating to reduce nonspecific binding and provide affinity for ST-poly-HRP labels. COVRs binds to S-proteins with binding strengths and capacities much larger than salivary proteins in 10% artificial saliva-0.01%-Triton X-100 (as virus deactivator). A limit of detection (LOD) of 200 TCID50/mL (TCID50 = tissue culture infectious dose 50%) in artificial saliva was obtained using the Color Grab smartphone app and verified using ImageJ. Viral load values obtained in 10% pooled human saliva spiked with inactivated SARS-COV-2 virus gave excellent correlation with viral loads obtained from qPCR (p = 0.0003, r = 0.99).