Development of Multimodal Fusion Technology for Tomato Maturity Assessment.

The maturity of fruits and vegetables such as tomatoes significantly impacts indicators of their quality, such as taste, nutritional value, and shelf life, making maturity determination vital in agricultural production and the food processing industry. Tomatoes mature from the inside out, leading to an uneven ripening process inside and outside, and these situations make it very challenging to judge their maturity with the help of a single modality. In this paper, we propose a deep learning-assisted multimodal data fusion technique combining color imaging, spectroscopy, and haptic sensing for the maturity assessment of tomatoes. The method uses feature fusion to integrate feature information from images, near-infrared spectra, and haptic modalities into a unified feature set and then classifies the maturity of tomatoes through deep learning. Each modality independently extracts features, capturing the tomatoes' exterior color from color images, internal and surface spectral features linked to chemical compositions in the visible and near-infrared spectra (350 nm to 1100 nm), and physical firmness using haptic sensing. By combining preprocessed and extracted features from multiple modalities, data fusion creates a comprehensive representation of information from all three modalities using an eigenvector in an eigenspace suitable for tomato maturity assessment. Then, a fully connected neural network is constructed to process these fused data. This neural network model achieves 99.4% accuracy in tomato maturity classification, surpassing single-modal methods (color imaging: 94.2%; spectroscopy: 87.8%; haptics: 87.2%). For internal and external maturity unevenness, the classification accuracy reaches 94.4%, demonstrating effective results. A comparative analysis of performance between multimodal fusion and single-modal methods validates the stability and applicability of the multimodal fusion technique. These findings demonstrate the key benefits of multimodal fusion in terms of improving the accuracy of tomato ripening classification and provide a strong theoretical and practical basis for applying multimodal fusion technology to classify the quality and maturity of other fruits and vegetables. Utilizing deep learning (a fully connected neural network) for processing multimodal data provides a new and efficient non-destructive approach for the massive classification of agricultural and food products.

Macro-micro exploration on dynamic interaction between aflatoxigenic Aspergillus flavus and maize kernels using Vis/NIR hyperspectral imaging and SEM technology.

Aspergillus flavus and its toxic metabolites-aflatoxins infect and contaminate maize kernels, posing a threat to grain safety and human health. Due to the complexity of microbial growth and metabolic processes, dynamic mechanisms among fungal growth, nutrient depletion of maize kernels and aflatoxin production is still unclear. In this study, visible/near infrared (Vis/NIR) hyperspectral imaging (HSI) combined with the scanning electron microscope (SEM) was used to elucidate the critical organismal interaction at kernel (macro-) and microscopic levels. As kernel damage is the main entrance for fungal invasion, maize kernels with gradually aggravated damages from intact to pierced to halved kernels with A. flavus were cultured for 0-120 h. The spectral fingerprints of the A. flavus-maize kernel complex over time were analyzed with principal components analysis (PCA) of hyperspectral images, where the pseudo-color score maps and the loading plots of the first three PCs were used to investigate the dynamic process of fungal infection and to capture the subtle changes in the complex with different hardness of the maize matrix. The dynamic growth process of A. flavus and the interactions of fungus-maize complexes were explained on a microscopic level using SEM. Specifically, fungus morphology, e.g., hyphae, conidia, and conidiophore (stipe) was accurately captured on the microscopic level, and the interaction process between A. flavus and nutrient loss from the maize kernel tissues (i.e., embryo, and endosperm) was described. Furthermore, the growth stage discrimination models based on PLSDA with the results of CCRC = 100 %, CCRV = 97 %, CCRIV = 93 %, and the prediction models of AFB1 based on PLSR with satisfactory performance (R2C = 0.96, R2V = 0.95, R2IV = 0.93 and RPD = 3.58) were both achieved. In conclusion, the results from both macro-level (Vis/NIR-HSI) and micro-level (SEM) assessments revealed the dynamic organismal interactions in A. flavus-maize kernel complex, and the detailed data could be used for modeling, and quantitative prediction of aflatoxin, which would establish a theoretical foundation for the early detection of fungal or toxin contaminated grains to ensure food security.

Semisupervised Deep Learning for the Detection of Foreign Materials on Poultry Meat with Near-Infrared Hyperspectral Imaging.

A novel semisupervised hyperspectral imaging technique was developed to detect foreign materials (FMs) on raw poultry meat. Combining hyperspectral imaging and deep learning has shown promise in identifying food safety and quality attributes. However, the challenge lies in acquiring a large amount of accurately annotated/labeled data for model training. This paper proposes a novel semisupervised hyperspectral deep learning model based on a generative adversarial network, utilizing an improved 1D U-Net as its discriminator, to detect FMs on raw chicken breast fillets. The model was trained by using approximately 879,000 spectral responses from hyperspectral images of clean chicken breast fillets in the near-infrared wavelength range of 1000-1700 nm. Testing involved 30 different types of FMs commonly found in processing plants, prepared in two nominal sizes: 2 × 2 mm2 and 5 × 5 mm2. The FM-detection technique achieved impressive results at both the spectral pixel level and the foreign material object level. At the spectral pixel level, the model achieved a precision of 100%, a recall of over 93%, an F1 score of 96.8%, and a balanced accuracy of 96.9%. When combining the rich 1D spectral data with 2D spatial information, the FM-detection accuracy at the object level reached 96.5%. In summary, the impressive results obtained through this study demonstrate its effectiveness at accurately identifying and localizing FMs. Furthermore, the technique's potential for generalization and application to other agriculture and food-related domains highlights its broader significance.

Detection of Aflatoxin B1 in Single Peanut Kernels by Combining Hyperspectral and Microscopic Imaging Technologies.

To study the dynamic changes of nutrient consumption and aflatoxin B1 (AFB1) accumulation in peanut kernels with fungal colonization, macro hyperspectral imaging technology combined with microscopic imaging was investigated. First, regression models to predict AFB1 contents from hyperspectral data ranging from 1000 to 2500 nm were developed and the results were compared before and after data normalization with Box-Cox transformation. The results indicated that the second-order derivative with a support vector regression (SVR) model using competitive adaptive reweighted sampling (CARS) achieved the best performance, with RC2 = 0.95 and RV2 = 0.93. Second, time-lapse microscopic images and spectroscopic data were captured and analyzed with scanning electron microscopy (SEM), transmission electron microscopy (TEM), and synchrotron radiation-Fourier transform infrared (SR-FTIR) microspectroscopy. The time-lapse data revealed the temporal patterns of nutrient loss and aflatoxin accumulation in peanut kernels. The combination of macro and micro imaging technologies proved to be an effective way to detect the interaction mechanism of toxigenic fungus infecting peanuts and to predict the accumulation of AFB1 quantitatively.

Spatio-temporal patterns of Aspergillus flavus infection and aflatoxin B1 biosynthesis on maize kernels probed by SWIR hyperspectral imaging and synchrotron FTIR microspectroscopy.

The dynamics mechanisms regulating the growth and AFB1 production of Aspergillus flavus during its interactions with maize kernels remain unclear. In this study, shortwave infrared hyperspectral imaging (SWIR-HSI) and synchrotron radiation Fourier transform infrared (SR-FTIR) microspectroscopy were combined to investigate chemical and spatial-temporal changes in incremental damaged maize kernels induced by A. flavus infection at macroscopic and microscopic levels. SWIR-HSI was employed to extract spectral information of A. flavus growth and quantitatively detect AFB1 levels. Satisfactory full-spectrum models and simplified multispectral models were obtained respectively by partial least squares regression (PLSR) for three types of samples. Furthermore, SR-FTIR microspectroscopy coupled with two-dimensional correlation spectroscopy (2DCOS) was utilized to reveal the possible sequence of dynamic changes of nutrient loss and trace AFB1 in maize kernels. It exhibited new insights on how to quantify the spatio-temporal patterns of fungal infection and AFB1 accumulation on maize and provided theoretical basis for online sorting.

Development of Imaging System for Online Detection of Chicken Meat with Wooden Breast Condition.

In recent years, the wooden breast condition has emerged as a major meat quality defect in the poultry industry worldwide. Broiler pectoralis major muscle with the wooden breast condition is characterized by hardness upon human palpation, which can lead to decrease in meat value or even reduced consumer acceptance. The current method of wooden breast detection involves a visual and/or tactile evaluation. In this paper, we present a sideview imaging system for online detection of chicken breast fillets affected by the wooden breast condition. The system can measure a physical deformation (bending) of an individual chicken-breast fillet through high-speed imaging at about 200 frames per second and custom image processing techniques. The developed image processing algorithm shows the over 95% classification performance in detecting wooden breast fillets.

Integration of spectral and textural features of visible and near-infrared hyperspectral imaging for differentiating between normal and white striping broiler breast meat.

White striping (WS), an emerging muscle myopathy in poultry industry, is gaining increasing attention globally. In this study, visible and near-infrared hyperspectral imaging (HSI, 400-1000 nm) was investigated for developing an optical sensing technique to differentiate WS broiler breast fillets (pectoralis major) from normal fillets. The minimum noise fraction (MNF), followed by an inverse MNF (IMNF), was conducted to improve the signal-to-noise ratio of hyperspectral images during the pre-processing process. Three regions of interest (ROIs) were selected at cranial, middle and caudal locations within each fillet image. Spectral principal component analysis (PCA) revealed that PC2 and PC3 were effective for the differentiation and key wavelengths (450, 492, 541, 581, 629, 869 and 980 nm) were selected from the corresponding PC loadings. Spatial texture features on corresponding score images were obtained using gray level co-occurrence matrix (GLCM) and grayscale histogram statistics (GHS), respectively. Partial least squares discriminant analysis (PLS-DA) models were evaluated with various inputs including spectral (full and key wavelengths), textural and fused features. GLCM features improved performance of multispectral imaging with the highest correct classification rate (CCR) of 91.7%, AUC value (0.917), and Kappa coefficient (0.833) for prediction set. Considering the complexity and heterogeneity of meat samples at different locations, the optimal sampling location was also analyzed and results provided the evidence that the cranial location worked most effectively for the differentiation between normal and WS samples. Overall, results confirmed the great potential of HSI in range of 400-1000 nm in differentiation between normal and WS chicken breast meat.

Tenderness classification of fresh broiler breast fillets using visible and near-infrared hyperspectral imaging.

The aim of this study was to classify and visualize tenderness of intact fresh broiler breast fillets using hyperspectral imaging (HSI) technique. A total of 75 chicken fillets were scanned by HSI system of 400-1000nm in reflectance mode. Warner-Bratzler shear force (WBSF) value was used as reference tenderness indicator and fillets were grouped into least, moderately and very tender categories accordingly. To extract additional image textural features, principal component analysis (PCA) transform of images were conducted and gray level co-occurrence matrix (GLCM) analysis was implemented in region of interests (ROIs) on first three PC score images. Partial least square discriminant analysis (PLS-DA) or radial basis function-support vector machine (RBF-SVM) was developed for predicting tenderness based on full wavelengths (CCR=0.92), selected wavelengths (CCR=0.84), textural or combined data (CCR=0.88). Classification maps were created by pixels prediction in images and breast fillet tenderness was readily discernible. Overall, HSI technique is a promising methodology for predicting tenderness of intact fresh broiler breast meat.

Rapid classification of intact chicken breast fillets by predicting principal component score of quality traits with visible/near-Infrared spectroscopy.

In this study visible/near-infrared spectroscopy (Vis/NIRS) was evaluated to rapidly classify intact chicken breast fillets. Five principal components (PC) were extracted from reference quality traits (L∗, pH, drip loss, expressible fluid, and salt-induced water gain). A quality grades classification method by PC1 score was proposed. With this method, 150 chicken fillets were properly classified into three quality grades, i.e., DFD (dark, firm and dry), normal, and PSE (pale, soft and exudative). Furthermore, PC1 score could be predicted using partial least squares regression (PLSR) model based on Vis/NIRS (R2p = 0.78, RPD = 1.9), without the measurement of any quality traits. Thresholds of PC1 classification method were applied to classify the predicted PC1 score values of each fillet into three quality grades. The classification accuracy of calibration and prediction set were 85% and 80%, respectively. Results revealed that PC1 score classification method is feasible, and with Vis/NIRS, this method could be rapidly implemented.

Evaluation of factors in development of Vis/NIR spectroscopy models for discriminating PSE, DFD and normal broiler breast meat.

1. To evaluate the performance of visible and near-infrared (Vis/NIR) spectroscopic models for discriminating true pale, soft and exudative (PSE), normal and dark, firm and dry (DFD) broiler breast meat in different conditions of preprocessing methods, spectral ranges, characteristic wavelength selection and water-holding capacity (WHC) indexes were assessed. 2. Quality attributes of 214 intact chicken fillets (pectoralis major), such as lightness (L*), pH and WHC indicators including drip loss (DL), water gain and expressible fluid were measured. Fillets were grouped into PSE, normal and DFD categories based on combination of L*, pH and WHC threshold criteria. Classification models were developed using support vector machine based methods on characteristic wavelengths selected from the unprocessed or 2nd-derivative spectra, respectively, in three spectral subsets of 400-2500, 400-1100 and 1100-2500 nm. 3. Better classification of three meat groups was obtained based on unprocessed spectra (72-94%) than 2nd-derivative spectra (55-72%). The classification based on 400-2500 nm (91% average) and 400-1100 nm (89% average) performed better than that on 1100-2500 nm (78% average). In terms of the three different WHC indicators, the combination of L*, pH and DL produced better results than the other two groups, with recognition accuracy of 94.4% using 400-2500-nm range. 4. These analytical results suggest that for a better classification of true PSE, normal and DFD broiler breast meat with Vis/NIR spectra, unprocessed spectra wavelengths should be used, ranges of 400-1000 nm should be included in the data collection, and DL as an indicator of WHC might provide a better prediction model.

Effect of Sample Preparation on the Discrimination of Bacterial Isolates Cultured in Liquid Nutrient Media Using Laser-Induced Breakdown Spectroscopy (LIBS).

Laser-induced breakdown spectroscopy (LIBS) is used as the basis for discrimination between two genera of gram-negative bacteria and two genera of gram-positive bacteria representing pathogenic threats commonly found in poultry processing rinse waters. Because LIBS-based discrimination relies primarily upon the relative proportions of inorganic cell components including Na, K, Mg, and Ca, this study aims to determine the effects of trace mineral content and pH found in the water source used to isolate the bacteria upon the reliability of the resulting discriminant analysis. All four genera were cultured using tryptic soy agar (TSA) as the nutrient medium, and were grown under identical environmental conditions. The only variable introduced is the source water used to isolate the cultured bacteria. Cultures of each bacterium were produced using deionized (DI) water under two atmosphere conditions, reverse osmosis (RO) water, tap water, phosphate buffered saline (PBS) water, and TRIS buffered water. After 3 days of culture growth, the bacteria were centrifuged and washed three times in the same water source. Bacteria were then freeze dried, mixed with microcrystalline cellulose, and a pellet was made for LIBS analysis. Principal component analysis (PCA) was used to extract related variations in LIBS spectral data among the four bacteria genera and six water types used to isolate the bacteria, and Mahalanobis discriminant analysis (MDA) was used for classification. Results indicate not only that the four genera can be discriminated from each other in each water type, but that each genus can be discriminated by water type used for isolation. It is concluded that in order for LIBS to be a reliable and repeatable method for discrimination of bacteria grown in liquid nutrient media, care must be taken to insure that the water source used in purification of the culture be precisely controlled regarding pH, ionic strength, and proportionate amounts of mineral cations present.

Detection by hyperspectral imaging of shiga toxin-producing Escherichia coli serogroups O26, O45, O103, O111, O121, and O145 on rainbow agar.

The U.S. Department of Agriculture, Food Safety Inspection Service has determined that six non-O157 Shiga toxin-producing Escherichia coli (STEC) serogroups (O26, O45, O103, O111, O121, and O145) are adulterants in raw beef. Isolate and phenotypic discrimination of non-O157 STEC is problematic due to the lack of suitable agar media. The lack of distinct phenotypic color variation among non-O157serogroups cultured on chromogenic agar poses a challenge in selecting colonies for confirmation. In this study, visible and near-infrared hyperspectral imaging and chemometrics were used to detect and classify non-O157 STEC serogroups grown on Rainbow agar O157. The method was first developed by building spectral libraries for each serogroup obtained from ground-truth regions of interest representing the true identity of each pixel and thus each pure culture colony in the hyperspectral agar-plate image. The spectral library for the pure-culture non-O157 STEC consisted of 2,171 colonies, with spectra derived from 124,347 of pixels. The classification models for each serogroup were developed with a k nearest-neighbor classifier. The overall classification training accuracy at the colony level was 99%. The classifier was validated with ground beef enrichments artificially inoculated with 10, 50, and 100 CFU/ml STEC. The validation ground-truth regions of interest of the STEC target colonies consisted of 606 colonies, with 3,030 pixels of spectra. The overall classification accuracy was 98%. The average specificity of the method was 98% due to the low false-positive rate of 1.2%. The sensitivity ranged from 78 to 100% due to the false-negative rates of 22, 7, and 8% for O145, O45, and O26, respectively. This study showed the potential of visible and near-infrared hyperspectral imaging for detecting and classifying colonies of the six non-O157 STEC serogroups. The technique needs to be validated with bacterial cultures directly extracted from meat products and positive identification of colonies by using confirmatory tests such as latex agglutination tests or PCR.

Classification and structural analysis of live and dead Salmonella cells using Fourier transform infrared spectroscopy and principal component analysis.

Fourier transform infrared spectroscopy (FT-IR) was used to detect Salmonella Typhimurium and Salmonella Enteritidis food-borne bacteria and to distinguish between live and dead cells of both serotypes. Bacteria cells were prepared in 10(8) cfu/mL concentration, and 1 mL of each bacterium was loaded individually on the ZnSe attenuated total reflection (ATR) crystal surface (45° ZnSe, 10 bounces, and 48 mm × 5 mm effective area of analysis on the crystal) and scanned for spectral data collection from 4000 to 650 cm(-1) wavenumber. Analysis of spectral signatures of Salmonella isolates was conducted using principal component analysis (PCA). Spectral data were divided into three regions such as 900-1300, 1300-1800, and 3000-2200 cm(-1) based on their spectral signatures. PCA models were developed to differentiate the serotypes and live and dead cells of each serotype. Maximum classification accuracy of 100% was obtained for serotype differentiation as well as for live and dead cells differentiation. Soft independent modeling of class analogy (SIMCA) analysis was carried out on the PCA model and applied to validation sample sets. It gave a predicted classification accuracy of 100% for both the serotypes and its live and dead cells differentiation. The Mahalanobis distance calculated in three different spectral regions showed maximum distance for the 1800-1300 cm(-1) region, followed by the 3000-2200 cm(-1) region, and then by the 1300-900 cm(-1) region. It showed that both of the serotypes have maximum differences in their nucleic acids, DNA/RNA backbone structures, protein, and amide I and amide II bands.