Deep reinforcement learning for the direct optimization of gradient separations in liquid chromatography.

While Reinforcement Learning (RL) has already proven successful in performing complex tasks, such as controlling large-scale epidemics, mitigating influenza and playing computer games beyond expert level, it is currently largely unexplored in the field of separation sciences. This paper therefore aims to introduce RL, specifically proximal policy optimization (PPO), in liquid chromatography, and evaluate whether it can be trained to optimize separations directly, based solely on the outcome of a single generic separation as input, and a reward signal based on the resolution between peak pairs (taking a value between [-1,1]). More specifically, PPO algorithms or agents were trained to select linear (1-segment) or multi-segment (2-, 3-, or 16-segment) gradients in 1 experiment, based on the outcome of an initial, generic linear gradient (ϕstart=0.3, ϕend=1.0, and tg=20min), to improve separations. The size of the mixtures to be separated varied between 10 and 20 components. Furthermore, two agents, selecting 16-segment gradients, were trained to perform this optimization using either 2 or 3 experiments, in sequence, to investigate whether the agents could improve separations further, based on previous outcomes. Results showed that the PPO agent can improve separations given the outcome of one generic scouting run as input, by selecting ϕ-programs tailored to the mixture under consideration. Allowing agents more freedom in selecting multi-segment gradients increased the reward from 0.891 to 0.908 on average; and allowing the agents to perform an additional experiment increased the reward from 0.908 to 0.918 on average. Finally, the agent outperformed random experiments as well as standard experiments (ϕstart=0.0, ϕend=1.0, and tg=20min) significantly; as random experiments resulted in average rewards between 0.220 and 0.283, and standard experiments resulted in average rewards of 0.840. In conclusion, while there is room for improvement, the results demonstrate the potential of RL in chromatography and present an interesting future direction for the automated optimization of separations.

Automated method development in high-pressure liquid chromatography.

Method development in liquid chromatography is a crucial step in the optimization of analytical separations for various applications. However, it is often a challenging endeavour due to its time-consuming, resource intensive and costly nature, which is further hampered by its complexity requiring highly skilled and experienced scientists. This review presents an examination of the methods that are required for a completely automated method development procedure in liquid chromatography, aimed at taking the human out of the decision loop. Some of the presented approaches have recently witnessed an important increase in interest as they offer the promise to facilitate, streamline and speed up the method development process. The review first discusses the mathematical description of the separation problem by means of multi-criteria optimization functions. Two different strategies to resolve this optimization are then presented; an experimental and a model-based approach. Additionally, methods for automated peak detection and peak tracking are reviewed, which, upon integration in an instrument, allow for a completely closed-loop method development process. For each of these approaches, various currently applied methods are presented, recent trends and approaches discussed, short-comings pointed out, and future prospects highlighted.

A perspective on the use of deep deterministic policy gradient reinforcement learning for retention time modeling in reversed-phase liquid chromatography.

Artificial intelligence and machine learning techniques are increasingly used for different tasks related to method development in liquid chromatography. In this study, the possibilities of a reinforcement learning algorithm, more specifically a deep deterministic policy gradient algorithm, are evaluated for the selection of scouting runs for retention time modeling. As a theoretical exercise, it is investigated whether such an algorithm can be trained to select scouting runs for any compound of interest allowing to retrieve its correct retention parameters for the three-parameter Neue-Kuss retention model. It is observed that three scouting runs are generally sufficient to retrieve the retention parameters with an accuracy (mean relative percentage error MRPE) of 1 % or less. When given the opportunity to select additional scouting runs, this does not lead to a significantly improved accuracy. It is also observed that the agent tends to give preference to isocratic scouting runs for retention time modeling, and is only motivated towards selecting gradient scouting runs when penalized (strongly) for large analysis/gradient times. This seems to reinforce the general power and usefulness of isocratic scouting runs for retention time modeling. Finally, the best results (lowest MRPE) are obtained when the agent manages to retrieve retention time data for % ACN at elution of the compound under consideration that spread the entire relevant range of ACN (5 % ACN to 95 % ACN) as well as possible, i.e., resulting in retention data at a low, intermediate and high % ACN. Based on the obtained results, we believe reinforcement learning holds great potential to automate and rationalize method development in liquid chromatography in the future.

Multi-model averaging improves the performance of model-guided infliximab dosing in patients with inflammatory bowel diseases.

Infliximab dosage de-escalation without prior knowledge of drug concentrations may put patients at risk for underexposure and trigger the loss of response. A single-model approach for model-informed precision dosing during infliximab maintenance therapy has proven its clinical benefit in patients with inflammatory bowel diseases. We evaluated the predictive performances of two multi-model approaches, a model selection algorithm and a model averaging algorithm, using 18 published population pharmacokinetic models of infliximab for guiding dosage de-escalation. Data of 54 patients with Crohn's disease and ulcerative colitis who underwent infliximab dosage de-escalation after an earlier escalation were used. A priori prediction (based solely on covariate data) and maximum a posteriori prediction (based on covariate data and trough concentrations) were compared using accuracy and precision metrics and the classification accuracy at the trough concentration target of 5.0 mg/L. A priori prediction was inaccurate and imprecise, with the lowest classification accuracies irrespective of the approach (median 59%, interquartile range 59%-63%). Using the maximum a posteriori prediction, the model averaging algorithm had systematically better predictive performance than the model selection algorithm or the single-model approach with any model, regardless of the number of concentration data. Only a single trough concentration (preferably at the point of care) sufficed for accurate and precise prediction. Predictive performance of both single- and multi-model approaches was robust to the lack of covariate data. Model averaging using four models demonstrated similar predictive performance with a five-fold shorter computation time. This model averaging algorithm was implemented in the TDMx software tool to guide infliximab dosage de-escalation in the forthcoming prospective MODIFI study (NCT04982172).

Convolutional neural network for automated peak detection in reversed-phase liquid chromatography.

Although commercially available software provides options for automatic peak detection, visual inspection and manual corrections are often needed. Peak detection algorithms commonly employed require carefully written rules and thresholds to increase true positive rates and decrease false positive rates. In this study, a deep learning model, specifically, a convolutional neural network (CNN), was implemented to perform automatic peak detection in reversed-phase liquid chromatography (RPLC). The model inputs a whole chromatogram and outputs predicted locations, probabilities, and areas of the peaks. The obtained results on a simulated validation set demonstrated that the model performed well (ROC-AUC of 0.996), and comparably or better than a derivative-based approach using the Savitzky-Golay algorithm for detecting peaks on experimental chromatograms (8.6% increase in true positives). In addition, predicted peak probabilities (typically between 0.5 and 1.0 for true positives) gave an indication of how confident the CNN model was in the peaks detected. The CNN model was trained entirely on simulated chromatograms (a training set of 1,000,000 chromatograms), and thus no effort had to be put into collecting and labeling chromatograms. A potential major drawback of this approach, namely training a CNN model on simulated chromatograms, is the risk of not capturing the actual "chromatogram space" well enough that is needed to perform accurate peak detection in real chromatograms.

Graph Convolutional Networks for Improved Prediction and Interpretability of Chromatographic Retention Data.

Machine learning is a popular technique to predict the retention times of molecules based on descriptors. Descriptors and associated labels (e.g., retention times) of a set of molecules can be used to train a machine learning algorithm. However, descriptors are fixed molecular features which are not necessarily optimized for the given machine learning problem (e.g., to predict retention times). Recent advances in molecular machine learning make use of so-called graph convolutional networks (GCNs) to learn molecular representations from atoms and their bonds to adjacent atoms to optimize the molecular representation for the given problem. In this study, two GCNs were implemented to predict the retention times of molecules for three different chromatographic data sets and compared to seven benchmarks (including two state-of-the art machine learning models). Additionally, saliency maps were computed from trained GCNs to better interpret the importance of certain molecular sub-structures in the data sets. Based on the overall observations of this study, the GCNs performed better than all benchmarks, either significantly outperforming them (5-25% lower mean absolute error) or performing similar to them (<5% difference). Saliency maps revealed a significant difference in molecular sub-structures that are important for predictions of different chromatographic data sets (reversed-phase liquid chromatography vs hydrophilic interaction liquid chromatography).

Deep convolutional autoencoder for the simultaneous removal of baseline noise and baseline drift in chromatograms.

Enhancement of chromatograms, such as the reduction of baseline noise and baseline drift, is often essential to accurately detect and quantify analytes in a mixture. Current methods have been well studied and adopted for decades and have assisted researchers in obtaining reliable results. However, these methods rely on relatively simple statistics of the data (chromatograms) which in some cases result in significant information loss and inaccuracies. In this study, a deep one-dimensional convolutional autoencoder was developed that simultaneously removes baseline noise and baseline drift with minimal information loss, for a large number and great variety of chromatograms. To enable the autoencoder to denoise a chromatogram to be almost, or completely, noise-free, it was trained on data obtained from an implemented chromatogram simulator that generated 190.000 representative simulated chromatograms. The trained autoencoder was then tested and compared to some of the most widely used and well-established denoising methods on testing datasets of tens of thousands of simulated chromatograms; and then further tested and verified on real chromatograms. The results show that the developed autoencoder can successfully remove baseline noise and baseline drift simultaneously with minimal information loss; outperforming methods like Savitzky-Golay smoothing, Gaussian smoothing and wavelet smoothing for baseline noise reduction (root mean squared error of 1.094 mAU compared to 2.074 mAU, 2.394 mAU and 2.199 mAU) and Savitkzy-Golay smoothing combined with asymmetric least-squares or polynomial fitting for baseline noise and baseline drift reduction (root mean absolute error of 1.171 mAU compared to 3.397 mAU and 4.923 mAU). Evidence is presented that autoencoders can be utilized to enhance and correct chromatograms and consequently improve and alleviate downstream data analysis, with the drawback of needing a carefully implemented simulator, that generates realistic chromatograms, to train the autoencoder.

Deep Q-learning for the selection of optimal isocratic scouting runs in liquid chromatography.

An important challenge in chromatography is the development of adequate separation methods. Accurate retention models can significantly simplify and expedite the development of adequate separation methods for complex mixtures. The purpose of this study was to introduce reinforcement learning to chromatographic method development, by training a double deep Q-learning algorithm to select optimal isocratic scouting runs to generate accurate retention models. These scouting runs were fit to the Neue-Kuss retention model, which was then used to predict retention factors both under isocratic and gradient conditions. The quality of these predictions was compared to experimental data points, by computing a mean relative percentage error (MRPE) between the predicted and actual retention factors. By providing the reinforcement learning algorithm with a reward whenever the scouting runs led to accurate retention models and a penalty when the analysis time of a selected scouting run was too high (> 1h); it was hypothesized that the reinforcement learning algorithm should by time learn to select good scouting runs for compounds displaying a variety of characteristics. The reinforcement learning algorithm developed in this work was first trained on simulated data, and then evaluated on experimental data for 57 small molecules - each run at 10 different fractions of organic modifier (0.05 to 0.90) and four different linear gradients. The results showed that the MRPE of these retention models (3.77% for isocratic runs and 1.93% for gradient runs), mostly obtained via 3 isocratic scouting runs for each compound, were comparable in performance to retention models obtained by fitting the Neue-Kuss model to all (10) available isocratic datapoints (3.26% for isocratic runs and 4.97% for gradient runs) and retention models obtained via a "chromatographer's selection" of three scouting runs (3.86% for isocratic runs and 6.66% for gradient runs). It was therefore concluded that the reinforcement learning algorithm learned to select optimal scouting runs for retention modeling, by selecting 3 (out of 10) isocratic scouting runs per compound, that were informative enough to successfully capture the retention behavior of each compound.

Fast liquid chromatography-tandem mass spectrometry methodology for the analysis of alkylphenols and their ethoxylates in wastewater samples from the tank truck cleaning industry.

A fast methodology to quantify 4-tert-octylphenol (4-t-OP) and 4-nonylphenol (4-NP) and their mono- and di-ethoxylates was developed, validated, and applied to real wastewater samples. Dispersive liquid-liquid microextraction was employed as a sample preparation step, leading to a pre-concentration factor of roughly 30. Analysis was carried out by liquid chromatography-tandem mass spectrometry with electrospray ionisation in multiple reaction monitoring mode. Average recoveries were generally between 80 and 120% for both the alkylphenols and their mono- and di-ethoxylates in influent and effluent wastewater. A minimum of 5 concentration levels per compound, ranging between 1 and 500 ng/mL, were prepared to construct calibration curves making use of isotopically labelled internal standards. The method presented good linearity and repeatability over the whole range of concentrations. Taking into account the concentration factor, and the recovery of the compounds, lower limits of quantification obtained in effluent wastewater were 0.04 ng/mL for 4-t-OP and 0.14 ng/mL for 4-NP, complying with European regulations, and between 0.03 ng/mL and 0.39 ng/mL for the ethoxylates. In influent wastewater, these limits were slightly higher. The total run time of 5 min for the alkylphenols and 8 min for the ethoxylates ensured high throughput. The developed method was applied to determine 4-t-OP and 4-NP and their mono- and di-ethoxylates in wastewater from several tank truck cleaning companies, which was subjected to ozonation and/or biological treatment. It was demonstrated that ozonation was best applied after the biological treatment, since in this case, the biological treatment could degrade most of the biodegradable organic matter, after which ozone could react directly with the recalcitrant organic pollutants. In this case, the concentrations of the target compounds in the wastewater of the investigated company decreased below the legally allowed concentration of the European water legislation.

Transfer Learning with Deep Convolutional Neural Networks for Classifying Cellular Morphological Changes.

The quantification and identification of cellular phenotypes from high-content microscopy images has proven to be very useful for understanding biological activity in response to different drug treatments. The traditional approach has been to use classical image analysis to quantify changes in cell morphology, which requires several nontrivial and independent analysis steps. Recently, convolutional neural networks have emerged as a compelling alternative, offering good predictive performance and the possibility to replace traditional workflows with a single network architecture. In this study, we applied the pretrained deep convolutional neural networks ResNet50, InceptionV3, and InceptionResnetV2 to predict cell mechanisms of action in response to chemical perturbations for two cell profiling datasets from the Broad Bioimage Benchmark Collection. These networks were pretrained on ImageNet, enabling much quicker model training. We obtain higher predictive accuracy than previously reported, between 95% and 97%. The ability to quickly and accurately distinguish between different cell morphologies from a scarce amount of labeled data illustrates the combined benefit of transfer learning and deep convolutional neural networks for interrogating cell-based images.

Evaluating parameters for ligand-based modeling with random forest on sparse data sets.

Ligand-based predictive modeling is widely used to generate predictive models aiding decision making in e.g. drug discovery projects. With growing data sets and requirements on low modeling time comes the necessity to analyze data sets efficiently to support rapid and robust modeling. In this study we analyzed four data sets and studied the efficiency of machine learning methods on sparse data structures, utilizing Morgan fingerprints of different radii and hash sizes, and compared with molecular signatures descriptor of different height. We specifically evaluated the effect these parameters had on modeling time, predictive performance, and memory requirements using two implementations of random forest; Scikit-learn as well as FEST. We also compared with a support vector machine implementation. Our results showed that unhashed fingerprints yield significantly better accuracy than hashed fingerprints ([Formula: see text]), with no pronounced deterioration in modeling time and memory usage. Furthermore, the fast execution and low memory usage of the FEST algorithm suggest that it is a good alternative for large, high dimensional sparse data. Both support vector machines and random forest performed equally well but results indicate that the support vector machine was better at using the extra information from larger values of the Morgan fingerprint's radius.