Scaling Up Q-Learning via Exploiting State-Action Equivalence.

Recent success stories in reinforcement learning have demonstrated that leveraging structural properties of the underlying environment is key in devising viable methods capable of solving complex tasks. We study off-policy learning in discounted reinforcement learning, where some equivalence relation in the environment exists. We introduce a new model-free algorithm, called QL-ES (Q-learning with equivalence structure), which is a variant of (asynchronous) Q-learning tailored to exploit the equivalence structure in the MDP. We report a non-asymptotic PAC-type sample complexity bound for QL-ES, thereby establishing its sample efficiency. This bound also allows us to quantify the superiority of QL-ES over Q-learning analytically, which shows that the theoretical gain in some domains can be massive. We report extensive numerical experiments demonstrating that QL-ES converges significantly faster than (structure-oblivious) Q-learning empirically. They imply that the empirical performance gain obtained by exploiting the equivalence structure could be massive, even in simple domains. To the best of our knowledge, QL-ES is the first provably efficient model-free algorithm to exploit the equivalence structure in finite MDPs.

Retention prediction using quantitative structure-retention relationships combined with the hydrophobic subtraction model in reversed-phase liquid chromatography.

The hydrophobic subtraction model (HSM) combined with quantitative structure-retention relationships (QSRR) methodology was utilized to predict retention times in reversed-phase liquid chromatography (RPLC). A selection of new analytes and new RPLC columns that had never been used in the QSRR modeling process were used to verify the proposed approach. This work is designed to facilitate early prediction of co-elution of analytes in pharmaceutical drug discovery applications where it is advantageous to predict whether impurities might be co-eluted with the active drug component. The QSRR models were constructed through partial least squares regression combined with a genetic algorithm (GA-PLS) which was employed as a feature selection method to choose the most informative molecular descriptors calculated using VolSurf+ software. The analyte hydrophobicity coefficient of the HSM was predicted for subsequent calculation of retention. Clustering approaches based on the local compound type and the local second dominant interaction were investigated to select the most appropriate training set of analytes from a larger database. Predicted retention times of five new compounds on five new RPLC C18 columns were compared with their measured retention times with percentage root-mean-square errors of 15.4 and 24.7 for the local compound type and local second dominant interaction clustering methods, respectively.

Retention Index Prediction Using Quantitative Structure-Retention Relationships for Improving Structure Identification in Nontargeted Metabolomics.

Structure identification in nontargeted metabolomics based on liquid-chromatography coupled to mass spectrometry (LC-MS) remains a significant challenge. Quantitative structure-retention relationship (QSRR) modeling is a technique capable of accelerating the structure identification of metabolites by predicting their retention, allowing false positives to be eliminated during the interpretation of metabolomics data. In this work, 191 compounds were grouped according to molecular weight and a QSRR study was carried out on the 34 resulting groups to eliminate false positives. Partial least squares (PLS) regression combined with a Genetic algorithm (GA) was applied to construct the linear QSRR models based on a variety of VolSurf+ molecular descriptors. A novel dual-filtering approach, which combines Tanimoto similarity (TS) searching as the primary filter and retention index (RI) similarity clustering as the secondary filter, was utilized to select compounds in training sets to derive the QSRR models yielding R2 of 0.8512 and an average root mean square error in prediction (RMSEP) of 8.45%. With a retention index filter expressed as ±2 standard deviations (SD) of the error, representative compounds were predicted with >91% accuracy, and for 53% of the groups (18/34), at least one false positive compound could be eliminated. The proposed strategy can thus narrow down the number of false positives to be assessed in nontargeted metabolomics.

Rapid Method Development in Hydrophilic Interaction Liquid Chromatography for Pharmaceutical Analysis Using a Combination of Quantitative Structure-Retention Relationships and Design of Experiments.

A design-of-experiment (DoE) model was developed, able to describe the retention times of a mixture of pharmaceutical compounds in hydrophilic interaction liquid chromatography (HILIC) under all possible combinations of acetonitrile content, salt concentration, and mobile-phase pH with R2 > 0.95. Further, a quantitative structure-retention relationship (QSRR) model was developed to predict retention times for new analytes, based only on their chemical structures, with a root-mean-square error of prediction (RMSEP) as low as 0.81%. A compound classification based on the concept of similarity was applied prior to QSRR modeling. Finally, we utilized a combined QSRR-DoE approach to propose an optimal design space in a quality-by-design (QbD) workflow to facilitate the HILIC method development. The mathematical QSRR-DoE model was shown to be highly predictive when applied to an independent test set of unseen compounds in unseen conditions with a RMSEP value of 5.83%. The QSRR-DoE computed retention time of pharmaceutical test analytes and subsequently calculated separation selectivity was used to optimize the chromatographic conditions for efficient separation of targets. A Monte Carlo simulation was performed to evaluate the risk of uncertainty in the model's prediction, and to define the design space where the desired quality criterion was met. Experimental realization of peak selectivity between targets under the selected optimal working conditions confirmed the theoretical predictions. These results demonstrate how discovery of optimal conditions for the separation of new analytes can be accelerated by the use of appropriate theoretical tools.

In Silico Screening of Two-Dimensional Separation Selectivity for Ion Chromatography × Capillary Electrophoresis Separation of Low-Molecular-Mass Organic Acids.

A prerequisite for ordered two-dimensional (2D) separations and full utilization of the enhanced 2D peak capacity is selective exploitation of the sample attributes, described as sample dimensionality. In order to take sample dimensionality into account prior to optimization of a 2D separation, a new concept based on construction of 2D separation selectivity maps is proposed and demonstrated for ion chromatography × capillary electrophoresis (IC×CE) separation of low-molecular-mass organic acids as test analytes. For this purpose, 1D separation selectivity maps were constructed based on calculation of pairwise separation factors and identification of critical pairs for four IC stationary phases and 28 levels of background electrolyte pH in CE. The derived IC and CE maps were then superimposed and the effectiveness of the respective 2D separations assessed using an in silico approach, followed by testing examples of one successful and one unsuccessful 2D combination experimentally. The results confirmed the efficacy of the predictions, which require a minimal number of experiments compared to the traditional one-at-a-time approach. Following the same principles, the proposed framework can also be adapted for optimization of separation selectivity in various 2D combinations and for other applications.

Benchmarking of Computational Methods for Creation of Retention Models in Quantitative Structure-Retention Relationships Studies.

Quantitative structure-retention relationship (QSRR) models are powerful techniques for the prediction of retention times of analytes, where chromatographic retention parameters are correlated with molecular descriptors encoding chemical structures of analytes. Many QSRR models contain geometrical descriptors derived from the three-dimensional (3D) spatial coordinates of computationally predicted structures for the analytes. Therefore, it is sensible to calculate these structures correctly, as any error is likely to carry over to the resulting QSRR models. This study compares molecular modeling, semiempirical, and density functional methods (both B3LYP and M06) for structure optimization. Each of the calculations was performed in a vacuum, then repeated with solvent corrections for both acetonitrile and water. We also compared Natural Bond Orbital analysis with the Mulliken charge calculation method. The comparison of the examined computational methods for structure calculation shows that, possibly due to the error inherent in descriptor creation methods, a quick and inexpensive molecular modeling method of structure determination gives similar results to experiments where structures are optimized using an expensive and time-consuming level of computational theory. Also, for structures with low flexibility, vacuum or gas phase calculations are found to be as effective as those calculations with solvent corrections added.

Semiautomated pH gradient ion-exchange chromatography of monoclonal antibody charge variants.

A new approach using a chromatography system equipped with isocratic pumps and an electrolytic eluent generator (EG) is introduced, replacing external pH gradient delivery using conventional gradient systems, in which bottled buffers with preadjusted pH are mixed using a gradient pump. The EG is capable of generating high purity base or acid required for online preparation of the buffer at the point of use, utilizing deionized water as the only carrier stream. Typically, the buffer was generated from online titration of a reagent composed of low molecular weight amines. The reagent was delivered isocratically into a static mixing tee, where it was titrated to the required pH with electrolytically generated base or acid. The required pH gradient was thus conveniently generated by electrically controlling the concentration of titrant. Also, since the pH was adjusted at the point of use, this approach offered enhanced throughput in terms of eluent preparation time and labor, and with a more reproducible pH profile. The performance of the system was demonstrated by running pH gradients ranging from pH 8.2 to 10.9 on a polymer monolith cation-exchange column for high throughput profiling of charge heterogeneity of intact, basic therapeutic monoclonal antibodies. A high degree of flexibility in modulating the key parameters of the pH gradient, including the buffer concentration, the pH gradient slope and the operating pH range was demonstrated. This enabled fine-tuning of the separation conditions for each individual antibody in order to enhance the chromatographic resolution.

Epoxy-based monoliths for capillary liquid chromatography of small and large molecules.

A versatile epoxy-based monolith was synthesised by polycondensation polymerisation of glycidyl ether 100 with ethylenediamine using a porogenic system consisting of polyethylene glycol, M(w) = 1000, and 1-decanol. Polymerisation was performed at 80 °C for 22 h. A simple acid hydrolysis of residual epoxides resulted in a mixed diol-amino chemistry. The modified column was used successfully for hydrophilic interaction liquid chromatography (HILIC) of small molecule probes such as nucleic acid bases and nucleosides, benzoic acid derivatives, as well as for peptides released from a tryptic digest of cytochrome c. The mixed-mode chemistry allowed both hydrophilic partitioning and ion-exchange (IEX) interactions to contribute to the separation, providing flexibility in selectivity control. Residual epoxide groups were also exploited for incorporating a mixed IEX chemistry. Alternatively, the surface chemistry of the monolith pore surface rendered hydrophobic via grafting of a co-polymerised hydrophobic hydrogel. The inherent hydrophilicity of the monolith scaffold also enabled high performance separation of proteins under IEX and hydrophobic interaction modes and in the absence of non-specific interactions.

Poly(ethylene glycol)-based monolithic capillary columns for hydrophobic interaction chromatography of immunoglobulin G subclasses and variants.

Polymer monoliths were prepared in 150 µm id capillaries by thermally initiated polymerization of PEG diacrylate for rapid hydrophobic interaction chromatography of immunoglobulin G (IgG) subclasses and related variants. Using only one monomer in the polymerization mixture allowed ease of optimization and synthesis of the monolith. The performance of the monolith was demonstrated by baseline resolution of IgG subclasses and variants, including mixtures of the κ variants of IgG1, IgG2, and IgG3 as well as the κ and λ variants associated with IgG1 and IgG2. The effect of eluent concentration and pH on the separation efficiency of studied proteins was also explored, allowing almost baseline resolution to be achieved for mixtures of the κ variants of IgG1, IgG2, IgG3, and IgG4 but also for the κ and λ variants of IgG1 and IgG2. The results showed significant improvement in the separations in terms of the tradeoff between analysis time and resolution, while maintaining a simple methodology, in comparison to previous reports. The synthesized monolith was also used for the separation of isoforms of a therapeutic monoclonal antibody.

Influence of musical context on sensorimotor synchronization in classical ballet solo dance.

Several studies have addressed motor coordination in dance, but few have addressed the influence of musical context on micro-timing during sensorimotor synchronization (SMS) in classical ballet. In this study, we analyze the Promenade in Arabesque of the Odile variations, first as a dance-music fragment non-embedded in a musical context, then as a dance-music fragment embedded in a musical context at two different instances. Given the musical structure of the fragments, there are repeats of patterns between and within the fragments. Four dancers were invited to perform the three fragments in twelve successive performances. The beats of the music were extracted and compared with the timing of the dancers' heel movements, using circular-linear smooth regression modelling, and circular statistics. The results reveal an effect of repeat within fragments, and an effect of musical context between fragments, on micro-timing anticipation in SMS. The methodology offers a framework for future work on dynamical aspects of SMS.

The Effect of Perianal Tramadol Infiltration on Postoperative Pain Following Hemorrhoidectomy.

INTRODUCTION: The present study was attempted to evaluate the effect of perianal infiltration of tramadol on postoperative pain in patients undergoing hemorrhoidectomy. METHOD: This double-blind clinical trial study was carried out on 90 patients with grade 3 and 4 hemorrhoids undergoing hemorrhoidectomy. Patients were randomly assigned into 3 groups of control or bupivacaine or tramadol. Before the surgery, perianal infiltration of .25% bupivacaine or tramadol or normal saline was prescribed to each group, respectively. Data on pain severity (based on the visual analog scale (VAS), the duration of surgery, sedation score, pain at the first defecation, first request time for additional analgesia, nausea and vomiting, and analgesic intakes) were evaluated and analyzed. RESULTS: Duration of surgery was almost similar in all 3 groups (P = .974). The results showed a significant difference in pain score between 3 groups (P ≤.05) at all times after the surgery. In addition, the means of sedation scores (P = .03), pain score at the first defecation (P = .001), the time to first analgesic request (P = .001), and ketorolac administration times (P = .01) were significantly different between 3 groups. Finally, no complication was reported regarding postoperative nausea and vomiting. CONCLUSION: Given the notable efficacy of tramadol in reducing pain after hemorrhoidectomy and its minor side effects, this medication is suggested as an effective topical anesthetic to decrease pain after hemorrhoidectomy.

The effectiveness of caffeinated chewing gum in ameliorating cognitive functions affected by sleep deprivation.

Objectives: This investigation aimed to compare caffeinated gums with two different dosages of caffeine (200mg vs. 300mg) by assessing their effectiveness on the improvement of cognitive functions among Iranian individuals voluntarily suffering from 30 hours of sleep deprivation. Material and Methods: Thirty-four healthy male volunteers with ages from 28 to 35 years old were randomly assigned to either 200 or 300mg caffeine intake. Each participant completed CANTAB subtests to assess their core cognitive functions including MOT, RTI, RVP, and SWM before and after sleep deprivation, as well as after being treated with caffeinated gum. Results: The 300mg caffeine intake group indicated higher levels of enhancement of core cognitive functions compared with those in the 200mg caffeine intake group. Conclusion: This study suggests that the dose of 300mg of caffeine could effectively enhance the cognitive functions of Iranian individuals suffering from sleep deprivation.

The nexus between sleep disturbances and mental health outcomes in military staff: a systematic review.

Objectives: Military personnel are unique occupational groups who happen to frequently experience sleep insuffciencies. Since sleep disorders are known to be linked to many psychiatric symptoms, sleep disturbance is a salient concern among active duty service members and veterans. Existing evidence indicates that although sleep disturbances co-occur with mental illnesses, there is a tendency to particularly label them as consequences of certain mental health issues. Material and Methods: This review focuses on the emerging evidence which identifies sleep disturbances as a precursor for mental illnesses. In this regard, the impact of sleep disturbance on the development of mental health outcomes including post-traumatic stress disorder (PTSD), depression, and anxiety has been thoroughly scrutinized. A systematic search was conducted using PubMed, Scopus, and Web of Science academic databases using appropriate keywords. Results: Reviewed evidence substantiates the predicting role of sleep complaints and disorders to herald PTSD, depression, and anxiety among military staff. Conclusion: Early diagnosis of sleep disturbances and properly addressing them in active-duty service members and veterans should be then sought to prevent the development and progression of consequent mental health- related comorbidities in this study group.

Validated high-throughput HPLC method for the analysis of flavonol aglycones myricetin, quercetin, and kaempferol in Rhus coriaria L. using a monolithic column.

A rapid and simple reversed-phase high-performance liquid chromatographic method using a monolithic column was developed and validated for the separation and quantification of myricetin, quercetin, and kaempferol in Rhus coriaria L. The method employed the isocratic mobile phase acetonitrile-10 mM potassium dihydrogen orthophosphate buffer adjusted to pH 3.0 using orthophosphoric acid (38 + 62, v/v) at a flow rate of 4.0 mL/min; a Chromolith Performance RP-18e (100 x 4.6 mm) monolithic column kept at 40 degrees C; and UV detection at 370 nm. Although the elution order was identical and the selectivity was equivalent, the comparison between monolithic and particulate columns showed that the monolithic column could reduce the separation time to < 1 min without sacrificing column efficiency and selectivity. The method was validated according to International Conference on Harmonization guidelines. The validation characteristics included accuracy, precision, linearity, range, specificity, LOQ, and robustness. The calibration curves were linear (r > 0.999) over the concentration range of 0.88-88.3 micro/mL for myricetin, 0.95-95 microg/mL for quercetin, and 1.43-143.3 microg/mL for kaempferol. The recoveries for all three compounds were above 89%. Myricetin was found to be the major flavonol in the examined plant extracts, followed by minor quantities of quercetin and kaempferol.

Demographic Evaluation of Oro-Dental Self-Injury for Insurance Deception; Evaluation of the Cases Referred to Shiraz Forensic Medicine Center.

STATEMENT OF THE PROBLEM: Nonsuicidal self-inflicted injuries are socially unacceptable and may cause mild to severe damages. PURPOSE: This study aimed to evaluate the demographic features of the subjects with orodental self-injuries referred to a forensic medicine center in Shiraz, Iran. MATERIALS AND METHOD: This cross-sectional study evaluated 51 participants (49 men and 2 women) with orodental injuries referred to forensic medicine administration. Orodental self-injury was detected in the subjects, based on the last forensic criterion of self-injuries, considering their history, clinical examinations, and panoramic radiographs. RESULTS: The findings of this study revealed that dental self-injuries were more prevalent among married men from urban areas with secondary education levels. Most of the cases were due to the monetary compensation received. In the majority of cases, a hard object was used for this self-injury. Moreover, no statistical association was observed between the economic status and orodental self-injury. CONCLUSION: This study concluded that dental self-injury could be regarded as an unplanned incident because no significant correlation was observed between the participants, their economic status, and the type of dental trauma. Furthermore, detailed investigations on the latent variables are required.

Micro solid-phase extraction for the analysis of per- and polyfluoroalkyl substances in environmental waters.

Growing concern over the environmental and health impacts of per- and polyfluoroalkyl substances (PFASs) has led to the development of increasingly stringent regulatory guidelines. To meet these guidelines for the determination of PFASs in surface-water, solid-phase extraction (SPE) is commonly implemented for clean-up and pre-concentration of samples. In this paper a micro-SPE method for the clean-up and pre-concentration of PFASs from surface-water was developed. A micro-SPE packing phase was created to retain 13 long and short chain PFAS after examining combinations of four 3 µm particle size sorbents, with the optimal phase consisting of a 50:50 mixture of C18 and aminopropyl silica. Micro-SPE achieved similar results to conventional SPE methods while reducing sample preparation time to 5 min and using only 2 mL of sample. The method was validated using spiked recoveries (100 ng L-1) from PFAS contaminated surface-water samples with recoveries ranging from 86% to 111% and relative standard deviations below 18%. Concentrations of the PFASs in the samples ranged from below the limit of quantification to 898 ± 15 ng L-1. Automation of sample preparation, including the micro-SPE extraction, was also demonstrated. These results show the potential for automated micro-SPE to replace conventional SPE, with the decreases in sample preparation time, sample and solvent volumes crucial for incorporation into routine analyses in commercial laboratories.

Retention prediction in reversed phase high performance liquid chromatography using quantitative structure-retention relationships applied to the Hydrophobic Subtraction Model.

Quantitative Structure-Retention Relationships (QSRR) methodology combined with the Hydrophobic Subtraction Model (HSM) have been utilized to accurately predict retention times for a selection of analytes on several different reversed phase liquid chromatography (RPLC) columns. This approach is designed to facilitate early prediction of co-elution of analytes, for example in pharmaceutical drug discovery applications where it is advantageous to predict whether impurities might be co-eluted with the active drug component. The QSRR model utilized VolSurf+ descriptors and a Partial Least Squares regression combined with a Genetic Algorithm (GA-PLS) to predict the solute coefficients in the HSM. It was found that only the hydrophobicity (η'H) term in the HSM was required to give the accuracy necessary to predict potential co-elution of analytes. Global QSRR models derived from all 148 compounds in the dataset were compared to QSRR models derived using a range of local modelling techniques based on clustering of compounds in the dataset by the structural similarity of compounds (as represented by the Tanimoto similarity index), physico-chemical similarity of compounds (represented by log D), the neutral, acidic, or basic nature of the compound, and the second dominant interaction between analyte and stationary phase after hydrophobicity. The global model showed reasonable prediction accuracy for retention time with errors of 30 s and less for up to 50% of modeled compounds. The local models for Tanimoto, nature of the compound and second dominant interaction approaches all exhibited prediction errors less than 30 s in retention time for nearly 70% of compounds for which models could be derived. Predicted retention times of five representative compounds on nine reversed-phase columns were compared with known experimental retention data for these columns and this comparison showed that the accuracy of the proposed modelling approach is sufficient to reliably predict the retention times of analytes based only on their chemical structures.

Chemometric-assisted method development in hydrophilic interaction liquid chromatography: A review.

With an enormous growth in the application of hydrophilic interaction liquid chromatography (HILIC), there has also been significant progress in HILIC method development. HILIC is a chromatographic method that utilises hydro-organic mobile phases with a high organic content, and a hydrophilic stationary phase. It has been applied predominantly in the determination of small polar compounds. Theoretical studies in computer-aided modelling tools, most importantly the predictive, quantitative structure retention relationship (QSRR) modelling methods, have attracted the attention of researchers and these approaches greatly assist the method development process. This review focuses on the application of computer-aided modelling tools in understanding the retention mechanism, the classification of HILIC stationary phases, prediction of retention times in HILIC systems, optimisation of chromatographic conditions, and description of the interaction effects of the chromatographic factors in HILIC separations. Additionally, what has been achieved in the potential application of QSRR methodology in combination with experimental design philosophy in the optimisation of chromatographic separation conditions in the HILIC method development process is communicated. Developing robust predictive QSRR models will undoubtedly facilitate more application of this chromatographic mode in a broader variety of research areas, significantly minimising cost and time of the experimental work.

Medication Interaction and Physicians' Compliance Assessment through Medication Reconciliation Forms in a University Affiliated Hospital.

Medication interactions are associated with various unwanted adverse drug reactions. Medication Reconciliation involves a process in which a complete list of patient's previously prescribed medications are recorded and subsequently evaluated within the context of concomitantly prescribed medications and present medical condition during the hospitalization. Medical records of randomly selected 270 patients hospitalized in internal medicine, cardiovascular and infectious diseases wards were evaluated. Drug interactions were checked by LexiComp® database. Each interaction was assigned a risk rating of A, B, C, D, or X. The progression from A to X was based on increased urgency for responding to the data. Completed reconciliation forms were attached to patient charts for evaluation of physicians' compliance. Drug interactions were observed in 65.2% (176/270) of cases. The risk rating of interactions was categorized as C, D and X in 54.2%, 32.4%, and 13.4% of cases, respectively. There was a positive correlation between the number of prescribed medications and the rate of interactions (p-value < 0.001, Kendall's correlation coefficient = 0.487). Moreover, the length of hospitalization and the rate of drug interactions were significantly correlated (p-value < 0.001, Kendall's correlation coefficient = 0.350). Cardiovascular agents constituted the largest proportion of interactions (25%) followed by antibiotics (18%) and immunosuppressive agents (6%). In 59.6% of cases, no corrective action was taken by the physicians. Medication discrepancies occur commonly in hospital settings. Structured medication reconciliation may have a positive impact on prevention of medication errors.

Towards a chromatographic similarity index to establish localised Quantitative Structure-Retention Relationships for retention prediction. III Combination of Tanimoto similarity index, logP, and retention factor ratio to identify optimal analyte training sets for ion chromatography.

Retention prediction for unknown compounds based on Quantitative Structure-Retention Relationships (QSRR) can lead to rapid "scoping" method development in chromatography by simplifying the selection of chromatographic parameters. The use of retention factor ratio (or k-ratio) as a chromatographic similarity index can be a potent method to cluster similar compounds into a training set to generate an accurate predictive QSRR model provided that its limitation - that the method is impractical for retention prediction for unknown compounds - is successfully addressed. In this work, we propose a localised QSRR modelling approach with the aim of compensating the critical limitation in the otherwise successful k-ratio filter-based QSRR modelling. The approach is to combine a k-ratio filter with both Tanimoto similarity (TS) and a ΔlogP index (i.e., logP-Dual filter). QSRR models for two retention parameters (a and b) in the linear solvent strength (LSS) model in ion chromatography (IC), logk=a - blog[eluent], were generated for larger organic cations (molecular mass up to 506) on a Thermo Fisher Scientific CS17 column. The application of the developed logP-Dual filter resulted in the production of successful QSRR models for 50 organic cations out of 87 in the dataset. The predicted a- and b-values of the models were then applied to the LSS model to predict the corresponding retention times. External validation showed that QSRR models for a-, b- and tR- values with excellent accuracy and predictability (Qext(F2)2 of 0.96, 0.95, and 0.96, RMSEP of 0.06, 0.02, and 0.38min) were created successfully, and these models can be employed to speed up the "scoping" phase of method development in IC.