In Defense of Metrics: Metrics Sufficiently Encode Typical Human Preferences Regarding Hydrological Model Performance.

Building accurate rainfall-runoff models is an integral part of hydrological science and practice. The variety of modeling goals and applications have led to a large suite of evaluation metrics for these models. Yet, hydrologists still put considerable trust into visual judgment, although it is unclear whether such judgment agrees or disagrees with existing quantitative metrics. In this study, we tasked 622 experts to compare and judge more than 14,000 pairs of hydrographs from 13 different models. Our results show that expert opinion broadly agrees with quantitative metrics and results in a clear preference for a Machine Learning model over traditional hydrological models. The expert opinions are, however, subject to significant amounts of inconsistency. Nevertheless, where experts agree, we can predict their opinion purely from quantitative metrics, which indicates that the metrics sufficiently encode human preferences in a small set of numbers. While there remains room for improvement of quantitative metrics, we suggest that the hydrologic community should reinforce their benchmarking efforts and put more trust in these metrics.

The sensitivity of simulated streamflow to individual hydrologic processes across North America.

Streamflow sensitivity to different hydrologic processes varies in both space and time. This sensitivity is traditionally evaluated for the parameters specific to a given hydrologic model simulating streamflow. In this study, we apply a novel analysis over more than 3000 basins across North America considering a blended hydrologic model structure, which includes not only parametric, but also structural uncertainties. This enables seamless quantification of model process sensitivities and parameter sensitivities across a continuous set of models. It also leads to high-level conclusions about the importance of water cycle components on streamflow predictions, such as quickflow being the most sensitive process for streamflow simulations across the North American continent. The results of the 3000 basins are used to derive an approximation of sensitivities based on physiographic and climatologic data without the need to perform expensive sensitivity analyses. Detailed spatio-temporal inputs and results are shared through an interactive website.

Great Lakes Runoff Intercomparison Project Phase 3: Lake Erie (GRIP-E).

Hydrologic model intercomparison studies help to evaluate the agility of models to simulate variables such as streamflow, evaporation, and soil moisture. This study is the third in a sequence of the Great Lakes Runoff Intercomparison Projects. The densely populated Lake Erie watershed studied here is an important international lake that has experienced recent flooding and shoreline erosion alongside excessive nutrient loads that have contributed to lake eutrophication. Understanding the sources and pathways of flows is critical to solve the complex issues facing this watershed. Seventeen hydrologic and land-surface models of different complexity are set up over this domain using the same meteorological forcings, and their simulated streamflows at 46 calibration and seven independent validation stations are compared. Results show that: (1) the good performance of Machine Learning models during calibration decreases significantly in validation due to the limited amount of training data; (2) models calibrated at individual stations perform equally well in validation; and (3) most distributed models calibrated over the entire domain have problems in simulating urban areas but outperform the other models in validation.

Spatial Retrieval of Broadband Dielectric Spectra.

A broadband soil dielectric spectra retrieval approach ( 1 MHz⁻ 2 GHz) has been implemented for a layered half space. The inversion kernel consists of a two-port transmission line forward model in the frequency domain and a constitutive material equation based on a power law soil mixture rule (Complex Refractive Index Model - CRIM). The spatially-distributed retrieval of broadband dielectric spectra was achieved with a global optimization approach based on a Shuffled Complex Evolution (SCE) algorithm using the full set of the scattering parameters. For each layer, the broadband dielectric spectra were retrieved with the corresponding parameters thickness, porosity, water saturation and electrical conductivity of the aqueous pore solution. For the validation of the approach, a coaxial transmission line cell measured with a network analyzer was used. The possibilities and limitations of the inverse parameter estimation were numerically analyzed in four scenarios. Expected and retrieved layer thicknesses, soil properties and broadband dielectric spectra in each scenario were in reasonable agreement. Hence, the model is suitable for an estimation of in-homogeneous material parameter distributions. Moreover, the proposed frequency domain approach allows an automatic adaptation of layer number and thickness or regular grids in time and/or space.

Parameter importance in FRAP acquisition and analysis: a simulation approach.

Fluorescence recovery after photobleaching (FRAP) is a widespread technique used to determine intracellular reaction and diffusion parameters. In recent years, due to technical advances and an increasing number of mathematical models for analysis, there was a resurging interest in FRAP applications. However, care has to be taken when inverting parameters from such data. We study potential influences on FRAP acquisition and analysis like initial fluorescence distribution, membrane passage, and geometrical aspects. Monte Carlo simulations are employed for the investigation of reaction-diffusion processes to additionally include cases in which no analytical description is available. To assess the importance of influencing factors we apply a sensitivity method based on elementary effects providing an estimate for the global parameter space. The combination of simulations and sensitivity measure helps us to predict ranges of parameters used in acquisition and analysis for which a reliably inversion of reaction-diffusion parameters is possible. Using this approach, we show that FRAP data are highly susceptible to misinterpretation. However, by identifying the parameters of susceptibility, our analysis provides the means for taking measures to significantly improve FRAP data interpretation and analysis.

Kinetic control of contaminant release from NAPLs--information potential of concentration time profiles.

Release of contaminants from non-aqueous phase liquids (NAPLs) is often limited by the dynamic exchange with aqueous solutions governed by a priori unknown kinetic laws. Release experiments require a thorough evaluation of the potential and limitations of kinetic models to reveal release processes. In this study, we investigated the characteristic concentration-time profiles of various models for the release of contaminants from an organic phase into an aqueous solution under no flow conditions. Criteria have been tested that allow for distinction of a first order one domain, a first order two domain, a spherical diffusion model, a spherical diffusion model with a time variable diffusion coefficient, a model for diffusion in a sphere with organic film, and a model for diffusion in a sphere with an aqueous film. The results can serve to evaluate the processes potentially governing release of organic contaminants from non-aqueous liquid phases.

Transcriptional signatures of regulatory and toxic responses to benzo-[a]-pyrene exposure.

BACKGROUND: Small molecule ligands often have multiple effects on the transcriptional program of a cell: they trigger a receptor specific response and additional, indirect responses ("side effects"). Distinguishing those responses is important for understanding side effects of drugs and for elucidating molecular mechanisms of toxic chemicals. RESULTS: We explored this problem by exposing cells to the environmental contaminant benzo-[a]-pyrene (B[a]P). B[a]P exposure activates the aryl hydrocarbon receptor (Ahr) and causes toxic stress resulting in transcriptional changes that are not regulated through Ahr. We sought to distinguish these two types of responses based on a time course of expression changes measured after B[a]P exposure. Using Random Forest machine learning we classified 81 primary Ahr responders and 1,308 genes regulated as side effects. Subsequent weighted clustering gave further insight into the connection between expression pattern, mode of regulation, and biological function. Finally, the accuracy of the predictions was supported through extensive experimental validation. CONCLUSION: Using a combination of machine learning followed by extensive experimental validation, we have further expanded the known catalog of genes regulated by the environmentally sensitive transcription factor Ahr. More broadly, this study presents a strategy for distinguishing receptor-dependent responses and side effects based on expression time courses.

Are assumptions about the model type necessary in reaction-diffusion modeling? A FRAP application.

At present, fluorescence recovery after photobleaching (FRAP) data are interpreted using various types of reaction-diffusion (RD) models: the model type is usually fixed first, and corresponding model parameters are inferred subsequently. In this article, we describe what we believe to be a novel approach for RD modeling without using any assumptions of model type or parameters. To the best of our knowledge, this is the first attempt to address both model-type and parameter uncertainties in inverting FRAP data. We start from the most general RD model, which accounts for a flexible number of molecular fractions, all mobile, with different diffusion coefficients. The maximal number of possible binding partners is identified and optimal parameter sets for these models are determined in a global search of the parameter-space using the Simulated Annealing strategy. The numerical performance of the described techniques was assessed using artificial and experimental FRAP data. Our general RD model outperformed the standard RD models used previously in modeling FRAP measurements and showed that intracellular molecular mobility can only be described adequately by allowing for multiple RD processes. Therefore, it is important to search not only for the optimal parameter set but also for the optimal model type.