Response to Comment on "Pushing the frontiers of density functionals by solving the fractional electron problem".

Gerasimov et al. claim that the ability of DM21 to respect fractional charge (FC) and fractional spin (FS) conditions outside of the training set has not been demonstrated in our paper. This is based on (i) asserting that the training set has a ~50% overlap with our bond-breaking benchmark (BBB) and (ii) questioning the validity and accuracy of our other generalization examples. We disagree with their analysis and believe that the points raised are either incorrect or not relevant to the main conclusions of the paper and to the assessment of general quality of DM21.

Performance of a one-parameter correlation factor for transcorrelation: Study on a series of second row atomic and molecular systems.

In this work, we investigate the performance of a recently proposed transcorrelated (TC) approach based on a single-parameter correlation factor [E. Giner, J. Chem. Phys. 154, 084119 (2021)] for systems involving more than two electrons. The benefit of such an approach relies on its simplicity as efficient numerical-analytical schemes can be set up to compute the two- and three-body integrals occurring in the effective TC Hamiltonian. To obtain accurate ground state energies within a given basis set, the present TC scheme is coupled to the recently proposed TC-full configuration interaction quantum Monte Carlo method [Cohen et al., J. Chem. Phys. 151, 061101 (2019)]. We report ground state total energies on the Li-Ne series, together with their first cations, computed with increasingly large basis sets and compare to more elaborate correlation factors involving electron-electron-nucleus coordinates. Numerical results on the Li-Ne ionization potentials show that the use of the single-parameter correlation factor brings on average only a slightly lower accuracy (1.2 mH) in a triple-zeta quality basis set with respect to a more sophisticated correlation factor. However, already using a quadruple-zeta quality basis set yields results within chemical accuracy to complete basis set limit results when using this novel single-parameter correlation factor. Calculations on the H2O, CH2, and FH molecules show that a similar precision can be obtained within a triple-zeta quality basis set for the atomization energies of molecular systems.

Pushing the frontiers of density functionals by solving the fractional electron problem.

Density functional theory describes matter at the quantum level, but all popular approximations suffer from systematic errors that arise from the violation of mathematical properties of the exact functional. We overcame this fundamental limitation by training a neural network on molecular data and on fictitious systems with fractional charge and spin. The resulting functional, DM21 (DeepMind 21), correctly describes typical examples of artificial charge delocalization and strong correlation and performs better than traditional functionals on thorough benchmarks for main-group atoms and molecules. DM21 accurately models complex systems such as hydrogen chains, charged DNA base pairs, and diradical transition states. More crucially for the field, because our methodology relies on data and constraints, which are continually improving, it represents a viable pathway toward the exact universal functional.

Transcorrelated coupled cluster methods.

Transcorrelated coupled cluster and distinguishable cluster methods are presented. The Hamiltonian is similarity transformed with a Jastrow factor in the first quantization, which results in up to three-body integrals. The coupled cluster with singles and doubles equations on this transformed Hamiltonian are formulated and implemented. It is demonstrated that the resulting methods have a superior basis set convergence and accuracy to the corresponding conventional and explicitly correlated methods. Additionally, approximations for three-body integrals are suggested and tested.

Binding curve of the beryllium dimer using similarity-transformed FCIQMC: Spectroscopic accuracy with triple-zeta basis sets.

We demonstrate how similarity-transformed full configuration interaction quantum Monte Carlo (FCIQMC) based on the transcorrelated Hamiltonian can be applied to make highly accurate predictions for the binding curve of the beryllium dimer, marking the first case study of a molecular system with this method. In this context, the non-Hermitian transcorrelated Hamiltonian, resulting from a similarity transformation with a Jastrow factor, serves the purpose to effectively address dynamic correlation beyond the used basis set and thus allows for obtaining energies close to the complete basis set limit from FCIQMC already with moderate basis sets and computational effort. Building on results from other explicitly correlated methods, we discuss the role of the Jastrow factor and its functional form, as well as potential sources for size consistency errors, and arrive at Jastrow forms that allow for high accuracy calculations of the vibrational spectrum of the beryllium dimer.

Exact Density Functional Obtained via the Levy Constrained Search.

A stochastic minimization method for a real-space wave function, Ψ(r1, r2...r n), constrained to a chosen density, ρ(r), is developed. It enables the explicit calculation of the Levy constrained search, F[ρ] = minΨ→ρ⟨Ψ| T̂ + V̂ ee|Ψ⟩, which gives the exact functional of density functional theory. This general method is illustrated in the evaluation of F[ρ] for densities in one dimension with a soft-Coulomb interaction. Additionally, procedures are given to determine the first and second functional derivatives, δ F/δρ(r) and δ2 F/[δρ(r)δρ(r')]. For a chosen external potential, v(r), the functional and its derivatives are used in minimizations over densities to give the exact energy, E v, without needing to solve the Schrödinger equation.

Fermionic Statistics in the Strongly Correlated Limit of Density Functional Theory.

Exact pieces of information on the adiabatic connection integrand, Wλ[ρ], which allows evaluation of the exchange-correlation energy of Kohn-Sham density functional theory, can be extracted from the leading terms in the strong coupling limit (λ → ∞, where λ is the strength of the electron-electron interaction). In this work, we first compare the theoretical prediction for the two leading terms in the strong coupling limit with data obtained via numerical implementation of the exact Levy functional in the simple case of two electrons confined in one dimension, confirming the asymptotic exactness of these two terms. We then carry out a first study on the incorporation of the Fermionic statistics at large coupling λ, both numerical and theoretical, confirming that spin effects enter at orders ∼e-√λ.

On the accuracy of density functional theory and wave function methods for calculating vertical ionization energies.

The best practice in computational methods for determining vertical ionization energies (VIEs) is assessed, via reference to experimentally determined VIEs that are corroborated by highly accurate coupled-cluster calculations. These reference values are used to benchmark the performance of density functional theory (DFT) and wave function methods: Hartree-Fock theory, second-order Møller-Plesset perturbation theory, and Electron Propagator Theory (EPT). The core test set consists of 147 small molecules. An extended set of six larger molecules, from benzene to hexacene, is also considered to investigate the dependence of the results on molecule size. The closest agreement with experiment is found for ionization energies obtained from total energy difference calculations. In particular, DFT calculations using exchange-correlation functionals with either a large amount of exact exchange or long-range correction perform best. The results from these functionals are also the least sensitive to an increase in molecule size. In general, ionization energies calculated directly from the orbital energies of the neutral species are less accurate and more sensitive to an increase in molecule size. For the single-calculation approach, the EPT calculations are in closest agreement for both sets of molecules. For the orbital energies from DFT functionals, only those with long-range correction give quantitative agreement with dramatic failing for all other functionals considered. The results offer a practical hierarchy of approximations for the calculation of vertical ionization energies. In addition, the experimental and computational reference values can be used as a standardized set of benchmarks, against which other approximate methods can be compared.

Local scaling correction for reducing delocalization error in density functional approximations.

Delocalization error is one of the most fundamental and dominant errors that plagues presently used density functional approximations. It is responsible for a large class of problems in the density functional theory calculations. For an effective and universal alleviation of the delocalization error, we develop a local scaling correction scheme by imposing the Perdew-Parr-Levy- Balduz linearity condition to local regions of a system. Our novel scheme is applicable to various mainstream density functional approximations. It substantially reduces the delocalization error, as exemplified by the significantly improved description of dissociating molecules, transition-state species, and charge-transfer systems. The usefulness of our novel scheme affirms that the explicit treatment of fractional electron distributions is essentially important for reducing the intrinsic delocalization error associated with approximate density functionals.

Qualitative breakdown of the unrestricted Hartree-Fock energy.

The stretching of closed-shell molecules is a qualitative problem for restricted Hartree-Fock that is usually circumvented by the use of unrestricted Hartree-Fock (UHF). UHF is well known to break the spin symmetry at the Coulson-Fischer point, leading to a discontinuous derivative in the potential energy surface and incorrect spin density. However, this is generally not considered as a major drawback. In this work, we present a set of two electron molecules which magnify the problem of symmetry breaking and lead to drastically incorrect potential energy surfaces with UHF. These molecules also fail with unrestricted density-functional calculations where a functional such as B3LYP gives both symmetry breaking and an unphysically low energy due to the delocalization error. The implications for density functional theory are also discussed.

The derivative discontinuity of the exchange-correlation functional.

The derivative discontinuity is a key concept in electronic structure theory in general and density functional theory in particular. The electronic energy of a quantum system exhibits derivative discontinuities with respect to different degrees of freedom that are a consequence of the integer nature of electrons. The classical understanding refers to the derivative discontinuity of the total energy as a function of the total number of electrons (N), but it can also manifest at constant N. Examples are shown in models including several hydrogen systems with varying numbers of electrons or nuclear charge (Z), as well as the 1-dimensional Hubbard model (1DHM). Two sides of the problem are investigated: first, the failure of currently used approximate exchange-correlation functionals in DFT and, second, the importance of the derivative discontinuity in the exact electronic structure of molecules, as revealed by full configuration interaction (FCI). Currently, all approximate functionals, including hybrids, miss the derivative discontinuity, leading to basic errors that can be seen in many ways: from the complete failure to give the total energy of H2 and H2(+), to the missing gap in Mott insulators such as stretched H2 and the thermodynamic limit of the 1DHM, or a qualitatively incorrect density in the HZ molecule with two electrons and incorrect electron transfer processes. Description of the exact particle behaviour of electrons is emphasised, which is key to many important physical processes in real systems, especially those involving electron transfer, and offers a challenge for the development of new exchange-correlation functionals.

Dramatic changes in electronic structure revealed by fractionally charged nuclei.

Discontinuous changes in the electronic structure upon infinitesimal changes to the Hamiltonian are demonstrated. These are revealed in one and two electron molecular systems by full configuration interaction (FCI) calculations when the realm of the nuclear charge is extended to be fractional. FCI electron densities in these systems show dramatic changes in real space and illustrate the transfer, hopping, and removal of electrons. This is due to the particle nature of electrons seen in stretched systems and is a manifestation of an energy derivative discontinuity at constant number of electrons. Dramatic errors of density functional theory densities are seen in real space as this physics is missing from currently used approximations. The movements of electrons in these simple systems encapsulate those in real physical processes, from chemical reactions to electron transport and pose a great challenge for the development of new electronic structure methods.

Extension of many-body theory and approximate density functionals to fractional charges and fractional spins.

The exact conditions for density functionals and density matrix functionals in terms of fractional charges and fractional spins are known, and their violation in commonly used functionals has been shown to be the root of many major failures in practical applications. However, approximate functionals are designed for physical systems with integer charges and spins, not in terms of the fractional variables. Here we develop a general framework for extending approximate density functionals and many-electron theory to fractional-charge and fractional-spin systems. Our development allows for the fractional extension of any approximate theory that is a functional of G(0), the one-electron Green's function of the non-interacting reference system. The extension to fractional charge and fractional spin systems is based on the ensemble average of the basic variable, G(0). We demonstrate the fractional extension for the following theories: (1) any explicit functional of the one-electron density, such as the local density approximation and generalized gradient approximations; (2) any explicit functional of the one-electron density matrix of the non-interacting reference system, such as the exact exchange functional (or Hartree-Fock theory) and hybrid functionals; (3) many-body perturbation theory; and (4) random-phase approximations. A general rule for such an extension has also been derived through scaling the orbitals and should be useful for functionals where the link to the Green's function is not obvious. The development thus enables the examination of approximate theories against known exact conditions on the fractional variables and the analysis of their failures in chemical and physical applications in terms of violations of exact conditions of the energy functionals. The present work should facilitate the calculation of chemical potentials and fundamental bandgaps with approximate functionals and many-electron theories through the energy derivatives with respect to the fractional charge. It should play an important role in developing accurate approximate density functionals and many-body theory.

Derivative discontinuity, bandgap and lowest unoccupied molecular orbital in density functional theory.

The conventional analysis of Perdew and Levy, and Sham and Schlüter shows that the functional derivative discontinuity of the exchange-correlation density functional plays a critical role in the correct prediction of bandgaps, or the chemical hardness. In a recent work by the present authors, explicit expressions for bandgap prediction with all common types of exchange-correlation functionals have been derived without invoking the concept of exchange-correlation energy functional derivative discontinuity at all. We here analyze the two approaches and establish their connection and difference. The present analysis further leads to several important results: (1) The lowest unoccupied molecular orbital (LUMO) in DFT has as much meaning in describing electron addition as the highest occupied molecular orbital (HOMO) in describing electron removal. (2) Every term in the total energy functional contributes to the energy gap because of the discontinuity of the derivative of the density (or density matrix) with respect to the number of electrons, ((∂ρ(s)(r('),r))/∂N)(v(s) ), at integers. (3) Consistent with the Perdew-Levy-Sham-Schlüter conclusion that the exact Kohn-Sham energy gap differs from the fundamental bandgap by a finite correction due to the functional derivative discontinuity of the exchange-correlation energy, we show that the exchange-correlation functional cannot be an explicit and differentiable functional of the electron density, either local or nonlocal. The last result is further strengthened when we consider Mott insulators. There, the exact exchange-correlation functional needs to have an explicitly discontinuous (nondifferentiable) dependence on the density or the density matrix. (4) We obtain exact conditions on the derivatives of total energy with respect to the spin-up and spin-down number of electrons.

Analytical evaluation of Fukui functions and real-space linear response function.

Many useful concepts developed within density functional theory provide much insight for the understanding and prediction of chemical reactivity, one of the main aims in the field of conceptual density functional theory. While approximate evaluations of such concepts exist, the analytical and efficient evaluation is, however, challenging, because such concepts are usually expressed in terms of functional derivatives with respect to the electron density, or partial derivatives with respect to the number of electrons, complicating the connection to the computational variables of the Kohn-Sham one-electron orbitals. Only recently, the analytical expressions for the chemical potential, one of the key concepts, have been derived by Cohen, Mori-Sánchez, and Yang, based on the potential functional theory formalism. In the present work, we obtain the analytical expressions for the real-space linear response function using the coupled perturbed Kohn-Sham and generalized Kohn-Sham equations, and the Fukui functions using the previous analytical expressions for chemical potentials of Cohen, Mori-Sánchez, and Yang. The analytical expressions are exact within the given exchange-correlation functional. They are applicable to all commonly used approximate functionals, such as local density approximation (LDA), generalized gradient approximation (GGA), and hybrid functionals. The analytical expressions obtained here for Fukui function and linear response functions, along with that for the chemical potential by Cohen, Mori-Sánchez, and Yang, provide the rigorous and efficient evaluation of the key quantities in conceptual density functional theory within the computational framework of the Kohn-Sham and generalized Kohn-Sham approaches. Furthermore, the obtained analytical expressions for Fukui functions, in conjunction with the linearity condition of the ground state energy as a function of the fractional charges, also lead to new local conditions on the exact functionals, expressed in terms of the second-order functional derivatives. We implemented the expressions and demonstrate the efficacy with some atomic and molecular calculations, highlighting the importance of relaxation effects.

Improving band gap prediction in density functional theory from molecules to solids.

A novel nonempirical scaling correction method is developed to tackle the challenge of band gap prediction in density functional theory. For finite systems the scaling correction largely restores the straight-line behavior of electronic energy at fractional electron numbers. The scaling correction can be generally applied to a variety of mainstream density functional approximations, leading to significant improvement in the band gap prediction. In particular, the scaled version of a modified local density approximation predicts band gaps with an accuracy consistent for systems of all sizes, ranging from atoms and molecules to solids. The scaled modified local density approximation thus provides a useful tool to quantitatively characterize the size-dependent effect on the energy gaps of nanostructures.

Revealing noncovalent interactions.

Molecular structure does not easily identify the intricate noncovalent interactions that govern many areas of biology and chemistry, including design of new materials and drugs. We develop an approach to detect noncovalent interactions in real space, based on the electron density and its derivatives. Our approach reveals the underlying chemistry that compliments the covalent structure. It provides a rich representation of van der Waals interactions, hydrogen bonds, and steric repulsion in small molecules, molecular complexes, and solids. Most importantly, the method, requiring only knowledge of the atomic coordinates, is efficient and applicable to large systems, such as proteins or DNA. Across these applications, a view of nonbonded interactions emerges as continuous surfaces rather than close contacts between atom pairs, offering rich insight into the design of new and improved ligands.

Discontinuous nature of the exchange-correlation functional in strongly correlated systems.

Standard approximations for the exchange-correlation functional have been found to give big errors for the linearity condition of fractional charges, leading to delocalization error, and the constancy condition of fractional spins, leading to static correlation error. These two conditions are now unified and extended to states with both fractional charge and fractional spin to give a much more stringent condition: the exact energy functional is a plane, linear along the fractional charge coordinate and constant along the fractional spin coordinate with a line of discontinuity at the integer. Violation of this condition underlies the failure of all known approximate functionals to describe the gaps in strongly correlated systems. It is shown that explicitly discontinuous functionals of the density or orbitals that go beyond these currently used smooth approximations is the key for the application of density functional theory to strongly correlated systems.

Second-Order Perturbation Theory with Fractional Charges and Fractional Spins.

In this work the behavior of MP2 for fractional occupations is investigated. The consideration of fractional charge behavior gives a simple derivation of an expression for the chemical potential (or the derivative of energy with respect to the number of electrons) of MP2. A generalized optimized effective potential formalism (OEP) has been developed in which the OEP is a nonlocal potential, which can be applied to explicit functionals of the orbitals and eigenvalues and also facilitates the evaluation of the chemical potential. The MP2 derivative improves upon the corresponding Koopmans' theorem in Hartree-Fock theory for the ionization energy and also gives a good estimate of the electron affinity. In strongly correlated systems with degeneracies and fractional spins, MP2 diverges, and another corrected second-order perturbative method ameliorates this failure for the energy but still does not recapture the correct behavior for the energy derivatives that yield the gap. Overall we present a view of wave function based methods and their behavior for fractional charges and spins that offers insight into the application of these methods to challenging chemical problems.