LLM Comparator: Interactive Analysis of Side-by-Side Evaluation of Large Language Models.

Evaluating large language models (LLMs) presents unique challenges. While automatic side-by-side evaluation, also known as LLM-as-a-judge, has become a promising solution, model developers and researchers face difficulties with scalability and interpretability when analyzing these evaluation outcomes. To address these challenges, we introduce LLM Comparator, a new visual analytics tool designed for side-by-side evaluations of LLMs. This tool provides analytical workflows that help users understand when and why one LLM outperforms or underperforms another, and how their responses differ. Through close collaboration with practitioners developing LLMs at Google, we have iteratively designed, developed, and refined the tool. Qualitative feedback from these users highlights that the tool facilitates in-depth analysis of individual examples while enabling users to visually overview and flexibly slice data. This empowers users to identify undesirable patterns, formulate hypotheses about model behavior, and gain insights for model improvement. LLM Comparator has been integrated into Google's LLM evaluation platforms and open-sourced.

Ultrafast isomerization initiated by X-ray core ionization.

Rapid proton migration is a key process in hydrocarbon photochemistry. Charge migration and subsequent proton motion can mitigate radiation damage when heavier atoms absorb X-rays. If rapid enough, this can improve the fidelity of diffract-before-destroy measurements of biomolecular structure at X-ray-free electron lasers. Here we study X-ray-initiated isomerization of acetylene, a model for proton dynamics in hydrocarbons. Our time-resolved measurements capture the transient motion of protons following X-ray ionization of carbon K-shell electrons. We Coulomb-explode the molecule with a second precisely delayed X-ray pulse and then record all the fragment momenta. These snapshots at different delays are combined into a 'molecular movie' of the evolving molecule, which shows substantial proton redistribution within the first 12 fs. We conclude that significant proton motion occurs on a timescale comparable to the Auger relaxation that refills the K-shell vacancy.