Assessing the risk of takeover catastrophe from large language models.

This article presents a risk analysis of large language models (LLMs), a type of "generative" artificial intelligence (AI) system that produces text, commonly in response to textual inputs from human users. The article is specifically focused on the risk of LLMs causing an extreme catastrophe in which they do something akin to taking over the world and killing everyone. The possibility of LLM takeover catastrophe has been a major point of public discussion since the recent release of remarkably capable LLMs such as ChatGPT and GPT-4. This arguably marks the first time when actual AI systems (and not hypothetical future systems) have sparked concern about takeover catastrophe. The article's analysis compares (A) characteristics of AI systems that may be needed for takeover, as identified in prior theoretical literature on AI takeover risk, with (B) characteristics observed in current LLMs. This comparison reveals that the capabilities of current LLMs appear to fall well short of what may be needed for takeover catastrophe. Future LLMs may be similarly incapable due to fundamental limitations of deep learning algorithms. However, divided expert opinion on deep learning and surprise capabilities found in current LLMs suggests some risk of takeover catastrophe from future LLMs. LLM governance should monitor for changes in takeover characteristics and be prepared to proceed more aggressively if warning signs emerge. Unless and until such signs emerge, more aggressive governance measures may be unwarranted.

Public health and nuclear winter: addressing a catastrophic threat.

Despite the end of the Cold War, the world still has thousands of nuclear weapons and adversarial relations between the countries that possess them. A nuclear war could cause large and abrupt global environmental change known as nuclear winter, with potentially devastating public health consequences. A significant line of natural science research characterizes nuclear winter and its potential effect on global food security, but less has been done on the human impacts and policy implications. Therefore, this Viewpoint proposes an interdisciplinary research and policy agenda to understand and address the public health implications of nuclear winter. Public health research can apply existing tools developed for the study of other environmental and military issues. Public health policy institutions can help build preparedness and community resilience to nuclear winter. Given the extreme potential severity of nuclear winter, it should be treated as a major global public health challenge to be addressed by public health institutions and researchers.

Pandemic refuges: Lessons from 2 years of COVID-19.

This paper relates evidence from the COVID-19 pandemic to the concept of pandemic refuges, as developed in literature on global catastrophic risk. In this literature, a refuge is a place or facility designed to keep a portion of the population alive during extreme global catastrophes. COVID-19 is not the most extreme pandemic scenario, but it is nonetheless a very severe global event, and it therefore provides an important source of evidence. Through the first 2 years of the COVID-19 pandemic, several political jurisdictions have achieved low spread of COVID-19 via isolation from the rest of the world and can therefore classify as pandemic refuges. Their suppression and elimination of COVID-19 demonstrates the viability of pandemic refuges as a risk management measure. Whereas prior research emphasizes island nations as pandemic refuges, this paper uses case studies of China and Western Australia to show that other types of jurisdictions can also successfully function as pandemic refuges. The paper also refines the concept of pandemic refuges and discusses implications for future pandemics.

Assessing natural global catastrophic risks.

The risk of global catastrophe from natural sources may be significantly larger than previous analyses have found. In the study of global catastrophic risk (GCR), one line of thinking posits that deep human history renders natural GCRs insignificant. Essentially, the fact that natural hazards did not cause human extinction at any previous time makes it unlikely that they would do so now. This paper finds flaws in this argument, refines the theory of natural GCR, analyzes the space of natural GCRs, and presents implications for decision-making and research. The paper analyzes natural climate change, natural pandemics, near-Earth objects (asteroids, comets, and meteors), space weather (coronal mass ejections, solar flares, and solar particle events), stellar explosions (gamma-ray bursts and supernovae), and volcanic eruptions. Almost all natural GCR scenarios involve important interactions between the natural hazard and human civilization. Several natural GCR scenarios may have high ongoing probability. Deep human history provides little information about the resilience of modern global civilization to natural global catastrophes. The natural GCRs should not be dismissed on grounds of deep human history. Work on natural GCRs should account for their important human dimensions. A case can even be made for abandoning the distinction between natural and anthropogenic GCR.

Nonhuman Value: A Survey of the Intrinsic Valuation of Natural and Artificial Nonhuman Entities.

To be intrinsically valuable means to be valuable for its own sake. Moral philosophy is often ethically anthropocentric, meaning that it locates intrinsic value within humans. This paper rejects ethical anthropocentrism and asks, in what ways might nonhumans be intrinsically valuable? The paper answers this question with a wide-ranging survey of theories of nonhuman intrinsic value. The survey includes both moral subjects and moral objects, and both natural and artificial nonhumans. Literatures from environmental ethics, philosophy of technology, philosophy of art, moral psychology, and related fields are reviewed, and gaps in these literatures are identified. Although the gaps are significant and much work remains to be done, the survey nonetheless demonstrates that those who reject ethical anthropocentrism have considerable resources available to develop their moral views. Given the many very high-stakes issues involving both natural and artificial nonhumans, and the sensitivity of these issues to how nonhumans are intrinsically valued, this is a vital project to pursue.

Risk-Risk Tradeoff Analysis of Nuclear Explosives for Asteroid Deflection.

To prevent catastrophic asteroid-Earth collisions, it has been proposed to use nuclear explosives to deflect away earthbound asteroids. However, this policy of nuclear deflection could inadvertently increase the risk of nuclear war and other violent conflict. This article conducts risk-risk tradeoff analysis to assess whether nuclear deflection results in a net increase or decrease in risk. Assuming nonnuclear deflection options are also used, nuclear deflection may only be needed for the largest and most imminent asteroid collisions. These are low-frequency, high-severity events. The effect of nuclear deflection on violent conflict risk is more ambiguous due to the complex and dynamic social factors at play. Indeed, it is not clear whether nuclear deflection would cause a net increase or decrease in violent conflict risk. Similarly, this article cannot reach a precise conclusion on the overall risk-risk tradeoff. The value of this article comes less from specific quantitative conclusions and more from providing an analytical framework and a better overall understanding of the policy decision. The article demonstrates the importance of integrated analysis of global risks and the policies to address them, as well as the challenge of quantitative evaluation of complex social processes such as violent conflict.

Climate change: evidence of human causes and arguments for emissions reduction.

In a recent editorial, Raymond Spier expresses skepticism over claims that climate change is driven by human actions and that humanity should act to avoid climate change. This paper responds to this skepticism as part of a broader review of the science and ethics of climate change. While much remains uncertain about the climate, research indicates that observed temperature increases are human-driven. Although opinions vary regarding what should be done, prominent arguments against action are based on dubious factual and ethical positions. Thus, the skepticisms in the recent editorial are unwarranted. This does not diminish the general merits of skeptical intellectual inquiry.

Intrinsic ethics regarding integrated assessment models for climate management.

In this essay we develop and argue for the adoption of a more comprehensive model of research ethics than is included within current conceptions of responsible conduct of research (RCR). We argue that our model, which we label the ethical dimensions of scientific research (EDSR), is a more comprehensive approach to encouraging ethically responsible scientific research compared to the currently typically adopted approach in RCR training. This essay focuses on developing a pedagogical approach that enables scientists to better understand and appreciate one important component of this model, what we call intrinsic ethics. Intrinsic ethical issues arise when values and ethical assumptions are embedded within scientific findings and analytical methods. Through a close examination of a case study and its application in teaching, namely, evaluation of climate change integrated assessment models, this paper develops a method and case for including intrinsic ethics within research ethics training to provide scientists with a comprehensive understanding and appreciation of the critical role of values and ethical choices in the production of research outcomes.