Robust to what?

The hidden civilizational invariants we rely on are not guaranteed, especially in the age of AI.

Articles published here are typically written by individual Torchbearer Community members, with input and editing from others. Views expressed do not represent an official position of Torchbearer Community as a whole unless otherwise noted.

In 1996, Abbott launched ritonavir, sold as Norvir, a drug that proved highly effective against HIV. Its manufacturing process and crystal structure were well understood. Then in 1998 a denser, near-insoluble second form of the drug appeared, and capsules stopped dissolving, becoming clinically useless.

The cause was the formation of a new crystal form, one that would not dissolve in water. Worse, the nucleation of these crystals was autocatalytic: each crystal of the new form acted as a seed template for producing more. Once this second form of ritonavir appeared, it seeded all production at the plant, contaminated equipment, and was likely carried as microscopic contamination of staff from the Chicago facility to a manufacturing site in Italy. In time, making the original form of ritonavir at either plant became impossible.

No atoms in the drug had changed. The elemental composition was identical. The atoms had simply reorganised, spontaneously, into a shape the body cannot absorb, and that arrangement spread until it crippled manufacturing capacity. There was no warning before the new form appeared, and no way to convert it back at scale. The eventual fix was a reformulated gel capsule, which took years and hundreds of millions of dollars to develop.

This was not a freak occurrence. A paper titled “How many ritonavir cases are there still out there?” estimated that 15-45% of marketed small-molecule drugs may possess an alternative, more stable form that nobody has found yet.

Our civilisation feels robust in many ways, but it is only robust to what it has met before. We rely on hidden invariants. By hidden invariant, I mean a condition our systems silently rely on because it has always held before: that this molecule dissolves, that predators recognise prey, that genes are passed on equally by each parent, that a shared software library is benign, that a crop monoculture will not meet its perfect pathogen, etc.

If these hidden invariants do not hold, the consequences are catastrophic. Intelligence allows for efficient search processes, and more powerful intelligence means a more powerful search through the space of ways our assumptions can fail.

Unlucky molecules

In December 2024, writing in Science, 38 researchers including Nobel laureates warned of the dangers of mirror life. All known life is homochiral: right-handed nucleotides, left-handed amino acids. Mirror life would be organisms with that handedness reversed. A mirror image of the molecules making up a cell does not sound dangerous on first impression, yet these scientists warned that it could devastate ecosystems and drive extinctions. The danger of mirror life lies in its invisibility. Immune systems, phages (the viruses that infect bacteria), and predators have all been tuned to one chirality. A mirror microbe could have carte blanche to romp through our ecosystems with no natural enemies to check its growth. Mirror life does not exist, and building it would likely be a decade or more of research away even for a determined effort (though ASI could radically shorten these timelines). There are also coordinated international efforts to halt work toward it.

When we picture a disease spreading epidemiologically, we usually think of bacteria, viruses, or fungi. But some diseases spread this way despite a causative agent that has no genome, metabolism, or cell. Prions are misfolded proteins, where the misfolded prion acts as a template, causing other proteins to misfold in turn. The most familiar example is mad cow disease. Prion diseases are untreatable, have no vaccine, and kill within months to years of infection. They resist most ordinary sterilisation and inactivation procedures, so they can form persistent reservoirs in the environment. Chronic wasting disease in deer and elk has spread across 37 US states. The animals shed prions in their faeces, urine, and saliva, which then persist in the soil, driving further spread from these environmental reservoirs alongside direct transmission. No human case has been confirmed to date, though it has not been conclusively ruled out.

Failures that reproduce

Most multicellular organisms reproduce sexually, so their genes show 50 percent inheritance, one allele from the mother and one from the father. Genetic variants are therefore diluted slowly through a population. But this inheritance can be biased through gene drives, where a gene is more likely than chance to be passed to offspring. CRISPR now makes designing and deploying gene drives feasible. In 2018, researchers built a CRISPR gene drive targeting the doublesex gene in Anopheles gambiae, a malaria-carrying mosquito species. It spread to 100 percent prevalence within seven to eleven generations and collapsed caged populations entirely. The same logic that could erase malaria from a region could, with a different target, reshape or remove a species before anyone had agreed that it should.

In 2019, scientists reported that at least 501 amphibian species had declined over fifty years, resulting in roughly 90 extinctions, the largest documented loss of biodiversity attributable to a single disease. The cause was the spread of the chytrid fungus Batrachochytrium dendrobatidis. Its global spread was driven by trade and development breaking down the ecological barriers that had once kept it contained.

Self-reproduction of failures is also perniciously present in computer networks. Software worms are code that replicates across networks without human action. A classic example is the 2017 NotPetya worm that spread worldwide from a single compromised software update in Ukraine. It crippled the shipping giant Maersk, the drugmaker Merck, and FedEx, causing an estimated ten billion dollars in damage. It was labelled by the White House as the most destructive and costly cyberattack in history.

Computer systems aren’t secure

When people hear about a cybersecurity failure, they usually hold a model in their heads where the software could have been made secure and the developers simply failed to do it. The reality is harder. Any computer system that allows sharing, general functionality, and transitive information flow cannot fully prevent the spread of malicious code, because the key detection problems are formally undecidable. The defender must secure every avenue of attack, while the attacker needs to find only one open path.

In 2024, a backdoor was inserted into xz Utils, a compression library buried deep in the Linux software supply chain, after a multi-year campaign to win the trust of its maintainers. It was caught almost by accident. An engineer noticed that logging in over SSH was taking a fraction of a second longer than it should, and investigated discovering the backdoor. That same year, a faulty update from the security vendor CrowdStrike, took down around 8.5 million Windows machines, resulting in grounded flights and disruption to hospitals and banks worldwide. The scale of our exposure is hard to overestimate. Verizon’s 2026 Data Breach Investigations Report found that exploiting software vulnerabilities had become the single most common way breaches begin, for the first time in the report’s nineteen-year history. They also found that AI is already compressing the time between a flaw becoming known and its being exploited from months to hours. It has been reported that General Joshua Rudd, who leads the National Security Agency and the Pentagon’s Cyber Command, said that Anthropic’s Mythos model “broke into almost all of our classified systems, not in weeks, but in hours”.

Efficiency makes monocultures

When you eat a banana in any Western nation, you are almost certainly eating one variety: the Cavendish. It makes up roughly half of global production and nearly all exported bananas. A soil fungus called TR4 threatens it, and more than 80 percent of the world’s banana production rests on TR4-susceptible varieties. The fungus has many transmission routes, spreading on soil, tools, boots, vehicles, and water. It survives in soil for decades, and it causes total yield loss in infected plants. We built this sameness in pursuit of efficiency, and in doing so we created a single point of failure.

The banana is not unusual. We have optimised software stacks, cloud providers, payment rails, operating systems, and the open-source libraries that everything else is built on into the same kind of monoculture. The “xz backdoor” mattered precisely because so much of the internet leans on that one small piece of shared code, the way the global fruit trade leans on one cultivar. Verizon’s same report found that breaches involving a third party now account for 48 percent of the total. Efficiency narrows the target, and a narrow target is easier to hit.

The blast is not the damage

When we think about the horror of nuclear weapons, we worry about the direct effects of the blast: the explosion and the radioactive fallout. The blast and fallout kill directly. The larger toll would likely come later, through effects on climate, crops, fisheries, and livestock. Scientists writing in Nature Food in 2022 estimated that a nuclear war between India and Pakistan could cause around two billion deaths through these knock-on effects, and a war between the United States and Russia around five billion.

Our exposure does not require a war to find us. A severe solar storm of the kind that struck in 1859, before we had wired the planet, would today threaten power grids, high-frequency communications, and satellites all at once. The National Academies have warned that the cascading social and economic effects of losing those systems together are poorly understood. Modernity is a stack of layers, and some of those layers are global.

The search through dark space

All of these vulnerabilities turn on the failure of a hidden invariant that our world’s stability quietly depends on. We keep finding them by luck, at the frontier of our scientific knowledge. We are still here because nothing has yet searched the full space of levers controlling those invariants. Evolution by natural selection searches that space slowly, rewarding only the local steps that raise an organism’s fitness. Human science and engineering search it narrowly, wandering the corridors lit by our previous discoveries. Both leave almost all of the space dark. The examples above are what we tripped over in the light.

A superintelligence, an entity of greater scientific capability than all of humanity combined, running scalably at machine speed, would be able to search a great deal of that darkness. How many more levers, and how much more powerful, remain to be found?

Imagine you are the common ancestor of humans and our chimp cousins. You cannot represent agriculture, writing, or a rifle. You cannot imagine cities, deforestation, or the burning of fossil fuels changing your climate. Your failure to imagine these harms to your species might feel like safety. That feeling is the error. When people ask how a superintelligence could possibly cause our extinction, I picture those ancestral chimps asking each other whether the upright cousins are going to steal their bananas.

Today’s AI models are not superintelligence. But unless the exponential curves of progress in AI bend soon, its creation looks like where we are heading. We keep discovering new ways in which our world is more vulnerable than we had realised. We cannot defend against the threats we are unable even to conceive. A superintelligence would find the levers controlling the invariants our civilisation rests on. The question is whether our civilisation has the sanity to stop AI development until we know how to stop it pulling them.