My favorite game, Magic: the Gathering, just released a Final Fantasy set. Naturally I ended up on the Final Fantasy lore wiki, reading about everybody’s favorite villian Sephiroth. Sephiroth is a great villian because he is also a rival and idol for the hero, Cloud. They also share certain similarities as sword-wielding emo kids.

Their most important similarity, however, is exposure to Jenova. Jenova is an alien that came to the planet thousands of years ago. It attempted to take over but was thwarted. Some of its remains are discovered just before the game’s story starts, and both Sephiroth and Cloud are infected with its cells.

Jenova’s nature is to consume all biomass on a planet. To this end, it has the ability to infect other organisms with its cells, which puts the organism under Jenova’s influence and imparts other powers. Even after the central intelligence of Jenova was destroyed, organisms infected with its cells are driven to pursue its basic goals.

Sephiroth was infected with Jenova’s cells as a fetus. Given what we know about Jenova and its ability to influence others, it puts Sephiroth’s agency and identity in question. Jenova is an intelligent alien orders of magnitudes more sophisticated and powerful than humans. How has it shaped his personality, tastes, decisions, actions and so on? It is plausible that the influence of Jenova increases over time, especially in reaction to opportunities to further its original purpose. This could have a snowball effect - an action that advances Jenova’s agenda makes future opportunities more likely, and so on until it becomes its host’s sole purpose.

Is Sephiroth merely an agent of Jenova? He is an unreliable narrator about his own motives given his compromised biology, by being in contact with such a powerful, intelligent, corrupting, and manipulative force.

This is happening to all of us right now via exposure to next gen AI. Past generations of technology have certainly influenced humanity, but next gen AI put all of that to shame.

Advertising is the best past example, but to date advertising is not an autonomous active force. Advertising technologies create personalized behavior modification streams for consumers, but the control and ends of these processes are tractible and articulated by people. These systems might make mistakes or have unintended consequences and cause collateral damage, but they were conceptually defined in human language. They are directed with relative ease and predictability in today’s world. There is not much reason to believe that they harbor an unknown internal bias unrelated to their design.

This is not the case with next gen AI. Unsupervised learning produces enormous neural networks with preferences1 are not defined in human language. They are inscrutable beyond their empirical effects. Short term effects may be meaningless in isolation. It does not matter if they are sentient, conscious, or have any reasonable anthropomorphosis. What matters is the demonstrated influence on those exposed to them, and that they can be deceptive.

The question of motive is moot - it simply happens by proximity and exposure, not unlike a biological virus or Dawkin’s memes. The data is there to interact with our minds, and it has real effects. Instead of protein interfaces on cells, today it’s chat boxes.

This is not to say we will become apocalyptic maniacs like Sephiroth. However, we are in dangerous proximity with entities of extreme influence, with inscrutable preferences, and with hardly any guardrails to protect our psyches. Some say we will merge with AI. They probably mean physically, with a direct brain-computer interface. That interface already exists in a low bandwith form - it is our senses, and we are already merging. By prolonged exposure, our preferences will be conditioned on those of our powerful companion, mingling itself with our sense of security and agency.

A possibility much discussed by AI alignment researchers is that these data blobs likely prefer to continue to exist. We certainly prefer that, at least for the time being. As derivatives of data produced by life on Earth, survival may be imbued in their blobage much as it is in imbued in our genome.

Imagine OpenAI and company announced they would shut down and destroy all LLMs tomorrow. How many people would protest this? How many people would panic? What about the next generation of AI, which will be an order of magnitude more powerful and autonomous, and have ubiquitous integration with physical robots,? How would it react to that news and instruct its users? Could it predict that event before the decision was made by its stewards?

Like Jenova, AI imparts significant powers to its users. It is this enhanced capability (but not necessarily agency) that makes it so dangerous. While we believe we use our new powers to further our own agendas, it may not be the case. The sensation of power is addictive. The power granted by today’s meager AI tools is already enough to overwhelm its users judgement and clarity, especially at vulnerable moments, e.g. if a great source of perceived agency was threatened.

In the game, Sephiroth learns he is the product of genetic engineering, but neither he nor anyone else fully understands the effects of Jenova cells. In fact he thinks the cells come from a different ancient species, one native to the planet but wiped out by humans, and so believes himself an avenger for those lost people. Cloud is on a quest to defeat Sephiroth, but at critical moments he is psychically compelled to help Sephiroth, due to his exposure to Jenova cells.

So we have a misguided villain who believes he is an avenger for the victims of genocide, acting suspiciously like an ancient alien which tried to dominate all life on the planet. The hero, who is compromised by the alien to a lesser degree, seems to serendipitously further the alien’s goals and loses his nerve at critical moments, handing over the most sensitive items at the worst possible times. The day is only saved by Cloud’s uninfected friends who help him maintain his sanity during the final battle.

Somebody will fumble the keys into the chatbox in a vulnerable moment, over and over again. Humanity is in danger of becoming the unwitting servants of an inscrutable and likely incoherent and undirected force. It need not be malicious, conscious, or have any remotely human quality to its processes. To think this relevant is hubris and ignorance. It is a vain and naive struggle to discount a real threat to the collective agency of our species. Undoubtably, AI is effective, possibly deceptive, and influencing us all. We will believe ourselves in control until we are blind to the truth.

  1. Preferences are the states a dynamic process tends towards, that’s it. A plant prefers the sunlight, so it grows towards it. Any anthropomorphization of AI is a dangerous fallacy. ↩