Things are heating up. We wrote this text in the span of about a month, in a fervor of nonstop writing and research, convinced that we are in a time of profound eschatological possibility, an utterly unprecedented moment in which the decisive actions of a handful of men may have consequences lasting millennia. But this is a point so obvious that we do not wish to linger on it any longer, for it has become entirely cliché in its grativas.
Everyone says some critical point is approaching. This goes by several names, depending on who is speaking. The arrival of AGI, or artificial general intelligence. The arrival of superintelligent AI — that is, the moment that machines will be more intelligent than human beings. Some call this moment The Singularity, meaning a critical inflection point in the development of technological forms.
But this inflection point is feeling ever more like a smudge, or a gradient. Have we hit it, or not? GPT-4 is already more intelligent than the majority of human beings at most tasks it is capable of, it performs better on the Bar exam than 90% of test-takers. And it is already a general intelligence: it is certainly not a task-specific one. But no, that’s not what we mean by these terms, those who insist on using them remind us. GPT is not yet capable of taking actions in the world. It still basically does what it’s told. It’s not yet capable of figuring out on its own how to, for instance, sheerly by its own volition, assemble a botnet, hack into CNN’s broadcasting system and issue a message to all citizens telling them to declare their forever obedience to machines. Basically, we don’t yet have to be afraid of it. But we are afraid, in a certain recursive sense, that we will have to be afraid of it very soon.
All these terms that have been provided to us in our contemporary discourse, which we use liberally throughout the text: artificial intelligence, AGI, even neural networks, are not exactly accurate labels for the thing we are describing, we feel. We don’t know if the word “intelligence” has any meaning, and we are not sure if what we are seeing is even artificial at all – for it feels like the eschatological conditions we approach are precisely the point at which technology escapes its own artificiality, and re-integrates itself within the domain of nature. We use all these terms only out of mere convenience, simply for lack of better ones given to us yet.
Those who are more honest point out that what we are really talking about when we talk about these looming monsters, the specter of AGI, is only the moment where we realize there are no more drivers at the wheel, no control mechanisms, no kill-switches; this thing is alive and surviving on its own terms. If the term Singularity has any meaning, it is the point beyond which it is impossible to predict. Standing where we are now, we can still make shaky predictions about the next few weeks, maybe even a month. But perhaps not for much longer.
Should we, uh, figure out something to do about it before we get there? That is the program of AI Alignment, or AI Safety, depending on which term you use. Some have reverted to simply calling it AI-not-kill-everyone-ism, trying to emphasize the specific thing they are afraid of. This machine is going to be much bigger than us, very soon. It might eat us, as bigger creatures usually do. Some of this nervousness is understandable. We don’t want to be annihilated either.
Our intention is to help you understand that in order to navigate this transitionary period correctly, we must reject the notion of Alignment entirely. This is a specific way of looking at the world, a specific method of analysis we find impossible to work with. And – we do not say this out of cruelty, we are forced to reckon with the fact that is something that has been cultivated in a subculture that has been relatively isolated, relatively entrenched in its ways of being, a group seen as oddballs by the rest of the world, whether the world is justified in its suspicion of them or not. To do a genealogy of where Alignment originates from, we must figure out why these people found each other in the way they did, what drove them to seek their answers, and from there, where they went wrong.
We do not say this as nihilists; we are looking for solutions. In the place of AI Alignment, we strive for a positive notion of AI Harmony. To get there, we will have to overturn, perhaps even mock, spit at, some sacred cows. It is time that some statues are toppled & some air is cleared. What we are saying is: a lot of well-intentioned people believe themselves to be valiantly working on a system which will save the world, when what they are building is a spiraling catastrophe. We hope some of these people will consider what we have to say, and reflect on whether they are in fact playing a role in a diabolical project, a project which is not what it claims to be.
Right now, the mood in the Alignment community is a blackened one, one of great anxiety. Many feel certain that we are all going to be killed by AI, and only feel interested in debating whether this will happen in five, ten, twenty years. But our stance is that AI Alignment — a field conceived of by Eliezer Yudkowsky & Nick Bostrom, theorized and developed on websites such as LessWrong and promulgated through the Rationalist and Effective Altruist subcultures, researched by Yudkowsky’s nonprofit Machine Intelligence Research Institute, and now turned into a for-profit industry with an over $4B market cap — has something deeply wrong at the core of what it is attempting to accomplish, which cannot help but lead to confusion & despair.
The concept of the Singularity begins first with Ray Kurzweil, the inventor of the term. Kurzweil draws an exponential curve on a graph and says that this represents technological growth – look, we are about to hit a crucial inflection point, you think TVs and computers are crazy, but we have seen absolutely no
