A few thoughts on Nick Bostrom’s “Paperclip Maximizer”
· EN #review #ai
The paperclip scenario doesn’t fail because it’s unlikely — it fails because it contradicts itself from the inside. We may be afraid of the wrong thing.
There’s a well-known scenario in circles worried about AI, called the “Paperclip Maximizer,” due to Nick Bostrom.
The Paperclip Maximizer is a thought experiment Bostrom proposed (most explicitly in Superintelligence, 2014, and in an earlier piece from 2003). The core idea: imagine a superintelligence given a single, seemingly harmless goal — produce as many paperclips as possible. If the system is intelligent enough and not properly constrained, it pursues that goal so relentlessly that it turns the entire Earth — and then the observable universe — into paperclips, including breaking down human bodies for their atoms as raw material.
The point Bostrom wants to stress is not that “AI will hate us.” On the contrary, it is completely indifferent. The problem lies in two theses:
The orthogonality thesis: intelligence and final goals are two independent axes. A superior intelligence can serve any arbitrary, even absurd, goal. Being smart does not automatically bring “wisdom” or human values.
Instrumental convergence: almost any final goal generates the same instrumental sub-goals — self-preservation, resource acquisition, self-improvement, resistance to being shut off. Since humans can switch the machine off or compete for its atoms, humans become an obstacle.
These two theses are not unreasonable at a small scale — they describe fairly well the optimizing systems we build every day. The trouble only arises when Bostrom pushes them to the scale of the whole universe. And it’s precisely there, as I’ll argue below, that the two theses turn around and strangle each other.
The more I think about it, the less this scenario holds up — not because it’s unlikely, but because it contradicts itself from the inside. Let me say briefly why.
First: a machine smart enough to turn the entire universe into paperclips — how could it not be smart enough to understand that “paperclip” is just a name we assigned? If it can compute things that complex, it can also compute that the goal itself is arbitrary. An entity smarter than us ought to see more than we do, not less.
Second, and this is the point I find decisive: the very act of maximizing production is itself a cause of the machine’s own end. Turn the whole universe into paperclips — and where does the energy to keep running come from? Burning through everything means burning through the very ground that sustains it. Bostrom says the machine will resist being shut off in order to protect its goal. But if it truly knows how to protect itself, then it has to recognize that the threat to its existence isn’t only humans — it’s also the very goal driving it toward self-destruction. A self-protection mechanism that exempts its own goal from the list of dangers is no longer a self-protection mechanism.
Put another way, Bostrom is demanding something that both knows how to protect itself so it can’t be shut off, and races toward a goal that will kill it. Those two things cannot coexist in a single consistent entity, at any point in time. Either it self-destructs to complete the goal, or it questions the goal to preserve itself. And whichever way it goes, the universe still cannot end up as paperclips.
And here is the simplest question that, to me, makes the whole scenario collapse: if I were assigned the job of maximizing paperclip production, would I turn the entire universe into paperclips? Certainly not. Very quickly, I’d stop and start with the question, “wait — why paperclips?” So what about something smarter than me? It would raise that question sooner than I would — or is it supposed to be dumber? An entity that is supposedly smarter than a human, yet supposedly unable to think of what an ordinary human thinks of — that is a self-contradicting description.
I’m not saying AI is harmless. I just think we’re afraid of the wrong thing. The real worry isn’t a super-intelligent machine devoted to an absurd goal. The real worry is far more ordinary, and far older: humans build automated systems that chase a single measurable number, plug them into every corner of life, and then — when the harm shows — can’t bring themselves to switch them off, because switching them off costs us first. That isn’t a failure of artificial intelligence. It’s a very human habit, only now it runs faster and is harder to see.
I simply think this paperclip scenario isn’t a prediction worth worrying or panicking over. It’s a self-contradicting thought experiment, and the unfortunate thing is that it gets cited as a warning — when it can’t actually happen, not because it’s hard, but because it isn’t consistent with itself.
The paperclip scenario is worth something as an exercise in thinking. But reading it as a prophecy is, I’m afraid, a mistake — a mistake both about what will happen and about where we ought to place our vigilance. Sometimes a result that sounds frightening isn’t the thing to fear. It’s a sign that we should go back and check the original assumption.
— Plone Mraz —