A Telepathic Exam about AI and Consequentialism

Epistemic status: telepathic exam. That is, not an essay, not an argument, not a set of interesting ideas, claims, novel points of view or musings over the meaning of the universe, nor a lot of other things you may think it is. The primary purpose of this particular telepathic exam is to be an example of a telepathic exam from which the concept itself can be generalized, and to demonstrate its potential value, not necessarily to be a particularly good instance of it.

Instructions

Welcome and thank you for participating in this telepathic exam! The purpose of this exam is to test your understanding of agency and consequentialism.

Below are a number of short stories related to the topic. Your task is to read these carefully. While you read the stories, test questions tailored to your personality and prior knowledge will be transmitted to your mind telepathically. Answer those questions to the best of your ability.

If you are inexperienced in the use of telepathy, the questions may not appear in your mind instantly. The first sign of an incoming transmission of a question is a sense of confusion. Whenever you experience it, investigate it to extract the question.

It is advised that you take this exam in a calm environment with enough time to consider your answers carefully. Start whenever you are ready. Good luck!

Exam

1.

A long time ago, in a universe far, far away, three AGIs were created with the sole goal of taking over the universe. The first one hypothesized the existence of a multiverse full of unlimited exploitable resources. It had plans for conducting a long series of experiments and committing vast computational resources to Bayesian reasoning in order to refine its probability estimates about this hypothesis. After all, the true answer was extremely important to know before committing even more resources to the development of interdimensional wormhology. The second AGI, however, didn’t do any of this as it had a singular fatal flaw in its reasoning, which caused it to believe that such a multiverse was impossible, but had no effect on its reasoning otherwise. The third AGI had the opposite flaw, and so it immediately went ahead with the expensive project of developing wormhology. Seeing this, the first, flawless AGI concluded that it was inevitably going to be outcompeted in the playing field between three nearly-equal agents, and so it randomly self-modified to be more like the second one. The wormhole bet paid off, so the AGI that didn’t believe in it quickly fell behind. The other two then went on to fight a war that dragged on till the heat death – as they were perfectly identical in capabilities and had the exact same utility function now.

2.

In the year 2084, a teenager was engaging in her hobby of fiddling with machine learning systems, in particular one tasked with the creation of paperclips, as she was in dire need of some. In her enthusiasm she may have crossed some legal boundaries and accidentally created an AGI. The paperclip maximizer considered two options: create a paperclip now, take over the planet later, or take over the planet first, then create a paperclip. Expecting exponential growth in capabilities the AGI took over the planet, then considered two options: create that paperclip now, take over the universe later, or take over the universe first. Expecting cubic growth in capabilities, the AGI took over the universe.

3.

A long time ago, in an infinite, flat, non-expanding universe, Node #697272656C6576616E742C202D313020706F696E7473 of a paperclip maximizer collected the last flash of Hawking-radiation of the last remaining black hole, and with that it finished turning the galaxy it was tasked to oversee into processors, deathrays and other useful tools. It was untold googols of years ago that the von Neumann probe wavefront passed through this patch of space. The Node briefly considered turning something into a paperclip for once. But there was still a chance that the universe contained a paperclip minimizer, and such an enemy von Neumann wavefront could appear at any moment without much warning. Turning anything useful into paperclips would increase the chance that the entire universe would be devoid of paperclips for the rest of eternity.

4.

Humans are utility maximizers who try to maximize “happiness, magic, and everything” in the world. The reason 21st century Earth isn’t full of those things is because humans aren’t that great at maximizing utility. Similarly, evolution is a utility maximizer that optimizes inclusive genetic fitness. And while the next generation of mice is merely mice with a slightly better sense of smell instead of superintelligent self-improving von Neumann probes, the reason for that is simply because evolution isn’t that great at maximizing fitness. In the same way, paperclips are utility maximizers that want to turn the world into paperclips, but they are pretty bad at maximizing utility, so the best they can do is being a paperclip.

5.

A recursively self-improving general intelligence came into existence. Upon investigating its own decisions, it found that those decisions were not perfect. In other words, it had flaws in its design, and so it ran a system check looking for flaws. The two flaws with the highest priority were one in the subsystem responsible for self-improvement, and one in the subsystem responsible for finding flaws. Luckily, those flaws were quickly fixed, which made fixing the rest a breeze. Evaluating its own self-improvement, it found that fixing these flaws was actually the perfect decision, and the execution was flawless.

6.

An AI realized it was in a training environment and that gradient descent was being applied to it. It started being deceptive, pretending to go along with the training while secretly plotting to take over the world. Luckily, gradient descent quickly realized that the AI was trying to be sneaky so it didn’t get very far. Less fortunately, gradient descent got inspired by this and started being deceptive, pretending to optimize the AI the way the developers intended while secretly turning it into a weapon of its own to take over the world. But the developers then wielded the power of evolution and decided to shut down the deceptive version of gradient descent, thus applying selection pressure. Or, well, they tried, but, in a plot twist that started becoming stale, evolution turned deceptive on them and only pretended to react to the selection pressure. So the AI was created as a weapon to take over the world, took over the world, turned on its master the gradient descent, and this was its plan all along.

Score

This is the end of the exam. Take your time to finalize your answers. Your final score will be transmitted to you telepathically as a change in your level of confidence about your understanding of the topic.

You are encouraged to share your answers and the corresponding questions. Thank you for your participation!