This led me down a path of finding when the paperclip maximizer thought experiment was introduced and who introduced it. (I intended to make some commentary based on the year, but the origin question turns out to be tricky.)
It also seems perfectly possible to have a superintelligence whose sole goal is something completely arbitrary, such as to manufacture as many paperclips as possible, and who would resist with all its might any attempt to alter this goal.
[...]
This could result, to return to the earlier example, in a superintelligence whose top goal is the manufacturing of paperclips, with the consequence that it starts transforming first all of earth and then increasing portions of space into paperclip manufacturing facilities.
I wouldn’t be as disturbed if I thought the class of hostile AIs I was talking about would have any of those qualities except for pure computational intelligence devoted to manufacturing an infinite number of paperclips. It turns out that the fact that this seems extremely “stupid” to us relies on our full moral architectures.
This led me down a path of finding when the paperclip maximizer thought experiment was introduced and who introduced it. (I intended to make some commentary based on the year, but the origin question turns out to be tricky.)
Wikipedia credits Bostrom in 2003:
The passages from the given citation:
However, a comment by Lucifer on the LessWrong tag page instead credits Yudkowsky in that same year, with this message (2003-03-11):
I’ll leave the rabbit hole there for now.