There’s another problem. Even if you get it so that it just answers questions using only its own processing power, if you ask it how to best make paperclips, it will most likely respond by designing a paperclip-maximizer.
Only if you ask it the most effective method of ultimately producing the largest number of paperclips, without upper limit and without regard to schedule and cost. But that’s not something anybody wants to know in the first place. What a stationery manufacturer actually wants to know is the lowest cost method of producing a certain number of paperclips within a certain schedule, and the answer to that will not be a paperclip maximizer.
If you can afford to build a paper clip maximizer for less than it costs for a given number of pape clips, then if you want to build at least that many paperclips, the cheapest way will be to build a paperclip maximizer. It’s not going to turn the world into paperclips in a reasonable time, but it probably will build enough within that time limit.
No it won’t. A paperclip maximizer is basically a two-stage process: first, conquer the world, then start producing paperclips via an efficient manufacturing process. If what you want is to turn the whole world into paperclips, and you don’t care about cost or schedule, then the first stage is necessary. If all you want is a billion paperclips as cheap as possible, then it’s entirely irrational to incur the cost and schedule impact of the first stage when you could hop directly to the second stage and just build an efficient paperclip factory.
There’s another problem. Even if you get it so that it just answers questions using only its own processing power, if you ask it how to best make paperclips, it will most likely respond by designing a paperclip-maximizer.
Only if you ask it the most effective method of ultimately producing the largest number of paperclips, without upper limit and without regard to schedule and cost. But that’s not something anybody wants to know in the first place. What a stationery manufacturer actually wants to know is the lowest cost method of producing a certain number of paperclips within a certain schedule, and the answer to that will not be a paperclip maximizer.
If you can afford to build a paper clip maximizer for less than it costs for a given number of pape clips, then if you want to build at least that many paperclips, the cheapest way will be to build a paperclip maximizer. It’s not going to turn the world into paperclips in a reasonable time, but it probably will build enough within that time limit.
No it won’t. A paperclip maximizer is basically a two-stage process: first, conquer the world, then start producing paperclips via an efficient manufacturing process. If what you want is to turn the whole world into paperclips, and you don’t care about cost or schedule, then the first stage is necessary. If all you want is a billion paperclips as cheap as possible, then it’s entirely irrational to incur the cost and schedule impact of the first stage when you could hop directly to the second stage and just build an efficient paperclip factory.
Good point. Perhaps it would build a paperclip-making Vonn Neumann machine, or just a machine that maximizes paperclips within a given time.
Besides rwallace’s point, I would think that there’s a large class of useful questions you could ask that wouldn’t produce such a dangerous response.
(Are you sure you’re not searching for problems with the idea until it looks like it couldn’t offer an advantage over building FAI?)