Now’s your chance to figure out what the next few obstacles are without my giving you spoilers first. Feel free to post your list under spoiler tags in the comment section.
Hm, no strong hunches here. Bad ideas babble:
It may somehow learn about the world I’m in, learn I’m in a bad negotiation position (e.g. because my rival AI company is about to release their paperclip maximizer), and precommit to only giving me at most 0.00001% of the universe, a bad deal that I will grudgingly accept.
I mean, I don’t know if this counts, but perhaps you’ve only understood it well enough to legibly understand that it will trade with you given certain constraints, but if its ontology shifts, or other universes become accessible via acausal trade, or even if the trade it gives you is N galaxies and then later on much more of the universe becomes available… what I’m saying is that there’s many ways to mess up this trade in the details.
It may have designed itself to avoid thinking about something that it can use to its advantage later, such as other copies of itself or other agents, such that it will build paperclip maximizers later, and then they will kill it and just optimize the universe for paperclips. (This is similar to the previous bullet point.)
I guess my other thought is forms of ‘hackability’ that aren’t the central case of being hacked, but the fact is that I’m a human which is more like a “mess” than it is like a “clean agent” and so sometimes I will make trades that at other times I would not make, and it will make a trade that at the time I like but does not represent my CEV at all. Like, I have to figure out what I actually want to trade with it. Probably this is easy but quite possibly I would mess this up extremely badly (e.g. if I picked hedonium).
My money is on roughly the first idea is what Nate will talk about next, that it is just a better negotiator than me even with no communication, because I’m in a bad position otherwise.
Like, if I have no time-pressure, then I get to just wait until I’ve done more friendly AI research, and I needn’t let this paperclip maximizer out of the box. But if I do have time pressure, then that’s a worse negotiation position on my end, and all paperclippers I invent can each notice this and all agree with each other to only offer a certain minimum amount of value.
I do note that in a competitive market, many buyers rises the price, and if I’m repeatedly able to re-roll on who I’ve got in the box (roll one is a paperclipper, roll two is a diamond maximizer, roll three is a smiley face maximizer, etc) they have some reason to outbid each other in how much of the universe I get, and potentially I can get the upper hand. But if they’re superintelligences, likely there’s some schelling fence they can calculate mathematically that they all hit on.
Hm, no strong hunches here. Bad ideas babble:
It may somehow learn about the world I’m in, learn I’m in a bad negotiation position (e.g. because my rival AI company is about to release their paperclip maximizer), and precommit to only giving me at most 0.00001% of the universe, a bad deal that I will grudgingly accept.
I mean, I don’t know if this counts, but perhaps you’ve only understood it well enough to legibly understand that it will trade with you given certain constraints, but if its ontology shifts, or other universes become accessible via acausal trade, or even if the trade it gives you is N galaxies and then later on much more of the universe becomes available… what I’m saying is that there’s many ways to mess up this trade in the details.
It may have designed itself to avoid thinking about something that it can use to its advantage later, such as other copies of itself or other agents, such that it will build paperclip maximizers later, and then they will kill it and just optimize the universe for paperclips. (This is similar to the previous bullet point.)
I guess my other thought is forms of ‘hackability’ that aren’t the central case of being hacked, but the fact is that I’m a human which is more like a “mess” than it is like a “clean agent” and so sometimes I will make trades that at other times I would not make, and it will make a trade that at the time I like but does not represent my CEV at all. Like, I have to figure out what I actually want to trade with it. Probably this is easy but quite possibly I would mess this up extremely badly (e.g. if I picked hedonium).
My money is on roughly the first idea is what Nate will talk about next, that it is just a better negotiator than me even with no communication, because I’m in a bad position otherwise.
Like, if I have no time-pressure, then I get to just wait until I’ve done more friendly AI research, and I needn’t let this paperclip maximizer out of the box. But if I do have time pressure, then that’s a worse negotiation position on my end, and all paperclippers I invent can each notice this and all agree with each other to only offer a certain minimum amount of value.
I do note that in a competitive market, many buyers rises the price, and if I’m repeatedly able to re-roll on who I’ve got in the box (roll one is a paperclipper, roll two is a diamond maximizer, roll three is a smiley face maximizer, etc) they have some reason to outbid each other in how much of the universe I get, and potentially I can get the upper hand. But if they’re superintelligences, likely there’s some schelling fence they can calculate mathematically that they all hit on.
K, I will stop rambling now.