“What for? If you had an aligned superintelligence, what would you tell it to do?” said Eliza.
“If I had an aligned superintelligence I wouldn’t be working for Bayeswatch. I wouldn’t be talking to you,” said Vi.
“Hypothetically,” said Eliza.
“Abstract morality is masturbation for philosophers. I live in the real world. Are you going to keep wasting my time or are you going to let me do my job?” said Vi.
Vi has fallen into Hanson’s mistake here: mistaking being actually serious about having something to protect with never childishly, irresponsibly indulging in reflecting on what you’re aiming at.
Yesterday I asked my esteemed co-blogger Robin what he would do with “unlimited power”, in order to reveal something of his character. Robin said that he would (a) be very careful and (b) ask for advice. I asked him what advice he would give himself. Robin said it was a difficult question and he wanted to wait on considering it until it actually happened. So overall he ran away from the question like a startled squirrel …
For it seems to me that Robin asks too little of the future. It’s all very well to plead that you are only forecasting, but if you display greater revulsion to the idea of a Friendly AI than to the idea of rapacious hardscrapple frontier folk...
I thought that Robin might be asking too little, due to not visualizing any future in enough detail. Not the future but any future. I’d hoped that if Robin had allowed himself to visualize his “perfect future” in more detail, rather than focusing on all the compromises he thinks he has to make, he might see that there were futures more desirable than the rapacious hardscrapple frontier folk.
It’s hard to see on an emotional level why a genie might be a good thing to have, if you haven’t acknowledged any wishes that need granting. It’s like not feeling the temptation of cryonics, if you haven’t thought of anything the Future contains that might be worth seeing.
Vi has fallen into Hanson’s mistake here: mistaking being actually serious about having something to protect with never childishly, irresponsibly indulging in reflecting on what you’re aiming at.