I don’t think it would make any meaningful difference whether I asked or not. In this hypothetical, there’s an unaligned magical genie on the loose! I probably won’t ask it anything, and hope that it goes away instead of responding to whatever sleep-addled nonsense I happen to think up on the spot.
If I asked it to solve anything, it would probably be the genie alignment problem. AI alignment is a potential future existential risk, while an unaligned being of immense magical power is a definite right now existential risk.
It’s just going to do whatever it’s inclined to do anyway, but maybe there’s a microscopic chance that “whatever it’s inclined to do” happens to be “what I’d like it to do when I ask”. It almost certainly won’t be, but I figure that maybe there’s a better chance than many of the other things I might do in response to a genie showing up at my bedroom window.
I don’t think it would make any meaningful difference whether I asked or not. In this hypothetical, there’s an unaligned magical genie on the loose! I probably won’t ask it anything, and hope that it goes away instead of responding to whatever sleep-addled nonsense I happen to think up on the spot.
If I asked it to solve anything, it would probably be the genie alignment problem. AI alignment is a potential future existential risk, while an unaligned being of immense magical power is a definite right now existential risk.
It’s just going to do whatever it’s inclined to do anyway, but maybe there’s a microscopic chance that “whatever it’s inclined to do” happens to be “what I’d like it to do when I ask”. It almost certainly won’t be, but I figure that maybe there’s a better chance than many of the other things I might do in response to a genie showing up at my bedroom window.