I don’t love “overwhelmingly superintelligent” because AIs don’t necessarily have to be qualitatively smarter than humanity to overwhelm it
I think this more feature-than-bug – the problem is that it’s overwhelming. There are multiple ways to be overwhelming, what we want to avoid is a situation where an overwhelming, unfriendly AI exists. One way is not build AI of a given power level. The other is to increase the robustness of civilization. (I agree the term is fuzzy, but I think realistically the territory is fuzzy).
I think it’s a mistake to just mention that second thing as a parenthetical. There’s a huge difference between AIs that are already galaxy-brained superintelligences and AIs that could quickly build galaxy-brained superintelligences or modify themselves into galaxy-brained superintelligences—we should try to prevent the former category of AIs from building galaxy-brained superintelligences in ways we don’t approve of.
(did you mean “latter category?”)
Were you suggesting something other than “remove the parentheses?” Or did it seem like I was thinking about it in a confused way? Not sure which direction you thought the mistake was in.
(I think “already overwhelmingly strong” and “a short hop away from being overwhelming strong” are both real worrisome. The latter somewhat less worrisome, although I’d really prefer not building either until we are much more confident about alignment/intepretability)
I think this more feature-than-bug – the problem is that it’s overwhelming. There are multiple ways to be overwhelming, what we want to avoid is a situation where an overwhelming, unfriendly AI exists. One way is not build AI of a given power level. The other is to increase the robustness of civilization. (I agree the term is fuzzy, but I think realistically the territory is fuzzy).
When you’re thinking about how to mitigate the risks, it really matters which of these we’re talking about. I think there is some level of AI capability at which it’s basically hopeless to control the AIs; this is what I use “galaxy-brained superintelligence” to refer to. If you just want to talk about AIs that pose substantial risk of takeover, you probably shouldn’t use the word superintelligence in there, because they don’t obviously have to be superintelligences to pose takeover risk. (And it’s weird to use “overwhelmingly” as an adverb that modifies “superintelligent”, because the overwhelmingness isn’t about the level of intelligence, it’s about that and also the world. You could say “overwhelming, superintelligent AI” if you want to talk specifically about AIs that are overwhelming and also superintelligent, but that’s normally not what we want to talk about.)
I might retract the exact phrasing of my reply comment.
I think I was originally using overwhelmingly basically the way you’re using “galaxy brained”, and I feel like I have quibbles about the exact semantics of that phrase that feel about as substantial as your concern about overwhelming. (i.e. there is also a substantive difference between a very powerful brain hosted in a datacenter on Earth, and an AI that with a galaxy of resources)
What I mean by “overwhelmingly superintelligent” is “so fucking smart that humanity would have to have qualitatively changed in a similar orders-of-magnitude degree”, which probably in practice means humans also have to have augmented their own intelligence, or have escalated their AI control schemes pretty far, carefully wielding significantly-[but-not-overwhelming/galaxy-brained]-AI that oversees all of Earth’s security and is either aligned or the humans are really at threading the needle on control for quite powerful systems.
Were you suggesting something other than “remove the parentheses?” Or did it seem like I was thinking about it in a confused way? Not sure which direction you thought the mistake was in.
I think that it is worth conceptually distinguishing AIs that are uncontrollable from AIs that are able to build uncontrollable AIs, because the way you should handle those two kinds of AI are importantly different.
I think this more feature-than-bug – the problem is that it’s overwhelming. There are multiple ways to be overwhelming, what we want to avoid is a situation where an overwhelming, unfriendly AI exists. One way is not build AI of a given power level. The other is to increase the robustness of civilization. (I agree the term is fuzzy, but I think realistically the territory is fuzzy).
(did you mean “latter category?”)
Were you suggesting something other than “remove the parentheses?” Or did it seem like I was thinking about it in a confused way? Not sure which direction you thought the mistake was in.
(I think “already overwhelmingly strong” and “a short hop away from being overwhelming strong” are both real worrisome. The latter somewhat less worrisome, although I’d really prefer not building either until we are much more confident about alignment/intepretability)
When you’re thinking about how to mitigate the risks, it really matters which of these we’re talking about. I think there is some level of AI capability at which it’s basically hopeless to control the AIs; this is what I use “galaxy-brained superintelligence” to refer to. If you just want to talk about AIs that pose substantial risk of takeover, you probably shouldn’t use the word superintelligence in there, because they don’t obviously have to be superintelligences to pose takeover risk. (And it’s weird to use “overwhelmingly” as an adverb that modifies “superintelligent”, because the overwhelmingness isn’t about the level of intelligence, it’s about that and also the world. You could say “overwhelming, superintelligent AI” if you want to talk specifically about AIs that are overwhelming and also superintelligent, but that’s normally not what we want to talk about.)
I might retract the exact phrasing of my reply comment.
I think I was originally using overwhelmingly basically the way you’re using “galaxy brained”, and I feel like I have quibbles about the exact semantics of that phrase that feel about as substantial as your concern about overwhelming. (i.e. there is also a substantive difference between a very powerful brain hosted in a datacenter on Earth, and an AI that with a galaxy of resources)
What I mean by “overwhelmingly superintelligent” is “so fucking smart that humanity would have to have qualitatively changed in a similar orders-of-magnitude degree”, which probably in practice means humans also have to have augmented their own intelligence, or have escalated their AI control schemes pretty far, carefully wielding significantly-[but-not-overwhelming/galaxy-brained]-AI that oversees all of Earth’s security and is either aligned or the humans are really at threading the needle on control for quite powerful systems.
I think that it is worth conceptually distinguishing AIs that are uncontrollable from AIs that are able to build uncontrollable AIs, because the way you should handle those two kinds of AI are importantly different.
I think I agree with that and didn’t think of my post as claiming otherwise?