Thanks for the comment! Taking your points in turn:
- I am curious that you see this as me saying superintelligent AI will be less dangerous, as to me it means it will be more. It will be able to dominate you in the usual hyper-competent sense but also may accidentally screw up some super-advanced physics and kill you that way too. It sounds like I should have stressed this more. I guess there are people that think AI sucks and will continue to suck, and therefore why worry about existential risk, so maybe by stressing AI fallibility I’m riding their energy a bit too hard to have made myself clear. I’ll add a footnote to clarify better.
- I agree that knowing-that reduces the amount of failure needed for knowing-how. My point is that the latter is the thing we actually care about though when we talk about intelligence. Memorising information is inconsequential without some practical purpose to put it to. Even if you’re just reading stuff to get your world model straight, it’s because you want to be able to use that model to take more successful actions in the world.
- I’m not completely sure I follow your questions about failure-reduction-potential upper-bounds. My best guess is that you mean can sufficient knowing-that reduce the amount of failure required to acquire new skills to a very low level? I think theoretical knowledge is mostly generated by practical action—trying stuff and writing down what happened—either individually or on a societal scale. So if an ASI wants to do something radically new then there won’t be any existing knowledge that can help it. For me, that means catastrophic or existential risk due to incompetence is a problem. I guess it reduces risk a little from the AI intentionally killing you, as it could mess up its plans in such a way as you survive, but long-term this reduction will be tiny as wiping out humans will not be in the ASI’s stretch zone for very long.
- Re your second point, I do not believe we will be able to recognise the errors an ASI is making. If it wants to kill us, it will be able to. My fear is that it will do it by accident anyway.
- Re your third point, I agree that AI is going to proliferate widely, and this is a big part of why I’m saying the usual recursive self-improvement story is too clean. There won’t be this gap between clearly dumber than humans to effectively omnipotent in which the AI is doing nothing but quietly gaining capabilities—labs will ship their products and people will use them and, while the shipped AI will be super impressive and useful, it will also screw a lot of things up. What I was getting at in my conclusion about the AI doing nothing out of fear of failure was more that if self-destructive actions we don’t understand come into its capabilities, and it knows this, we might find it gets risk-averse and reluctant to do some of the things we ask it to.
Ah, yes, that does clear it up! I definitely am much more on board, sorry I misread the first time, and the footnote helps a lot.
As for the questions I asked that weren’t clear, they’re much less relevant now that I have your clarification. But the idea was: I’m off the opinion that we have a lot more know-how buried and latent in all our know-that data such that many things humans have never done or even thought of being able to do could nevertheless be overdetermined (or nearly so) without additional experimental data.
Thanks for the comment! Taking your points in turn:
- I am curious that you see this as me saying superintelligent AI will be less dangerous, as to me it means it will be more. It will be able to dominate you in the usual hyper-competent sense but also may accidentally screw up some super-advanced physics and kill you that way too. It sounds like I should have stressed this more. I guess there are people that think AI sucks and will continue to suck, and therefore why worry about existential risk, so maybe by stressing AI fallibility I’m riding their energy a bit too hard to have made myself clear. I’ll add a footnote to clarify better.
- I agree that knowing-that reduces the amount of failure needed for knowing-how. My point is that the latter is the thing we actually care about though when we talk about intelligence. Memorising information is inconsequential without some practical purpose to put it to. Even if you’re just reading stuff to get your world model straight, it’s because you want to be able to use that model to take more successful actions in the world.
- I’m not completely sure I follow your questions about failure-reduction-potential upper-bounds. My best guess is that you mean can sufficient knowing-that reduce the amount of failure required to acquire new skills to a very low level? I think theoretical knowledge is mostly generated by practical action—trying stuff and writing down what happened—either individually or on a societal scale. So if an ASI wants to do something radically new then there won’t be any existing knowledge that can help it. For me, that means catastrophic or existential risk due to incompetence is a problem. I guess it reduces risk a little from the AI intentionally killing you, as it could mess up its plans in such a way as you survive, but long-term this reduction will be tiny as wiping out humans will not be in the ASI’s stretch zone for very long.
- Re your second point, I do not believe we will be able to recognise the errors an ASI is making. If it wants to kill us, it will be able to. My fear is that it will do it by accident anyway.
- Re your third point, I agree that AI is going to proliferate widely, and this is a big part of why I’m saying the usual recursive self-improvement story is too clean. There won’t be this gap between clearly dumber than humans to effectively omnipotent in which the AI is doing nothing but quietly gaining capabilities—labs will ship their products and people will use them and, while the shipped AI will be super impressive and useful, it will also screw a lot of things up. What I was getting at in my conclusion about the AI doing nothing out of fear of failure was more that if self-destructive actions we don’t understand come into its capabilities, and it knows this, we might find it gets risk-averse and reluctant to do some of the things we ask it to.
- Agree completely with your fourth point.
Ah, yes, that does clear it up! I definitely am much more on board, sorry I misread the first time, and the footnote helps a lot.
As for the questions I asked that weren’t clear, they’re much less relevant now that I have your clarification. But the idea was: I’m off the opinion that we have a lot more know-how buried and latent in all our know-that data such that many things humans have never done or even thought of being able to do could nevertheless be overdetermined (or nearly so) without additional experimental data.