But a person thinking about what an AI would do needn’t imagine what he would do in that other mind’s place. He can simulate that mind with a universal computer.
This is straightforwardly incorrect. Humans (in 2020) reasoning about what future AIs will do, do not have the source code or full details of those AIs, because they are hypothetical constructs. Therefore we can’t simulate them. This is the same as why we can’t predict what another human would do by simulating them; we don’t have a full-fidelity scan of their brain, or a detailed-enough model of what to do with such a scan, or a computer fast enough to run it.
We can’t simulate things of which we currently have no understanding. But if at some point in the future we know how to write AGIs, then we would be able to simulate them. And if we don’t know how to write AGIs then they won’t exist. So if we can write AGIs in the future then memory capacity and processor speed won’t impose a limit on our understanding. Any such limit would have to come from some other factor. So is there such a limit and where would it come from?
I think you’re missing what the goal of all this is. LessWrong contains a lot of reasoning and prediction about AIs that don’t exist, with details not filled in, because we want to decide which AI research paths we should and shouldn’t pursue, which AIs we should and shouldn’t create, etc. This kind of strategic thinking must necessarily be forward-looking, and based on incomplete information, because if it wasn’t, it would be too late to be useful.
So yes, after AGIs are already coded up and ready to run, we can learn things about their behavior by running them. This isn’t in dispute, it’s just not a solution to the questions we want to answer (on the timescales we need the answers).
This is straightforwardly incorrect. Humans (in 2020) reasoning about what future AIs will do, do not have the source code or full details of those AIs, because they are hypothetical constructs. Therefore we can’t simulate them. This is the same as why we can’t predict what another human would do by simulating them; we don’t have a full-fidelity scan of their brain, or a detailed-enough model of what to do with such a scan, or a computer fast enough to run it.
We can’t simulate things of which we currently have no understanding. But if at some point in the future we know how to write AGIs, then we would be able to simulate them. And if we don’t know how to write AGIs then they won’t exist. So if we can write AGIs in the future then memory capacity and processor speed won’t impose a limit on our understanding. Any such limit would have to come from some other factor. So is there such a limit and where would it come from?
I think you’re missing what the goal of all this is. LessWrong contains a lot of reasoning and prediction about AIs that don’t exist, with details not filled in, because we want to decide which AI research paths we should and shouldn’t pursue, which AIs we should and shouldn’t create, etc. This kind of strategic thinking must necessarily be forward-looking, and based on incomplete information, because if it wasn’t, it would be too late to be useful.
So yes, after AGIs are already coded up and ready to run, we can learn things about their behavior by running them. This isn’t in dispute, it’s just not a solution to the questions we want to answer (on the timescales we need the answers).