Another meta line of argument is to consider how many people have strongly held, but mutually incompatible philosophical positions.
I’ve been banging my head against figuring out why this line of argument doesn’t seem convincing to many people for at least a couple of years. I think, ultimately, it’s probably because it feels defeatable by plans like “we will make AIs solve alignment for us, and solving alignment includes solving metaphilosophy & then object-level philosophy”. I think those plans are doomed in a pretty fundamental sense, but if you don’t think that, then they defeat many possible objections, including this one.
As they say: Everyone who is hopeful has their own reason for hope. Everyone who is doomful[1]...
There seem to me different categories of being doomful.
There are people who think that for theoretic reasons AI alignment is hard or impossible.
There are also people who are more focused practical issues like AI companies being run in a profit maximizing way and having no incentives to care for most of the population.
Saying, “You can’t AI box for theoretical reasons” is different from saying “Nobody will AI box for economic reasons”.
I’ve been banging my head against figuring out why this line of argument doesn’t seem convincing to many people for at least a couple of years. I think, ultimately, it’s probably because it feels defeatable by plans like “we will make AIs solve alignment for us, and solving alignment includes solving metaphilosophy & then object-level philosophy”. I think those plans are doomed in a pretty fundamental sense, but if you don’t think that, then they defeat many possible objections, including this one.
As they say: Everyone who is hopeful has their own reason for hope. Everyone who is doomful[1]...
In fact it’s not clear to me. I think there’s less variation, but still a fair bit.
There seem to me different categories of being doomful.
There are people who think that for theoretic reasons AI alignment is hard or impossible.
There are also people who are more focused practical issues like AI companies being run in a profit maximizing way and having no incentives to care for most of the population.
Saying, “You can’t AI box for theoretical reasons” is different from saying “Nobody will AI box for economic reasons”.