Yes, if it is impossible to remain in control of AIs then you will have value drift
Wait, that’s not my argument. I was saying that while people like you are trying to develop technologies that let you “remain in control”, others with shorter planning horizons or think they have simple, easy to transmit values will already be deploying new AGI capabilities, so you’ll fall behind with every new development. This is what I’m suggesting only a singleton can prevent.
You could try to minimize this kind of value drift by speeding up “AI control” progress but it’s really hard for me to see how you can speed it up enough to not lose a competitive race with those who do not see a need to solve this problem, or think they can solve a much easier problem. The way I model AGI development in a slow-FOOM scenario is that AGI capability will come in spurts along with changing architectures, and it’s hard to do AI safety work “ahead of time” because of dependencies on AI architecture. So each time there is a big AGI capability development, you’ll be forced to spend time to develop new AI safety tech for that capability/architecture, while others will not wait to deploy it. Even a small delay can lead to a large loss since AIs can be easily copied and more capable but uncontrolled AIs would quickly take over economic niches occupied by existing humans and controlled AIs. Even assuming secure rights for what you already own on Earth, your share of the future universe will become smaller and smaller as most of the world’s new wealth goes to uncontrolled AIs or AIs with simple values.
Where do you see me going wrong here? If you think I’m just too confident in this model, what alternative scenario can you suggest, where people like you and I (or our values) get to keep a large share of the future universe just by speeding up the onset of serious AI safety work?
Wait, that’s not my argument. I was saying that while people like you are trying to develop technologies that let you “remain in control”, others with shorter planning horizons or think they have simple, easy to transmit values will already be deploying new AGI capabilities, so you’ll fall behind with every new development. This is what I’m suggesting only a singleton can prevent.
You could try to minimize this kind of value drift by speeding up “AI control” progress but it’s really hard for me to see how you can speed it up enough to not lose a competitive race with those who do not see a need to solve this problem, or think they can solve a much easier problem. The way I model AGI development in a slow-FOOM scenario is that AGI capability will come in spurts along with changing architectures, and it’s hard to do AI safety work “ahead of time” because of dependencies on AI architecture. So each time there is a big AGI capability development, you’ll be forced to spend time to develop new AI safety tech for that capability/architecture, while others will not wait to deploy it. Even a small delay can lead to a large loss since AIs can be easily copied and more capable but uncontrolled AIs would quickly take over economic niches occupied by existing humans and controlled AIs. Even assuming secure rights for what you already own on Earth, your share of the future universe will become smaller and smaller as most of the world’s new wealth goes to uncontrolled AIs or AIs with simple values.
Where do you see me going wrong here? If you think I’m just too confident in this model, what alternative scenario can you suggest, where people like you and I (or our values) get to keep a large share of the future universe just by speeding up the onset of serious AI safety work?