(I’m inspired to write this comment on the notion of empowerment because of Richard Ngo’s recent comment in another post on Towards a scale-free theory of intelligent agency, so I’ll both respond to the empowerment motion and part of the comment linked below:
Lastly: if at each point in time, the set of agents who are alive are in conflict with potentially-simpler future agents in a very destructive way, then they should all just Do Something Else. In particular, if there’s some decision-theoretic argument roughly like “more powerful agents should continue to spend some of their resources on the values’of their less-powerful ancestors, to reduce the incentives for inter-generational conflict”, even agents with very simple goals might be motivated by it. I call this “the generational contract”.
This depends a lot on how the conflict started, and in particular, I don’t think that we should do something else if the conflict arose out of severe alignment failures of AIs/radically augmented people, since the generational contract/UDT/LDT/FDT cannot be used as a substitute for alignment (this was the takeaway I got from reading Nate Soares’s post on Decision theory does not imply that we get to have nice things, and while @ryan_greenblatt commented that it’s unlikely that alignment failures end up in us getting extinct without decision theory saving us, note that us being saved can still be really, really rough, and probably ends up with billions of present humans dying without everyone else dying (though I don’t think about decision theory much), so conflicts with future agents cannot always be avoided if we mess up hard enough on the alignment problems of the future).
I like the idea of fractal empowerment, and IMO is one of my biggest ideals if we succeed at getting out of the risky state we are in, only rivaled by infra-Bayes Physicalism’s plan for alignment, which I currently think is called Physical Super-Imitation after the monotonicity principle managed to be removed, meaning way more preferences could be fit into than before:
That said, I have a couple of problems with the idea.
One of those problems is that empowering humans in the moment can conflict a lot with empowerment in the long run, and unfortunately I’m less confident than you that disempowering humans in the short-term is not going to be necessary in order to empower humans in the long run.
In particular, we might already have this conflict in the near future with biological design tools allowing people to create their own pandemics/superviruses:
Another problem is that the set of incentives that holds up democracy and makes it’s inefficiencies tolerable will absolutely be shredded by AGI, and the main incentive that goes away is you no longer need to consider the mass opinion on a lot of very important stuff, which fundamentally hurts democracy’s foundations, and importantly makes moderate redistribution not necessary for the economy to function, which means the elites won’t do it by default, combined with extreme redistribution being both unnecessary and counter-productive in industrial economies like ours, but unfortunately in the automation/intelligence age the only 2 sources of income are passive investment and whatever welfare you can get, so extreme redistribution is both less counter-productive (because it’s easier to confiscate the future sources of wealth) and more necessary for commoners to survive.
Finally, as @quetzal_rainbow has said, UDT works on your logical ancestor, not literal ancestors, and there needs to be some shared knowledge in order to coordinate, and thus the inter-temporal bargaining doesn’t really work out if you expect that current generations will have way less knowledge than future generations, which I expect to be the case.
(I’m inspired to write this comment on the notion of empowerment because of Richard Ngo’s recent comment in another post on Towards a scale-free theory of intelligent agency, so I’ll both respond to the empowerment motion and part of the comment linked below:
https://www.lesswrong.com/posts/5tYTKX4pNpiG4vzYg/towards-a-scale-free-theory-of-intelligent-agency#nigkBt47pLMi5tnGd):
To address this part of the linked comment:
This depends a lot on how the conflict started, and in particular, I don’t think that we should do something else if the conflict arose out of severe alignment failures of AIs/radically augmented people, since the generational contract/UDT/LDT/FDT cannot be used as a substitute for alignment (this was the takeaway I got from reading Nate Soares’s post on Decision theory does not imply that we get to have nice things, and while @ryan_greenblatt commented that it’s unlikely that alignment failures end up in us getting extinct without decision theory saving us, note that us being saved can still be really, really rough, and probably ends up with billions of present humans dying without everyone else dying (though I don’t think about decision theory much), so conflicts with future agents cannot always be avoided if we mess up hard enough on the alignment problems of the future).
https://www.lesswrong.com/posts/rP66bz34crvDudzcJ/decision-theory-does-not-imply-that-we-get-to-have-nice
Now I’ll address the fractal empowerment idea.
I like the idea of fractal empowerment, and IMO is one of my biggest ideals if we succeed at getting out of the risky state we are in, only rivaled by infra-Bayes Physicalism’s plan for alignment, which I currently think is called Physical Super-Imitation after the monotonicity principle managed to be removed, meaning way more preferences could be fit into than before:
https://www.lesswrong.com/posts/DobZ62XMdiPigii9H/non-monotonic-infra-bayesian-physicalism
https://www.lesswrong.com/posts/ZwshvqiqCvXPsZEct/the-learning-theoretic-agenda-status-2023#Physicalist_Superimitation
That said, I have a couple of problems with the idea.
One of those problems is that empowering humans in the moment can conflict a lot with empowerment in the long run, and unfortunately I’m less confident than you that disempowering humans in the short-term is not going to be necessary in order to empower humans in the long run.
In particular, we might already have this conflict in the near future with biological design tools allowing people to create their own pandemics/superviruses:
https://michaelnotebook.com/xriskbrief/index.html
Another problem is that the set of incentives that holds up democracy and makes it’s inefficiencies tolerable will absolutely be shredded by AGI, and the main incentive that goes away is you no longer need to consider the mass opinion on a lot of very important stuff, which fundamentally hurts democracy’s foundations, and importantly makes moderate redistribution not necessary for the economy to function, which means the elites won’t do it by default, combined with extreme redistribution being both unnecessary and counter-productive in industrial economies like ours, but unfortunately in the automation/intelligence age the only 2 sources of income are passive investment and whatever welfare you can get, so extreme redistribution is both less counter-productive (because it’s easier to confiscate the future sources of wealth) and more necessary for commoners to survive.
More below:
https://forum.effectivealtruism.org/posts/TMCWXTayji7gvRK9p/is-democracy-a-fad#Why_So_Much_Democracy__All_of_a_Sudden_
https://forum.effectivealtruism.org/posts/TMCWXTayji7gvRK9p/is-democracy-a-fad#Automation_and_Democracy
Finally, as @quetzal_rainbow has said, UDT works on your logical ancestor, not literal ancestors, and there needs to be some shared knowledge in order to coordinate, and thus the inter-temporal bargaining doesn’t really work out if you expect that current generations will have way less knowledge than future generations, which I expect to be the case.