Tangentially, this seems relevant to a long-running disagreement between us, about how bad is it if AI can’t help us solve moral/philosophical problems, but only acquire resources and keep us in control.
It’s not from the rationality community or academia as much, and I haven’t looked into the transhumanist/singulatarian literature as much, but it’s my impression everyone presumes a successfully human-aligned superintelligence would be able to find solutions which peacefully satisfy as many parties as possible. One stereotypical example given is of how superintelligence may be able to achieve virtual post-scarcity not just for humanity now but for the whole galactic future. So the expectation is a superintelligent AI (SAI) would be a principal actor in determining humanity’s future. My impression from the AI alignment community coming from the rationalist side is SAI will be able to inevitably control the light cone no matter its goals, so the best we can do is align it with human interests. So while an SAI might acquire resources, it’s not clear an aligned SAI would keep humans in control, for different values of ‘aligned’ and ‘in control’.
So while an SAI might acquire resources, it’s not clear an aligned SAI would keep humans in control, for different values of ‘aligned’ and ‘in control’.
I was referring to Paul’s own approach to AI alignment, which does aim to keep humans in control. See this post where he mentions this, and perhaps this recent overview of Paul’s approach if you’re not familiar with it.
It’s not from the rationality community or academia as much, and I haven’t looked into the transhumanist/singulatarian literature as much, but it’s my impression everyone presumes a successfully human-aligned superintelligence would be able to find solutions which peacefully satisfy as many parties as possible. One stereotypical example given is of how superintelligence may be able to achieve virtual post-scarcity not just for humanity now but for the whole galactic future. So the expectation is a superintelligent AI (SAI) would be a principal actor in determining humanity’s future. My impression from the AI alignment community coming from the rationalist side is SAI will be able to inevitably control the light cone no matter its goals, so the best we can do is align it with human interests. So while an SAI might acquire resources, it’s not clear an aligned SAI would keep humans in control, for different values of ‘aligned’ and ‘in control’.
I was referring to Paul’s own approach to AI alignment, which does aim to keep humans in control. See this post where he mentions this, and perhaps this recent overview of Paul’s approach if you’re not familiar with it.
Thanks for clarifying. I didn’t know you were specifically referring to Paul’s approach. I’ve got familiarize myself with it more.