Can I write a “proof” on why we shouldn’t rely on human feedback?
Possibly such a proof exists. With more assumptions, you can get better information on human values, see here. This obviously doesn’t solve all concerns.
Iterated Distillation-Amplification seems pointless though cause the humans need to scale with the AGI
Can you elaborate on that point?
Should I write a list of bad assumptions people keep making in alignment work? [...] that suffering is a relevant risk from AGI (suffering is inefficient, it’s an anti-convergent goal)
Only a few people think about this a lot—I currently can only think of the Center on Long-Term Risk on the intersection of suffering focus and AI Safety. Given how bad suffering is, I’m glad that there are people thinking about it, and do not think that a simple inefficiency argument is enough.
Ethical constraints of fiddling with brains [...] We could solve this is we could fully simulate the human brain...
I hope I don’t misrepresent you by putting these two quotes together. Is your position that the ethical dilemmas of “fiddling with human brains” would be solved by, instead, just fiddling with simulated brains? If so, then I disagree: I think simulated brains are also moral patients, to the same degree that physical brains are. I like this fiction a lot.
Possibly such a proof exists. With more assumptions, you can get better information on human values, see here. This obviously doesn’t solve all concerns.
Those are great references! I’m going to add them to my reading list, thank you.
Only a few people think about this a lot—I currently can only think of the Center on Long-Term Risk on the intersection of suffering focus and AI Safety. Given how bad suffering is, I’m glad that there are people thinking about it, and do not think that a simple inefficiency argument is enough.
I’d have to flesh out my thinking here more, which was why it was a very short note. But essentially, I suspect generating suffering as a subgoal for an AGI is something like an anti-convergent goal: It makes almost all other goals harder to achieve. An intuitive example is the bio-industry which currently generates a lot of suffering. However, as soon as we develop ways to grow meat in labs, this will be vastly more efficient, and thus we will converge to using that. An animal (or human) uses up energy while suffering, and the suffering itself tends to lower productivity and health in so many ways that it is both inefficient in purpose and resources. That said, there can be a transition period (such as we have now with the bio-industry) where the high suffering state is the optimum for some window of time till a more efficient method is generated (e.g. lab grown meat). In that window, there would then of course but very much suffering for humanity. I wouldn’t expect that window be particularly big though, cause human suffering achieves very few goals (as in, it covers very little of the possible goal space an AGI might target) and if recursive self-improvement is true, then the window would simply pass fairly quickly.
I hope I don’t misrepresent you by putting these two quotes together. Is your position that the ethical dilemmas of “fiddling with human brains” would be solved by, instead, just fiddling with simulated brains? If so, then I disagree: I think simulated brains are also moral patients, to the same degree that physical brains are. I like this fiction a lot.
Hmm, good point. I’m struck by how I had considered this issue when in a conversation with someone else 6 weeks ago, but now didn’t surface this consideration in my notes… I feel something may be going on with being in a generative frame versus a critique frame. And I can probably use awareness of that to generate better ideas.
But essentially, I suspect generating suffering as a subgoal for an AGI is something like an anti-convergent goal: It makes almost all other goals harder to achieve.
I think I basically agree (though maybe not with as much high confidence as you), but I think that doesn’t mean that huge amounts of suffering will not dominate the future. For example, if there will be not one but many superintelligent AI systems determining the future, this might create suffering due to cooperation failures.
Thank you for writing this!
Possibly such a proof exists. With more assumptions, you can get better information on human values, see here.
This obviously doesn’t solve all concerns.
Can you elaborate on that point?
Only a few people think about this a lot—I currently can only think of the Center on Long-Term Risk on the intersection of suffering focus and AI Safety. Given how bad suffering is, I’m glad that there are people thinking about it, and do not think that a simple inefficiency argument is enough.
I hope I don’t misrepresent you by putting these two quotes together. Is your position that the ethical dilemmas of “fiddling with human brains” would be solved by, instead, just fiddling with simulated brains? If so, then I disagree: I think simulated brains are also moral patients, to the same degree that physical brains are. I like this fiction a lot.
Thank you for the comment!
Those are great references! I’m going to add them to my reading list, thank you.
I’d have to flesh out my thinking here more, which was why it was a very short note. But essentially, I suspect generating suffering as a subgoal for an AGI is something like an anti-convergent goal: It makes almost all other goals harder to achieve. An intuitive example is the bio-industry which currently generates a lot of suffering. However, as soon as we develop ways to grow meat in labs, this will be vastly more efficient, and thus we will converge to using that. An animal (or human) uses up energy while suffering, and the suffering itself tends to lower productivity and health in so many ways that it is both inefficient in purpose and resources. That said, there can be a transition period (such as we have now with the bio-industry) where the high suffering state is the optimum for some window of time till a more efficient method is generated (e.g. lab grown meat). In that window, there would then of course but very much suffering for humanity. I wouldn’t expect that window be particularly big though, cause human suffering achieves very few goals (as in, it covers very little of the possible goal space an AGI might target) and if recursive self-improvement is true, then the window would simply pass fairly quickly.
Hmm, good point. I’m struck by how I had considered this issue when in a conversation with someone else 6 weeks ago, but now didn’t surface this consideration in my notes… I feel something may be going on with being in a generative frame versus a critique frame. And I can probably use awareness of that to generate better ideas.
I think I basically agree (though maybe not with as much high confidence as you), but I think that doesn’t mean that huge amounts of suffering will not dominate the future. For example, if there will be not one but many superintelligent AI systems determining the future, this might create suffering due to cooperation failures.