Thanks for the post! What follows is a bit of a rant.
I’m a bit torn as to how much we should care about AI sentience initially. On one hand, ignoring sentience could lead us to do some really bad things to AIs. On the other hand, if we take sentience seriously, we might want to avoid a lot of techniques, like boxing, scalable oversight, and online training. In a recent talk, Buck compared humanity controlling AI systems to dictators controlling their population.
One path we might take as a civilization is that we initially align our AI systems in an immoral way (using boxing, scalable oversight, etc) and then use these AIs to develop techniques to align AI systems in a moral way. Although this wouldn’t be ideal, it might still be better than creating a sentient squiggle maximizer and letting it tile the universe.
There are also difficult moral questions here, like if you create a sentient AI system with different preferences than yours, is it okay to turn it off?
Thanks for the post! What follows is a bit of a rant.
I’m a bit torn as to how much we should care about AI sentience initially. On one hand, ignoring sentience could lead us to do some really bad things to AIs. On the other hand, if we take sentience seriously, we might want to avoid a lot of techniques, like boxing, scalable oversight, and online training. In a recent talk, Buck compared humanity controlling AI systems to dictators controlling their population.
One path we might take as a civilization is that we initially align our AI systems in an immoral way (using boxing, scalable oversight, etc) and then use these AIs to develop techniques to align AI systems in a moral way. Although this wouldn’t be ideal, it might still be better than creating a sentient squiggle maximizer and letting it tile the universe.
There are also difficult moral questions here, like if you create a sentient AI system with different preferences than yours, is it okay to turn it off?