but crucially we actually want future AGIs to have human values!
SIAI doesn’t, at least. When making an AI to self-improve to superintelligence, why make it so that it gets horny?
Exactly human values, not analogous to human values. So if humans value having sex but don’t value FAI having sex then the FAI will value humans having sex but not value having sex itself.
Exactly human values, not analogous to human values. So if humans value having sex but don’t value FAI having sex then the FAI will value humans having sex but not value having sex itself.