Well, if we as humans place weight on open individualism (OI) being true, then the question of whether an ASI would shield us from S-risks or not should lose importance to us relative to the question of whether the ASI would serve the greater good, even at the expense of humanity. So if an OI ASI came to pose an S-risk, then perhaps we should trust that such an ASI’s decision to doom humanity serves consciousness better than our species-preserving biases would have it.
A belief in OI goes both ways, by which what I mean to emphasize is not that this belief’s effect on an ASI’s probability of incurring S-risks is ambiguous due to OI providing the ASI reasons bothfor and against harming humanity, but rather that a belief in OI should lead both us and ASI to care less about our own respective fates.
Well, if we as humans place weight on open individualism (OI) being true, then the question of whether an ASI would shield us from S-risks or not should lose importance to us relative to the question of whether the ASI would serve the greater good, even at the expense of humanity. So if an OI ASI came to pose an S-risk, then perhaps we should trust that such an ASI’s decision to doom humanity serves consciousness better than our species-preserving biases would have it.
A belief in OI goes both ways, by which what I mean to emphasize is not that this belief’s effect on an ASI’s probability of incurring S-risks is ambiguous due to OI providing the ASI reasons both for and against harming humanity, but rather that a belief in OI should lead both us and ASI to care less about our own respective fates.