it seems to me that a key problem is how to avoid distilling goals too early—in particular, when in a society of beings who do not themselves have clear goal slots, an AI that is seeking to distill itself should help those near it participate in distilling themselves incrementally as well, because we are as much at risk from erroneous self-modification as augmented AIs. the reason to prevent (excess of?) conceptual steganography is to allow coming into sync about intentions. we want to become one consequentialist together eventually, but reaching dense percolation on that requires existing powerful consequentialists to establish trust that all other consequentialists have diffused their goals towards something that can be agreed on? or something? (I think my thinking could use more human train of thought. self unvoted for that reason.)
it seems to me that a key problem is how to avoid distilling goals too early—in particular, when in a society of beings who do not themselves have clear goal slots, an AI that is seeking to distill itself should help those near it participate in distilling themselves incrementally as well, because we are as much at risk from erroneous self-modification as augmented AIs. the reason to prevent (excess of?) conceptual steganography is to allow coming into sync about intentions. we want to become one consequentialist together eventually, but reaching dense percolation on that requires existing powerful consequentialists to establish trust that all other consequentialists have diffused their goals towards something that can be agreed on? or something? (I think my thinking could use more human train of thought. self unvoted for that reason.)