I do think this is a pretty good point about how human value formation tends to happen.
I think something sort-of-similar might happen to happen a little, nearterm, with LLM-descended AI. But, AI just doesn’t have any of the same social machinery actually embedded in it the same way, so if it’s doing something similar, it’d be happening because LLMs vaguely ape human tendencies. (And I expect this to stop being a major factor as the AI gets smarter. I don’t expect it to install the sort of social drives itself that humans have, and “imitate humans” has pretty severe limits of how smart you can get, so if we get to AI much smarter than that, it’ll probably be doing a different thing)
I think the more important here is “notice that you’re (probably) wrong about about how you actually do your value-updating, and this may be warping your expectations about how AI would do it.”
But, that doesn’t leave me with any particular other idea than the current typical bottom-up story.
(obviously if we did something more like uploads, or upload-adjacent, it’d be a whole different story)
I do think this is a pretty good point about how human value formation tends to happen.
I think something sort-of-similar might happen to happen a little, nearterm, with LLM-descended AI. But, AI just doesn’t have any of the same social machinery actually embedded in it the same way, so if it’s doing something similar, it’d be happening because LLMs vaguely ape human tendencies. (And I expect this to stop being a major factor as the AI gets smarter. I don’t expect it to install the sort of social drives itself that humans have, and “imitate humans” has pretty severe limits of how smart you can get, so if we get to AI much smarter than that, it’ll probably be doing a different thing)
I think the more important here is “notice that you’re (probably) wrong about about how you actually do your value-updating, and this may be warping your expectations about how AI would do it.”
But, that doesn’t leave me with any particular other idea than the current typical bottom-up story.
(obviously if we did something more like uploads, or upload-adjacent, it’d be a whole different story)