Thanks! Seems like the upshot is that humans have a complex soup of desires around learning from and epistemically interacting with AIs, and you’re proposing to use the word “legitimate” to describe that soup, which I agree is a helpful way of thinking about it (and probably better than “not-manipulating”), but also not a True Name in the John Wentworth sense.
The proposal in the last paragraph seems unhelpful for me-in-particular, because I’m working on how to install goals in brain-like AGIs, and I have ideas that seem promising but only work for a limited number of goals (they kinda have to be simple, concrete, “atomic”, and/or directly related to people’s feelings, and/or have a ground truth that can be calculated explicitly, more-or-less). The last paragraph seems to entail making an AGI that wants humans to come quickly to the conclusions that they would otherwise have reached eventually on their own. But that’s pretty complex and abstract, and not something I have any good idea of how to install as a goal / motivation, alas.
No comment about whether it’s a viable approach in a more LLM-like paradigm.
Thanks! Seems like the upshot is that humans have a complex soup of desires around learning from and epistemically interacting with AIs, and you’re proposing to use the word “legitimate” to describe that soup, which I agree is a helpful way of thinking about it (and probably better than “not-manipulating”), but also not a True Name in the John Wentworth sense.
The proposal in the last paragraph seems unhelpful for me-in-particular, because I’m working on how to install goals in brain-like AGIs, and I have ideas that seem promising but only work for a limited number of goals (they kinda have to be simple, concrete, “atomic”, and/or directly related to people’s feelings, and/or have a ground truth that can be calculated explicitly, more-or-less). The last paragraph seems to entail making an AGI that wants humans to come quickly to the conclusions that they would otherwise have reached eventually on their own. But that’s pretty complex and abstract, and not something I have any good idea of how to install as a goal / motivation, alas.
No comment about whether it’s a viable approach in a more LLM-like paradigm.