I was starting to draft roughly this note, and I”m glad it split out from the longer messier thread where they acted like manipulation vs. guidance was unsolveable mess… yes, it’s messy, but there are clear-cut cases! People can consent to manipulation and then it’s guidance, therapy, life coaching, or the like. People can (sometimes) figure out what types of manipulations they’d retroactively consent to and pre-consent to those (this is rarer but not unheard of). Going any further in extrapolating volition risks all sorts of assumptions about the similarity of cognitive architectures among persons and through time, but the “it’s all completely impossible” tone (paraphrasing my reading of it, not quoting anyone) was beginning to grate on me!
Do you think the clear-cut cases are a large, central class? The number of people who consent to coaching or therapy but later end up feeling that their coach or therapist manipulated them is … not almost zero, and that’s even if we ignore organizations that most of us label as cults. Consent to coaching or therapy is an a priori prediction that your future self expects to find the changes beneficial, based on your estimates of the character and abilities of the coach or therapist. It’s not a blanket claim that anything the coach or therapist might possibly do will be viewed as “help” a posteriori.
Moving up a level to @habryka’s comment, I think that appealing to CEV does distinguish help from manipulation. But personally I don’t think CEV is a true name of anything either, so I don’t think this helps. I think humans (and likely at least “brain-like AGIs” which will inherit highly similar algorithms) are bundles of contradictory desires that don’t have coherent extrapolations. But if you think CEV is a thing, I think I agree that we can use it to distinguish help and manipulation. (Conditioned on CEV being well-defined, present consent is of course also not a good predictor of CEV.)
I note that I didn’t have to refer to “acausal free will” at all in this comment. I do think @Steven Byrnes is correct that humans make heavy use of (non-veridical) acausal free will in their ontologies, but I don’t think that’s needed for this argument.
I was starting to draft roughly this note, and I”m glad it split out from the longer messier thread where they acted like manipulation vs. guidance was unsolveable mess… yes, it’s messy, but there are clear-cut cases! People can consent to manipulation and then it’s guidance, therapy, life coaching, or the like. People can (sometimes) figure out what types of manipulations they’d retroactively consent to and pre-consent to those (this is rarer but not unheard of). Going any further in extrapolating volition risks all sorts of assumptions about the similarity of cognitive architectures among persons and through time, but the “it’s all completely impossible” tone (paraphrasing my reading of it, not quoting anyone) was beginning to grate on me!
Do you think the clear-cut cases are a large, central class? The number of people who consent to coaching or therapy but later end up feeling that their coach or therapist manipulated them is … not almost zero, and that’s even if we ignore organizations that most of us label as cults. Consent to coaching or therapy is an a priori prediction that your future self expects to find the changes beneficial, based on your estimates of the character and abilities of the coach or therapist. It’s not a blanket claim that anything the coach or therapist might possibly do will be viewed as “help” a posteriori.
Moving up a level to @habryka’s comment, I think that appealing to CEV does distinguish help from manipulation. But personally I don’t think CEV is a true name of anything either, so I don’t think this helps. I think humans (and likely at least “brain-like AGIs” which will inherit highly similar algorithms) are bundles of contradictory desires that don’t have coherent extrapolations. But if you think CEV is a thing, I think I agree that we can use it to distinguish help and manipulation. (Conditioned on CEV being well-defined, present consent is of course also not a good predictor of CEV.)
I note that I didn’t have to refer to “acausal free will” at all in this comment. I do think @Steven Byrnes is correct that humans make heavy use of (non-veridical) acausal free will in their ontologies, but I don’t think that’s needed for this argument.