Can you imagine what it would be like to try to convey to pre-music folk even that music is real and that it might be worth learning how to listen to it?
This analogy is capturing my current understanding of this post and its various comments pretty well: Looking is like music, in that it is a difficult, voluntaryish act of observing and manipulating hidden mental states. This will result in wireheading, among other things, but it might be sometimes useful. (Note that music is also wireheading, but it can still be useful in narrow contexts.)
FWIW, I’ve come to think that wireheading is an anti-concept as applied to humans. It’s one of those “presume the conclusion” type mental movements. In practice it seems to act like a back door for arguments based on belief residue like the Protestant work ethic / “pleasure is sinful” stuff.
(A little more concretely: It makes sense to talk about some system engaging in wireheading only when there’s a goal imposed from outside the system. It’s like glorified Goodharting. But if the goals come from within the system, it stops being clear what “wireheading” means. On the inside it might feel like “Oh, I just found a vastly easier way to get what I want — and what I want wasn’t what I thought I wanted!” Without an external evaluation criterion, that actually just becomes correct.)
With that said, I think I intuit what you mean by calling music and Looking “wireheading”. I don’t mean to dismiss that. Stuff like, if you meditate enough to get Great Insights™ such that you don’t bother to eat food anymore and you die, that seems like a pretty dramatic failure and kind of throws those “insights” into question.
An interesting fact came to my mind; music that affects one’s mental state is forbidden in Islam.
Perhaps the reason this theme of “sinful pleasure” keeps repeating is the observation that pleasure is a reward signal that does not quite match the utility functions of the conscious mind. At least, that has always been the key motivator of this idea subspace to me.
This analogy is capturing my current understanding of this post and its various comments pretty well: Looking is like music, in that it is a difficult, voluntaryish act of observing and manipulating hidden mental states. This will result in wireheading, among other things, but it might be sometimes useful. (Note that music is also wireheading, but it can still be useful in narrow contexts.)
Cool.
FWIW, I’ve come to think that wireheading is an anti-concept as applied to humans. It’s one of those “presume the conclusion” type mental movements. In practice it seems to act like a back door for arguments based on belief residue like the Protestant work ethic / “pleasure is sinful” stuff.
(A little more concretely: It makes sense to talk about some system engaging in wireheading only when there’s a goal imposed from outside the system. It’s like glorified Goodharting. But if the goals come from within the system, it stops being clear what “wireheading” means. On the inside it might feel like “Oh, I just found a vastly easier way to get what I want — and what I want wasn’t what I thought I wanted!” Without an external evaluation criterion, that actually just becomes correct.)
With that said, I think I intuit what you mean by calling music and Looking “wireheading”. I don’t mean to dismiss that. Stuff like, if you meditate enough to get Great Insights™ such that you don’t bother to eat food anymore and you die, that seems like a pretty dramatic failure and kind of throws those “insights” into question.
An interesting fact came to my mind; music that affects one’s mental state is forbidden in Islam.
Perhaps the reason this theme of “sinful pleasure” keeps repeating is the observation that pleasure is a reward signal that does not quite match the utility functions of the conscious mind. At least, that has always been the key motivator of this idea subspace to me.