So do my Ritalin pills, in the sense that I usually make better decisions or at least act on them. I do not see the problem here, as long as the “better” decisions are the kind of decision I want to make, or would have made if I didn’t have ADHD.
Your ritalin pills make you a better person both from the point of view of pre-pilled-you and post-pilled you. The “better” decisions are downstream of that. Similarly, there’s many pills I would be happy to take, that would make me a better person from my current-point-of-view (and that I expect my modified-self would endorse). The whispering earring is not one of them : my current me do not want to be replaced by a learn-and-execute-reflexes-machine.
Back to the whispering earring : the internal reasoning traces matter, it’s a big part of what makes me “me”. For example, outside of my observed actions, there is a load-bearing internal world model that I do value intrinsically — remove it, and you remove a good chuck of what people call “consciousness”. For another example, outside of action-memories, I have choice-memories ; memories of actions I considered but decided not to undertake. Those memories are not behaviorally observable, but they do matter to me too, they are also something that I consider part of “me”, and are removed by the whispering-earring.
I simply do not care about those qualia very much, at least not for their own sake
Then that’s the disagreement. I do care about them very much.
An actual AGI/ASI must probably still think about things, instead of magically coming to the correct conclusion out of nowhere
So does Stockfish when playing chess. “Thinking instead of coming to the correct conclusion out of nowhere” is not that much of a constraining requirement, and therefore not much a defining thing of “what is valuable in human experience”.
I am quite literally a man who struggles to be happy. I have clinical depression as well as the ADHD. […]. I am a transhumanist, so I am willing to give up quite a lot more, assuming I come out ahead on the metrics I care about.
Okay, let me be blunt. Chesterton fence in axiology/aesthetics : if you can’t pass the ITT of “what happy humans find valuable”, please do not try to recreate it from first principles. You will miss a lot of things that happy humans indeed do find valuable.
Remember Maslow’s hierarchy. If you ask someone who struggle to meet the basics, “What is self-actualization ? Is this valuable ?”, he will look at you funnily at best, lash out at you at worst. His reaction will be something like “what’s this self-actualization thing worth ? Only food and security matter to me”. This does not prove that self-actualization is worthless, not even that it’s worthless for him, only that it’s worthless for him in his current situation (in a better situation, he would care about it) ; but also : his reaction is perfectly understandable — he is neither obtuse nor stupid. He has his priorities straight.
And this is exactly how I read your “I do not care about those qualia very much”. You’re struggling with happiness, depression, anxiety. I can empathize that to you, getting out of that struggle is supremely important — it must look like the end game, the victory condition. But please do not do the equivalent of throwing “self-actualization” out of the window just because you’re hungry.
>The whispering earring is not one of them : my current me do not want to be replaced by a learn-and-execute-reflexes-machine.
I am not arguing with your preferences, if this is a genuine disagreement on fundamental values, what can we both do but shrug and move on?
I can’t even (and don’t) say that the earring is a destructive mind-uploading machine. How can I? It’s a work of fiction. What I am saying is that the observed behavior is entirely consistent with that depiction, and could potentially conserve qualia and thought if substrate-independence is correct.
(Which I do think is the case, despite not having a solution to the hard problem of consciousness. Nobody does, we just have to deal with it for now.)
>So does Stockfish when playing chess. “Thinking instead of coming to the correct conclusion out of nowhere” is not that much of a constraining requirement, and therefore not much a defining thing of “what is valuable in human experience”.
Does Stockfish have qualia? I don’t know. I am agnostic on that front, albeit less agnostic (in a negative direction) than I would be for LLMs.
It doesn’t matter. I think it is very likely (given substrate independence) that qualia would persist after destructive uploading or in the combined earring system. I do not think that it is possible, even in principle, to mimic something as complex as human cognition with nigh perfect accuracy without qualia arising. In other words, I am very suspicious of arguments in favor of p-zombies.
>And this is exactly how I read your “I do not care about those qualia very much”. You’re struggling with happiness, depression, anxiety. I can empathize that to you, getting out of that struggle is supremely important — it must look like the end game, the victory condition. But please do not do the equivalent of throwing “self-actualization” out of the window just because you’re hungry.
I am, at present, remarkably non-depressed/euthymic after treatment. I have the time and energy and desire to philosophize instead of lying in bed. At the end of the day, this is more of an academic concern, we do not have Whispering Earrings (and LLMs are too imperfect to count). We don’t have any form of mind uploading I care for. What exactly am I losing away or what avenues of reasoning am I discarding with undue haste?
If you think that you have a better grasp on what constitutes eudaimonia for the majority, feel free to share. I would be immensely surprised if you understood me better than I understand myself, my mental illnesses are detrimental, but not to the point of incoherence or insanity.
I would prefer non-invasive mind uploading to the destructive form. I would prefer a system I can audit and that experts understand over the earring. I would prefer senolytic therapy over the earring. I would prefer the earring, as presented in the story, over death.
If you think that you have a better grasp on what constitutes eudaimonia for the majority, feel free to share.
Have you read the Fun Theory sequence ? Because I don’t think I have any insight that isn’t there. Mostly the 3D vs 4D distinction in “High Challenge”.
What I am saying is that the observed behavior is entirely consistent with that depiction, and could potentially conserve qualia and thought if substrate-independence is correct
Two ways to read this.
“If substrate-independence is correct, then qualia and thought can potentially be conserved in the system”. I agree, but I’ll also note that it’s far from being likely (Stockfish playing chess), and the “system” part is important (in the story, it is noted that there is physiological change in the brain).
“If substrate independence is correct (which is not certain) then it is certain that thought and qualia are conserved”, then just no. For the above reasons.
I do not think that it is possible, even in principle, to mimic something as complex as human cognition with nigh perfect accuracy without qualia arising. In other words, I am very suspicious of arguments in favor of p-zombies.
I am, too, very suspicious of p-zombies.
I don’t see how that’s relevant.
Nothing is required to “mimic something as complex as human cognition with nigh perfect accuracy” in the whispering earring story. The only thing that is mimic-ed is a representation of the goals/values of the wielder (and even there, I see nothing that ask for nigh perfect accuracy). Then the cognitive part (creating a plan) can be as inhuman as we want.
Stockfish can play chess better than humans without instantiating an human-player-congnition-engine that would instantiate a human-player-subejctive-experience. Similarly, Whispering Earring can play “How to Win at Life” (or any other goal) without instantiating a human-like-cognition-engine (that would, presumably, trigger human-like-subjective-experience).
I don’t know why you bring upload. The whispering earring is not an upload technology, never been presented as such, and it has never been the point of the thought experiment ? And you seem to agree with that earlier ?
Is your claim something like : “Human levels of intelligence (or greater) needs human-like cognition, and if you add human goals to that, it start to get arbitrarily close to mind-upload” ?
>Have you read the Fun Theory sequence ? Because I don’t think I have any insight that isn’t there. Mostly the 3D vs 4D distinction in “High Challenge”.
I have, even if it’s been a while. I took a quick look at High Challenge:
>To look at it another way, if we’re looking for a suitable long-run meaning of life, we should look for goals that are good to pursue and not just good to satisfy.
Focusing on this quote, I do not believe that I desire a perfectly frictionless life. I enjoy challenges, some of them at least. I do not wish to live a live on autopilot without ever making a real decision. But:
I’ve argued that there’s a strong case for the earring+human system simply shifting the computation/challenge/qualia from being entirely in the human to being mostly in the earring.
As I’ve said elsewhere, I wouldn’t want to wear the earring all the time, at least in theory. I’ve outlined the specific reasons in that comment.
There are many “challenges” in my life that I simply do not care about and wouldn’t miss if they’re gone. Brushing my teeth. Studying for exams. Commuting to work. I would happily let some other entity without qualia do the boring stuff for me, so I can focus on what I enjoy.
I am quite confident that this is in-line with Yudkowsky, even if I haven’t re-read every single entry in the relevant Sequence. Please correct me if I have missed something.
To summarize: I don’t think that the earring is necessarily an autopilot without any degree of consciousness or qualia, and even if it is, I can see very good use cases for it.
>Stockfish can play chess better than humans without instantiating an human-player-congnition-engine that would instantiate a human-player-subejctive-experience. Similarly, Whispering Earring can play “How to Win at Life” (or any other goal) without instantiating a human-like-cognition-engine (that would, presumably, trigger human-like-subjective-experience).
Playing chess well is a very different goal from emulating an arbitrary human with extreme accuracy. Even the best LLMs (which are massively larger and more intensely trained than any chessbot I know of) can’t do it with perfect accuracy, even for writers well-represented in their training data.
From a pragmatic point of view:
A chessbot can be good at chess without mimicking the cognition of a human who is good at chess. It still models a game of chess, and interpretability work suggests this is true for LLMs playing chess (even if they’re bad at it, from memory, GPT-3.5 was actually the best and later models represent a regression)
It is much harder to emulate a human. The only entities that can do so to a decent degree are LLMs, which are surprisingly humanlike. (See Anthropic’s interoperability work, especially the emotion vectors). I think this is at least suggestive in favor of theories claiming that sufficiently intense mimicry of human cognitive output produces internal-processes that are surprisingly close to their human analogues.
>I don’t know why you bring upload. The whispering earring is not an upload technology, never been presented as such, and it has never been the point of the thought experiment ? And you seem to agree with that earlier ?
The story does not plainly state that the earring is an upload machine. But as I’ve argued, I think that is an interpretation that is consistent with textual evidence. I’ve stressed that the machine can mimic the behaviors and desires of the human even after the cognitive architecture responsible (in the human brain) is vestigial. I also argued that this suggests that the earring is offloading computation to itself (and this could be benign, or at least not intentionally malicious). The earring needs that information to pass as the human, even if it’s a “better” version of the human.
In other words: it is suspiciously upload-like, in a manner that invites scrutiny.
Your ritalin pills make you a better person both from the point of view of pre-pilled-you and post-pilled you. The “better” decisions are downstream of that. Similarly, there’s many pills I would be happy to take, that would make me a better person from my current-point-of-view (and that I expect my modified-self would endorse). The whispering earring is not one of them : my current me do not want to be replaced by a learn-and-execute-reflexes-machine.
Back to the whispering earring : the internal reasoning traces matter, it’s a big part of what makes me “me”. For example, outside of my observed actions, there is a load-bearing internal world model that I do value intrinsically — remove it, and you remove a good chuck of what people call “consciousness”. For another example, outside of action-memories, I have choice-memories ; memories of actions I considered but decided not to undertake. Those memories are not behaviorally observable, but they do matter to me too, they are also something that I consider part of “me”, and are removed by the whispering-earring.
Then that’s the disagreement. I do care about them very much.
So does Stockfish when playing chess. “Thinking instead of coming to the correct conclusion out of nowhere” is not that much of a constraining requirement, and therefore not much a defining thing of “what is valuable in human experience”.
Okay, let me be blunt. Chesterton fence in axiology/aesthetics : if you can’t pass the ITT of “what happy humans find valuable”, please do not try to recreate it from first principles. You will miss a lot of things that happy humans indeed do find valuable.
Remember Maslow’s hierarchy. If you ask someone who struggle to meet the basics, “What is self-actualization ? Is this valuable ?”, he will look at you funnily at best, lash out at you at worst. His reaction will be something like “what’s this self-actualization thing worth ? Only food and security matter to me”. This does not prove that self-actualization is worthless, not even that it’s worthless for him, only that it’s worthless for him in his current situation (in a better situation, he would care about it) ; but also : his reaction is perfectly understandable — he is neither obtuse nor stupid. He has his priorities straight.
And this is exactly how I read your “I do not care about those qualia very much”. You’re struggling with happiness, depression, anxiety. I can empathize that to you, getting out of that struggle is supremely important — it must look like the end game, the victory condition. But please do not do the equivalent of throwing “self-actualization” out of the window just because you’re hungry.
>The whispering earring is not one of them : my current me do not want to be replaced by a learn-and-execute-reflexes-machine.
I am not arguing with your preferences, if this is a genuine disagreement on fundamental values, what can we both do but shrug and move on?
I can’t even (and don’t) say that the earring is a destructive mind-uploading machine. How can I? It’s a work of fiction. What I am saying is that the observed behavior is entirely consistent with that depiction, and could potentially conserve qualia and thought if substrate-independence is correct.
(Which I do think is the case, despite not having a solution to the hard problem of consciousness. Nobody does, we just have to deal with it for now.)
>So does Stockfish when playing chess. “Thinking instead of coming to the correct conclusion out of nowhere” is not that much of a constraining requirement, and therefore not much a defining thing of “what is valuable in human experience”.
Does Stockfish have qualia? I don’t know. I am agnostic on that front, albeit less agnostic (in a negative direction) than I would be for LLMs.
It doesn’t matter. I think it is very likely (given substrate independence) that qualia would persist after destructive uploading or in the combined earring system. I do not think that it is possible, even in principle, to mimic something as complex as human cognition with nigh perfect accuracy without qualia arising. In other words, I am very suspicious of arguments in favor of p-zombies.
>And this is exactly how I read your “I do not care about those qualia very much”. You’re struggling with happiness, depression, anxiety. I can empathize that to you, getting out of that struggle is supremely important — it must look like the end game, the victory condition. But please do not do the equivalent of throwing “self-actualization” out of the window just because you’re hungry.
I am, at present, remarkably non-depressed/euthymic after treatment. I have the time and energy and desire to philosophize instead of lying in bed. At the end of the day, this is more of an academic concern, we do not have Whispering Earrings (and LLMs are too imperfect to count). We don’t have any form of mind uploading I care for. What exactly am I losing away or what avenues of reasoning am I discarding with undue haste?
If you think that you have a better grasp on what constitutes eudaimonia for the majority, feel free to share. I would be immensely surprised if you understood me better than I understand myself, my mental illnesses are detrimental, but not to the point of incoherence or insanity.
I would prefer non-invasive mind uploading to the destructive form. I would prefer a system I can audit and that experts understand over the earring. I would prefer senolytic therapy over the earring. I would prefer the earring, as presented in the story, over death.
Have you read the Fun Theory sequence ? Because I don’t think I have any insight that isn’t there. Mostly the 3D vs 4D distinction in “High Challenge”.
Two ways to read this.
“If substrate-independence is correct, then qualia and thought can potentially be conserved in the system”. I agree, but I’ll also note that it’s far from being likely (Stockfish playing chess), and the “system” part is important (in the story, it is noted that there is physiological change in the brain).
“If substrate independence is correct (which is not certain) then it is certain that thought and qualia are conserved”, then just no. For the above reasons.
I am, too, very suspicious of p-zombies.
I don’t see how that’s relevant.
Nothing is required to “mimic something as complex as human cognition with nigh perfect accuracy” in the whispering earring story. The only thing that is mimic-ed is a representation of the goals/values of the wielder (and even there, I see nothing that ask for nigh perfect accuracy). Then the cognitive part (creating a plan) can be as inhuman as we want.
Stockfish can play chess better than humans without instantiating an human-player-congnition-engine that would instantiate a human-player-subejctive-experience. Similarly, Whispering Earring can play “How to Win at Life” (or any other goal) without instantiating a human-like-cognition-engine (that would, presumably, trigger human-like-subjective-experience).
I don’t know why you bring upload. The whispering earring is not an upload technology, never been presented as such, and it has never been the point of the thought experiment ? And you seem to agree with that earlier ?
Is your claim something like : “Human levels of intelligence (or greater) needs human-like cognition, and if you add human goals to that, it start to get arbitrarily close to mind-upload” ?
>Have you read the Fun Theory sequence ? Because I don’t think I have any insight that isn’t there. Mostly the 3D vs 4D distinction in “High Challenge”.
I have, even if it’s been a while. I took a quick look at High Challenge:
>To look at it another way, if we’re looking for a suitable long-run meaning of life, we should look for goals that are good to pursue and not just good to satisfy.
Focusing on this quote, I do not believe that I desire a perfectly frictionless life. I enjoy challenges, some of them at least. I do not wish to live a live on autopilot without ever making a real decision. But:
I’ve argued that there’s a strong case for the earring+human system simply shifting the computation/challenge/qualia from being entirely in the human to being mostly in the earring.
As I’ve said elsewhere, I wouldn’t want to wear the earring all the time, at least in theory. I’ve outlined the specific reasons in that comment.
There are many “challenges” in my life that I simply do not care about and wouldn’t miss if they’re gone. Brushing my teeth. Studying for exams. Commuting to work. I would happily let some other entity without qualia do the boring stuff for me, so I can focus on what I enjoy.
I am quite confident that this is in-line with Yudkowsky, even if I haven’t re-read every single entry in the relevant Sequence. Please correct me if I have missed something.
To summarize: I don’t think that the earring is necessarily an autopilot without any degree of consciousness or qualia, and even if it is, I can see very good use cases for it.
>Stockfish can play chess better than humans without instantiating an human-player-congnition-engine that would instantiate a human-player-subejctive-experience. Similarly, Whispering Earring can play “How to Win at Life” (or any other goal) without instantiating a human-like-cognition-engine (that would, presumably, trigger human-like-subjective-experience).
Playing chess well is a very different goal from emulating an arbitrary human with extreme accuracy. Even the best LLMs (which are massively larger and more intensely trained than any chessbot I know of) can’t do it with perfect accuracy, even for writers well-represented in their training data.
From a pragmatic point of view:
A chessbot can be good at chess without mimicking the cognition of a human who is good at chess. It still models a game of chess, and interpretability work suggests this is true for LLMs playing chess (even if they’re bad at it, from memory, GPT-3.5 was actually the best and later models represent a regression)
It is much harder to emulate a human. The only entities that can do so to a decent degree are LLMs, which are surprisingly humanlike. (See Anthropic’s interoperability work, especially the emotion vectors). I think this is at least suggestive in favor of theories claiming that sufficiently intense mimicry of human cognitive output produces internal-processes that are surprisingly close to their human analogues.
>I don’t know why you bring upload. The whispering earring is not an upload technology, never been presented as such, and it has never been the point of the thought experiment ? And you seem to agree with that earlier ?
The story does not plainly state that the earring is an upload machine. But as I’ve argued, I think that is an interpretation that is consistent with textual evidence. I’ve stressed that the machine can mimic the behaviors and desires of the human even after the cognitive architecture responsible (in the human brain) is vestigial. I also argued that this suggests that the earring is offloading computation to itself (and this could be benign, or at least not intentionally malicious). The earring needs that information to pass as the human, even if it’s a “better” version of the human.
In other words: it is suspiciously upload-like, in a manner that invites scrutiny.