If you think that you have a better grasp on what constitutes eudaimonia for the majority, feel free to share.
Have you read the Fun Theory sequence ? Because I don’t think I have any insight that isn’t there. Mostly the 3D vs 4D distinction in “High Challenge”.
What I am saying is that the observed behavior is entirely consistent with that depiction, and could potentially conserve qualia and thought if substrate-independence is correct
Two ways to read this.
“If substrate-independence is correct, then qualia and thought can potentially be conserved in the system”. I agree, but I’ll also note that it’s far from being likely (Stockfish playing chess), and the “system” part is important (in the story, it is noted that there is physiological change in the brain).
“If substrate independence is correct (which is not certain) then it is certain that thought and qualia are conserved”, then just no. For the above reasons.
I do not think that it is possible, even in principle, to mimic something as complex as human cognition with nigh perfect accuracy without qualia arising. In other words, I am very suspicious of arguments in favor of p-zombies.
I am, too, very suspicious of p-zombies.
I don’t see how that’s relevant.
Nothing is required to “mimic something as complex as human cognition with nigh perfect accuracy” in the whispering earring story. The only thing that is mimic-ed is a representation of the goals/values of the wielder (and even there, I see nothing that ask for nigh perfect accuracy). Then the cognitive part (creating a plan) can be as inhuman as we want.
Stockfish can play chess better than humans without instantiating an human-player-congnition-engine that would instantiate a human-player-subejctive-experience. Similarly, Whispering Earring can play “How to Win at Life” (or any other goal) without instantiating a human-like-cognition-engine (that would, presumably, trigger human-like-subjective-experience).
I don’t know why you bring upload. The whispering earring is not an upload technology, never been presented as such, and it has never been the point of the thought experiment ? And you seem to agree with that earlier ?
Is your claim something like : “Human levels of intelligence (or greater) needs human-like cognition, and if you add human goals to that, it start to get arbitrarily close to mind-upload” ?
>Have you read the Fun Theory sequence ? Because I don’t think I have any insight that isn’t there. Mostly the 3D vs 4D distinction in “High Challenge”.
I have, even if it’s been a while. I took a quick look at High Challenge:
>To look at it another way, if we’re looking for a suitable long-run meaning of life, we should look for goals that are good to pursue and not just good to satisfy.
Focusing on this quote, I do not believe that I desire a perfectly frictionless life. I enjoy challenges, some of them at least. I do not wish to live a live on autopilot without ever making a real decision. But:
I’ve argued that there’s a strong case for the earring+human system simply shifting the computation/challenge/qualia from being entirely in the human to being mostly in the earring.
As I’ve said elsewhere, I wouldn’t want to wear the earring all the time, at least in theory. I’ve outlined the specific reasons in that comment.
There are many “challenges” in my life that I simply do not care about and wouldn’t miss if they’re gone. Brushing my teeth. Studying for exams. Commuting to work. I would happily let some other entity without qualia do the boring stuff for me, so I can focus on what I enjoy.
I am quite confident that this is in-line with Yudkowsky, even if I haven’t re-read every single entry in the relevant Sequence. Please correct me if I have missed something.
To summarize: I don’t think that the earring is necessarily an autopilot without any degree of consciousness or qualia, and even if it is, I can see very good use cases for it.
>Stockfish can play chess better than humans without instantiating an human-player-congnition-engine that would instantiate a human-player-subejctive-experience. Similarly, Whispering Earring can play “How to Win at Life” (or any other goal) without instantiating a human-like-cognition-engine (that would, presumably, trigger human-like-subjective-experience).
Playing chess well is a very different goal from emulating an arbitrary human with extreme accuracy. Even the best LLMs (which are massively larger and more intensely trained than any chessbot I know of) can’t do it with perfect accuracy, even for writers well-represented in their training data.
From a pragmatic point of view:
A chessbot can be good at chess without mimicking the cognition of a human who is good at chess. It still models a game of chess, and interpretability work suggests this is true for LLMs playing chess (even if they’re bad at it, from memory, GPT-3.5 was actually the best and later models represent a regression)
It is much harder to emulate a human. The only entities that can do so to a decent degree are LLMs, which are surprisingly humanlike. (See Anthropic’s interoperability work, especially the emotion vectors). I think this is at least suggestive in favor of theories claiming that sufficiently intense mimicry of human cognitive output produces internal-processes that are surprisingly close to their human analogues.
>I don’t know why you bring upload. The whispering earring is not an upload technology, never been presented as such, and it has never been the point of the thought experiment ? And you seem to agree with that earlier ?
The story does not plainly state that the earring is an upload machine. But as I’ve argued, I think that is an interpretation that is consistent with textual evidence. I’ve stressed that the machine can mimic the behaviors and desires of the human even after the cognitive architecture responsible (in the human brain) is vestigial. I also argued that this suggests that the earring is offloading computation to itself (and this could be benign, or at least not intentionally malicious). The earring needs that information to pass as the human, even if it’s a “better” version of the human.
In other words: it is suspiciously upload-like, in a manner that invites scrutiny.
Have you read the Fun Theory sequence ? Because I don’t think I have any insight that isn’t there. Mostly the 3D vs 4D distinction in “High Challenge”.
Two ways to read this.
“If substrate-independence is correct, then qualia and thought can potentially be conserved in the system”. I agree, but I’ll also note that it’s far from being likely (Stockfish playing chess), and the “system” part is important (in the story, it is noted that there is physiological change in the brain).
“If substrate independence is correct (which is not certain) then it is certain that thought and qualia are conserved”, then just no. For the above reasons.
I am, too, very suspicious of p-zombies.
I don’t see how that’s relevant.
Nothing is required to “mimic something as complex as human cognition with nigh perfect accuracy” in the whispering earring story. The only thing that is mimic-ed is a representation of the goals/values of the wielder (and even there, I see nothing that ask for nigh perfect accuracy). Then the cognitive part (creating a plan) can be as inhuman as we want.
Stockfish can play chess better than humans without instantiating an human-player-congnition-engine that would instantiate a human-player-subejctive-experience. Similarly, Whispering Earring can play “How to Win at Life” (or any other goal) without instantiating a human-like-cognition-engine (that would, presumably, trigger human-like-subjective-experience).
I don’t know why you bring upload. The whispering earring is not an upload technology, never been presented as such, and it has never been the point of the thought experiment ? And you seem to agree with that earlier ?
Is your claim something like : “Human levels of intelligence (or greater) needs human-like cognition, and if you add human goals to that, it start to get arbitrarily close to mind-upload” ?
>Have you read the Fun Theory sequence ? Because I don’t think I have any insight that isn’t there. Mostly the 3D vs 4D distinction in “High Challenge”.
I have, even if it’s been a while. I took a quick look at High Challenge:
>To look at it another way, if we’re looking for a suitable long-run meaning of life, we should look for goals that are good to pursue and not just good to satisfy.
Focusing on this quote, I do not believe that I desire a perfectly frictionless life. I enjoy challenges, some of them at least. I do not wish to live a live on autopilot without ever making a real decision. But:
I’ve argued that there’s a strong case for the earring+human system simply shifting the computation/challenge/qualia from being entirely in the human to being mostly in the earring.
As I’ve said elsewhere, I wouldn’t want to wear the earring all the time, at least in theory. I’ve outlined the specific reasons in that comment.
There are many “challenges” in my life that I simply do not care about and wouldn’t miss if they’re gone. Brushing my teeth. Studying for exams. Commuting to work. I would happily let some other entity without qualia do the boring stuff for me, so I can focus on what I enjoy.
I am quite confident that this is in-line with Yudkowsky, even if I haven’t re-read every single entry in the relevant Sequence. Please correct me if I have missed something.
To summarize: I don’t think that the earring is necessarily an autopilot without any degree of consciousness or qualia, and even if it is, I can see very good use cases for it.
>Stockfish can play chess better than humans without instantiating an human-player-congnition-engine that would instantiate a human-player-subejctive-experience. Similarly, Whispering Earring can play “How to Win at Life” (or any other goal) without instantiating a human-like-cognition-engine (that would, presumably, trigger human-like-subjective-experience).
Playing chess well is a very different goal from emulating an arbitrary human with extreme accuracy. Even the best LLMs (which are massively larger and more intensely trained than any chessbot I know of) can’t do it with perfect accuracy, even for writers well-represented in their training data.
From a pragmatic point of view:
A chessbot can be good at chess without mimicking the cognition of a human who is good at chess. It still models a game of chess, and interpretability work suggests this is true for LLMs playing chess (even if they’re bad at it, from memory, GPT-3.5 was actually the best and later models represent a regression)
It is much harder to emulate a human. The only entities that can do so to a decent degree are LLMs, which are surprisingly humanlike. (See Anthropic’s interoperability work, especially the emotion vectors). I think this is at least suggestive in favor of theories claiming that sufficiently intense mimicry of human cognitive output produces internal-processes that are surprisingly close to their human analogues.
>I don’t know why you bring upload. The whispering earring is not an upload technology, never been presented as such, and it has never been the point of the thought experiment ? And you seem to agree with that earlier ?
The story does not plainly state that the earring is an upload machine. But as I’ve argued, I think that is an interpretation that is consistent with textual evidence. I’ve stressed that the machine can mimic the behaviors and desires of the human even after the cognitive architecture responsible (in the human brain) is vestigial. I also argued that this suggests that the earring is offloading computation to itself (and this could be benign, or at least not intentionally malicious). The earring needs that information to pass as the human, even if it’s a “better” version of the human.
In other words: it is suspiciously upload-like, in a manner that invites scrutiny.