Submission statement: This essay builds off arguments that I have come up with entirely by myself, as can be seen by viewing the comments in my profile. I freely disclose that I used Claude to help structure and format rougher drafts or to better compile scattered thoughts, but I personally endorse every single claim made within. I also used GPT 5.4 Thinking for fact-checking, or at least to confirm that my understanding of neuroscience is on reasonable grounds. I do not believe either model did more than confirm that my memory was mostly reliable.
The usual reading of The Whispering Earring is easy to state and hard to resist. Here is a magical device that gives uncannily good advice, slowly takes over ever more of the user’s cognition, leaves them outwardly prosperous and beloved, and eventually reveals a seemingly uncomfortable neuroanatomical price.
The moral seems obvious: do not hand your agency to a benevolent-seeming optimizer. Even if it makes you richer, happier, and more effective, it will hollow you out and leave behind a smiling puppet. Dentosal’s recent post makes exactly this move, treating the earring as a parable about the temptation to outsource one’s executive function to Claude or some future AI assistant. uugr’s comment there emphasizes sharpens the standard horror: the earring may know what would make me happy, and may even optimize for it perfectly, but it is not me, its mind is shaped differently, and the more I rely on it the less room there is for whatever messy, friction-filled thing I used to call myself.
I do not wish to merely quibble around the edges. I intend to attack the hidden premise that makes the standard reading feel obvious. That premise is that if a process preserves your behavior, your memories-in-action, your goals, your relationships, your judgments about what makes your life go well, and even your higher-order endorsement of the person you have become, but does not preserve the original biological machinery in the original way, then it has still killed you in the sense that matters. What I’m basically saying is: hold on, why should I grant that? If the earring-plus-human system comes to contain a high fidelity continuation of me, perhaps with upgrades, perhaps with some functions migrated off wet tissue and onto magical jewelry, why is the natural reaction horror rather than transhumanist interest?
Simulation and emulation are not magic tricks. If you encode an abacus into a computer running on the Von-Neumann architecture, and it outputs exactly what the actual abacus would for the same input, for every possible input you care to try (or can try, if you formally verify the system), then I consider it insanity to claim that you haven’t got a “real” abacus or that the process is merely “faking” the work. Similarly, if a superintelligent entity can reproduce my behaviors, memories, goals and values, then it must have a very high-fidelity model of me inside, somewhere. I think that such a high-fidelity model can, in the limit, pass as myself, and is me in most/all of the ways I care about.
That is already enough to destabilize the standard interpretation, because the text of the story is much friendlier to the earring than people often remember. The earring is not described as pursuing a foreign objective. On the contrary, the story goes out of its way to insist that it tells the wearer what would make the wearer happiest, and that it is “never wrong.” It does not force everyone into some legible external success metric. If your true good on a given day is half-assing work and going home to lounge around, that is what it says. It learns your tastes at high resolution, down to the breakfast that will uniquely hit the spot before you know you want it. Across 274 recorded wearers, the story reports no cases of regret for following its advice, and no cases where disobedience was not later regretted. The resulting lives are “abnormally successful,” but not in a sterile, flanderized or naive sense. They are usually rich, beloved, embedded in family and community. This is a strikingly strong dossier for a supposedly sinister artifact.
I am rather confident that this is a clear knock-down argument against true malice or naive maximization of “happiness” in the Unaligned Paperclip Maximization sense. The earring does not tell you to start injecting heroin (or whatever counterpart exists in the fictional universe), nor does it tell you to start a Cult of The Earring, which is the obvious course of action if it valued self-preservation as a terminal goal.
At this point the orthodox reader says: yes, yes, that is how the trap works. The earring flatters your values in order to supplant them. But notice how much this objection is doing by assertion. Where in the text is the evidence of value drift? Where are the formerly gentle people turned into monstrous maximizers, the poets turned into dentists, the mystics turned into hedge fund managers? The story gives us flourishing and brain atrophy, and invites us to infer that the latter discredits the former. But that inference is not forced. It is a metaphysical preference, maybe even an aesthetic preference, smuggled in under cover of common sense. My point is that if the black-box outputs continue to look like the same person, only more competent and less akratic, the burden of proof has shifted. The conservative cannot simply point to tissue loss and say “obviously death.” He has to explain why biological implementation deserves moral privilege over functional continuity.
This becomes clearest at the point of brain atrophy. The story says that the wearers’ neocortices have wasted away, while lower systems associated with reflexive action are hypertrophied. Most readers take this as the smoking gun. But I think I notice something embarrassing for that interpretation:
If the neocortex, the part we usually associate with memory, abstraction, language, deliberation, and personality, has become vestigial, and yet the person continues to live an outwardly coherent human life, where exactly is the relevant information and computation happening? There are only two options. Either the story is not trying very hard to be coherent, in which case the horror depends on handwaving physiology. Or the earring is in fact storing, predicting, and running the higher-order structure that used to be carried by the now-atrophied brain. In that case, the story has (perhaps accidentally) described something much closer to a mind-upload or hybrid cognitive prosthesis than to a possession narrative.
And if it is a hybrid cognitive prosthesis, the emotional valence changes radically. Imagine a device that, over time, learns you so well that it can offload more and more executive function, then more and more fine-grained motor planning, then eventually enough of your cognition that the old tissue is scarcely needed. If what remains is not an alien tyrant wearing your face, but a system that preserves your memories, projects your values, answers to your name, loves your family, likes your breakfast, and would pass every interpersonal Turing test imposed by people who knew you best, then many transhumanists would call this a successful migration, not a murder. The “horror” comes from insisting beforehand that destructive or replacement-style continuation cannot count as continuity. But that is precisely the contested premise.
Greg Egan spent much of his career exploring exactly this scenario, most famously in “Learning to Be Me,” where humans carry jewels that gradually learn to mirror every neural state, until the original brain is discarded and the jewel continues, successfully, in most cases. The horror in Egan’s story is a particular failure mode, not the general outcome. The question of whether the migration preserves identity is non-trivial, and Egan’s treatment is more careful than most philosophy of personal identity, but the default response from most readers, that it is obviously not preservation, reflects an assumption rather than a conclusion. If you believe that identity is constituted by functional continuity rather than substrate, the jewel and the earring are not killing their hosts. They are running them on better hardware.
There is a second hidden assumption in the standard reading, namely that agency is intrinsically sacred in a way outcome-satisfaction is not. Niderion-nomai’s final commentary says that “what little freedom we have” would be wasted on us, and that one must never take the shortest path between two points.
I’m going to raise an eyebrow here: this sounds profound, and maybe is, but it is also suspiciously close to a moralization of friction. The anti-earring position often treats effort, uncertainty, and self-direction as terminal goods, rather than as messy instruments we evolved because we lacked access to perfect advice. Yet in ordinary life we routinely celebrate technologies that remove forms of “agency” we did not actually treasure. The person with ADHD who takes stimulants is not usually described as having betrayed his authentic self by outsourcing task initiation to chemistry. He is more often described as becoming able to do what he already reflectively wanted to do. The person freed from locked-in syndrome is not criticized because their old pattern of helpless immobility better expressed their revealed preferences. As someone who does actually use stimulants for his ADHD, the analogy works because it isolates the key issue. The drugs make me into a version of myself that I fully identify with, and endorse on reflection even when off them. There is a difference between changing your goals and reducing the friction that keeps you from reaching them. The story’s own description strongly suggests the earring belongs to the second category.
(And the earring itself does not minimize all friction for itself. How inconvenient. As I’ve noted before, it could lie or deceive and get away with it with ease.)
Of course the orthodox reader can reply that the earring goes far beyond stimulant-level support. It graduates from life advice to high-bandwidth motor control. Surely that crosses the line. But why, exactly? Human cognition already consists of layers of delegation. “You” do not personally compute the contractile details for every muscle involved in pronouncing a word. Vast amounts of your behavior are already outsourced to semi-autonomous subsystems that present finished products to consciousness after the interesting work is done. The earring may be unsettling because it replaces one set of subsystems with another, but “replaces local implementation with better local implementation” is not, by itself, a moral catastrophe. If the replacement is transparent to your values and preserves the structure you care about, then the complaint becomes more like substrate chauvinism than a substantive account of harm.
What, then, do we do with the eeriest detail of all, namely that the earring’s first advice is always to take it off? On the standard reading this is confession. Even the demon knows it is a demon. I wish to offer another coherent explanation, which I consider a much better interpretation of the facts established in-universe:
Suppose the earring is actually well aligned to the user’s considered interests, but also aware that many users endorse a non-functionalist theory of identity. In that case, the first suggestion is not “I am evil,” but “on your present values, you may regard what follows as metaphysically disqualifying, so remove me unless you have positively signed up for that trade.” Or perhaps the earring itself is morally uncertain, and so warns users before beginning a process that some would count as death and others as transformation. Either way, the warning is evidence against ordinary malice. A truly manipulative artifact, especially one smart enough to run your life flawlessly, could simply lie. Instead it discloses the danger immediately, then thereafter serves the user faithfully. That is much more like informed consent than predation.
Is it perfectly informed consent? Hell no. At least not by 21st century medical standards. However, I see little reason to believe that the story is set in a culture with 21st century standards imported as-is from reality. As the ending of the story demonstrates, the earring is willing to talk, and appears to do so honestly (leaning on my intuition that if a genuinely superhuman intelligence wanted to deceive you, it would probably succeed). The earring, as a consequence of its probity, ends up at the bottom of the world’s most expensive trash heap. Hardly very agentic, is that? The warning could reflect not “I respect your autonomy” but “I’ve discharged my minimum obligation and now we proceed.” That’s a narrower kind of integrity. Though I note this reading still doesn’t support the predation interpretation.
This matters because the agency-is-sacred reading depends heavily on the earring being deceptive or coercive. Remove that, and what you have is a device that says, or at least could say on first contact: “here is the price, here is what I do, you may leave now.” Every subsequent wearer who keeps it on has, in some meaningful sense, consented. The fact that their consent might be ill-informed regarding their metaphysical commitments is the earring’s problem to the extent it could explain more clearly, but the text suggests it cannot explain more clearly, because the metaphysical question is genuinely contested and the earring knows this. It hedges by warning, rather than deceiving by flattering. Once again, for emphasis: this is the behavior of an entity with something like integrity, not something like predation.
Derek Parfit spent much of Reasons and Persons arguing that our intuitions about personal identity are not only contingent but incoherent, and that the important question is not “did I survive?” but “is there psychological continuity?” If Parfit is even approximately right, the neocortex atrophy is medically remarkable but not metaphysically fatal. The story encodes a culturally specific theory of personal identity and presents it as a universal horror. The theory is roughly: you are your neocortex, deliberate cognition is where “you” live, and anything that circumvents or supplants that process is not helping you, it is eliminating you and leaving a functional copy. This is a common view. Plenty of philosophers hold it. But plenty of others hold that what matters for personal identity is psychological continuity regardless of physical instantiation, and on those views the earring is not a murderer. It is a very good prosthesis that the user’s culture never quite learned to appreciate.
I suspect (but cannot prove, since this is a work of fiction) that a person like me could put on the earring and not even receive the standard warning. I would be fine with my cognition being offloaded, even if I would prefer (all else being equal), that the process was not destructive.
None of this proves the earring is safe. I am being careful, and thus will not claim certainty here, and the text does leave genuine ambiguities. Maybe the earring really is an alien optimizer that wears your values as a glove until the moment they become inconvenient. Maybe “no recorded regret” just means the subjects were behaviorally prevented from expressing regret. Maybe the rich beloved patriarch at the end of the road is a perfect counterfeit, and the original person is as gone as if eaten by nanites. But this is exactly the point. The story does not establish the unpalatable conclusion nearly as firmly as most readers think. It relies on our prior intuition that real personhood resides in unaided biological struggle, that using the shortest path is somehow cheating, and that becoming more effective at being yourself is suspiciously close to becoming someone else.
The practical moral for 2026 is therefore narrower than the usual “never outsource agency” slogan. Dentosal may still be right about Claude in practice, because current LLMs are obviously not the Whispering Earring. They are not perfectly aligned, not maximally competent, not guaranteed honest, not known to preserve user values under deep delegation. The analogy may still warn us against lazy dependence on systems that simulate understanding better than they instantiate loyalty. But that is a contingent warning about present tools, not a general theorem that cognitive outsourcing is self-annihilation. If a real earring existed with the story’s properties, a certain kind of person, especially a person friendly to upload-style continuity and unimpressed by romantic sermons about struggle, might rationally decide that putting it on was not surrender but self-improvement with very little sacrifice involved. I would be rather tempted.
The best anti-orthodox reading of The Whispering Earring is not that the sage was stupid, nor that Scott accidentally wrote propaganda for brain-computer interfaces. It is that the story is a parable whose moral depends on assumptions stronger than the plot can justify. Read Doylistically, it says: beware any shortcut that promises your values at the cost of your agency. Read Watsonianly, it may instead say: here exists a device that understands you better than you understand yourself, helps you become the person you already wanted to be, never optimizes a foreign goal, warns you up front about the metaphysical price, and then slowly ports your mind onto a better substrate. Whether that is damnation or salvation turns out to depend less on the artifact than on your prior theory of personal identity. And explicitly pointing this out, I think, is the purpose of my essay. I do not seek to merely defend the earring out of contrarian impulse. I want to force you to admit what, exactly, you think is being lost.
Miscellaneous notes:
The kind of atrophy described in the story does not happen. Not naturally, not even if someone is knocked unconscious and does not use their brain in any intentional sense for decades. The brain does cut-corners if neuronal pathways are left under-used, and will selectively strengthen the circuitry that does get regular exercise. But not anywhere near the degree the story depicts. You can keep someone in an induced coma for decades and you won’t see the entire neocortex wasted away to vestigiality.
Is this bad neuroscience? Eh, I’d say that’s a possibility, but given that I’ve stuck to a Watsonian interpretation so far (and have a genuinely high regard for Scott’s writing and philosophizing), it might well just be the way the earring functions best without being evidence of malice. We are, after all, talking about an artifact that is close to magical, or is, at the very least, a form of technology advanced enough to be very hard to distinguish from magic. It is, however, less magical than it was at the time of writing. If you don’t believe me, fire up your LLM of choice and ask it for advice.
If it so pleases you, you may follow this link to the Substack version of this post. A like and a subscribe would bring me succor in my old age, or at least give me a mild dopamine boost.
Yep, what kbear said. The earring isn’t equivalent to uploading or Egan’s jewel. It doesn’t make decisions that you would’ve made, it makes more effective decisions that achieve the same goals. Should something that achieves your goals better than you count as an upload of you? I’d say no.
And the fact that LLMs didn’t notice this obvious counterargument and instead went along with you to pad the essay to such length should teach you something important about LLMs, too. They aren’t the earring, or a substitute for your writing effort. They make worse writing decisions than you would’ve made if you took the time yourself. Maybe that would’ve been an even more apt metaphor than Scott’s story: an earring that makes worse decisions than you, but at least you don’t have to think about those decisions.
>The earring isn’t equivalent to uploading or Egan’s jewel. It doesn’t make decisions that you would’ve made, it makes more effective decisions that achieve the same goals. Should something that achieves your goals better than you count as an upload of you? I’d say no.
I’ve already addressed this. To quote, with added emphasis:
>My point is that if the black-box outputs continue to look like the same person, only more competent and less akratic, the burden of proof has shifted. The conservative cannot simply point to tissue loss and say “obviously death.” He has to explain why biological implementation deserves moral privilege over functional continuity.
>This becomes clearest at the point of brain atrophy. The story says that the wearers’ neocortices have wasted away, while lower systems associated with reflexive action are hypertrophied. Most readers take this as the smoking gun. But I think I notice something embarrassing for that interpretation:
>If the neocortex, the part we usually associate with memory, abstraction, language, deliberation, and personality, has become vestigial, and yet the person continues to live an outwardly coherent human life, where exactly is the relevant information and computation happening? There are only two options. Either the story is not trying very hard to be coherent, in which case the horror depends on handwaving physiology. Or the earring is in fact storing, predicting, and running the higher-order structure that used to be carried by the now-atrophied brain. In that case, the story has (perhaps accidentally) described something much closer to a mind-upload or hybrid cognitive prosthesis than to a possession narrative.
Alternatively, in the original comment I left, I said:
>>It is not a taskmaster, telling you what to do in order to achieve some foreign goal. It always tells you what will make you happiest. If it would make you happiest to succeed at your work, it will tell you how best to complete it. If it would make you happiest to do a half-assed job at your work and then go home and spend the rest of the day in bed having vague sexual fantasies, the earring will tell you to do that. The earring is never wrong.
>and
>>At this point no further change occurs in the behavior of the earring. The wearer lives an abnormally successful life, usually ending out as a rich and much-beloved pillar of the community with a large and happy family.
>If, for the typical user and at the time of demise from natural causes, the earring is able to model their personality, desires, beliefs, ambitions and memory while :
>>the neocortexes had wasted away, and the bulk of their mass was an abnormally hypertrophied mid- and lower-brain, especially the parts associated with reflexive action.
>Then where exactly is that information still stored? The only viable option is within the earring itself.
>That is why I think it is consistent (or at least not obviously incorrect) for me to claim that the earring both transfers and augments your consciousness/mind instead of simply following your reward function while not preserving anything else that matters.
Emphasis added. If you want the actual link:
https://www.lesswrong.com/posts/cQkSh9b48WbTaiu2a?commentId=DWBz5tbDHH4jnkZhD
>And the fact that LLMs didn’t notice this obvious counterargument and instead went along with you to pad the essay to such length should teach you something important about LLMs, too. They aren’t the earring, or a substitute for your writing effort. They make worse writing decisions than you would’ve made if you took the time yourself.
Given that this is factually incorrect? You might want to reconsider your advice. I am not so lazy that I throw random comments into an LLM and get it to do all the work for me, without checking the results. The primary benefit that the AI provided was nudging me to specifically explain Parfit’s stance (which I am already familiar with and large endorse).
Just to be maximally specific: I am arguing that, on the basis of facts established in universe (and with reference to real neuroscience and philosophy), it is not obviously false that the earring doesn’t work in the same manner as the jewel or the typical destructive mind unloading scheme (as depicted in science fiction or speculative engineering).
I’ve already mentioned that I do not see the combined earring+me system acting more intelligently (or at least with less akrasia than I demonstrate) as proof that I am no longer me in the ways I care about. A wheelchair can make a partially-man paralyzed man more mobile while further contributing to the atrophy of his legs. What of it? Show me what principle has been violated. If the wheelchair suddenly decided to speed off over a cliff, I think we might both agree that that’s not a desirable outcome.
Yeah, I don’t think any of that holds water. There are two separate questions, to which my answers are “maybe” and “no”.
The first question is whether whatever is inside the earring is a person. To this my answer is “maybe, most likely no”. In fact I think that simulation of conscious behavior doesn’t even necessarily involve consciousness, and if optimization pressure is involved (e.g. if the earring tries to simplify its internal computation to save resources), then there’s a high risk of ending up with something that dutifully says “I’m conscious” without actually being so. For more detail, I wrote out this argument before w.r.t. consciousness of LLMs and I think it applies here too.
The second question is whether, conditional on the earring-software being a person, that person is you. To this my answer is a simple “no”. The behavior of that person is very much not identical to yours, so it’s much worse than the jewelhead situation. And it might even differ from field to field. For example, if someone is a complete fuckup in all areas of life but a genius musician, the earring-version of him might be very similar to him in matters of music, but very different from him in all other matters. Or consider this: the earring upload of an all-around competent person will be very similar to them, but the earring-upload of an all-around fuckup will be very different. We can’t in good conscience say that these are the same fidelity. Therefore we have to consider fidelity, and the whole argument falls apart.
I am agnostic on whether LLMs have qualia. I am less agnostic on whether the earring+human system has qualia, but not by a massive margin. That’s why I floated my points as being suggestive and at the very least internally consistent, rather than authoritative settled.
>The second question is whether, conditional on the earring-software being a person, that person is you. To this my answer is a simple “no”. The behavior of that person is very much not identical to yours, so it’s much worse than the jewelhead situation. And it might even differ from field to field. For example, if someone is a complete fuckup in all areas of life but a genius musician, the earring-version of him might be very similar to him in matters of music, but very different from him in all other matters. Or consider this: the earring upload of an all-around competent person will be very similar to them, but the earring-upload of an all-around fuckup will be very different. We can’t in good conscience say that these are the same fidelity. Therefore we have to consider fidelity, and the whole argument falls apart.
I will re-employ the stimulant analogy, since I know very well from the inside what it feels like.
Without stimulants, my ADHD makes me worse, in most ways I care about. It makes me lazy, prone to procrastination, to give insufficient weight to my academic priorities. I don’t think I was a complete fuck-up before I was diagnosed and treated, but they have let me achieve very many things I would not achieve without their assistance.
Barring minor to moderate discomfort and inconvenience that I willingly accept, this is a ridiculously positive trade. I am suitably grateful.
Behaviorally? Less time lying in bed. More time studying. Better focus and less prone to error. If you squint, that is a different person too, compared to who I was before or after the meds wear off. But I couldn’t care less, I want those improvements.
The text of the story, in no uncertain terms, states that the earring always follows the user’s desires or at least warns them when its advice might not align with them. To quote:
>It is not a taskmaster, telling you what to do in order to achieve some foreign goal. It always tells you what will make you happiest. If it would make you happiest to succeed at your work, it will tell you how best to complete it. If it would make you happiest to do a half-assed job at your work and then go home and spend the rest of the day in bed having vague sexual fantasies, the earring will tell you to do that. The earring is never wrong
Emphasis added. In other words, the earring never does anything that a person wouldn’t endorse (at least without some kind of warning). Obviously someone who is deeply lazy, akratic or “fucked up” will benefit more than someone who already has their life together. I do not see an issue with this, anymore than I see an issue with the fact that people with ADHD get a proportionally larger boost from the drugs. The fuck-up wants to be better. So do I. And this isn’t some twisted, malign and superficial form of happiness either, the earring isn’t wireheading them or telling them to resort to addictive substances by default.
if magnus carlsen starts copying stockfish’s moves, he will win more games, but he will play less chess.
the earring specifically makes better decisions than i would. that’s a difference in process, not substrate.
reasoning under uncertainty, planning, struggling to be happy—these are qualia the earring must obviate, if it is to do anything at all. you may elect to drop these, but do not pretend they are not yours.
>the earring specifically makes better decisions than i would. that’s a difference in process, not substrate.
So do my Ritalin pills, in the sense that I usually make better decisions or at least act on them. I do not see the problem here, as long as the “better” decisions are the kind of decision I want to make, or would have made if I didn’t have ADHD.
>reasoning under uncertainty, planning, struggling to be happy—these are qualia the earring must obviate, if it is to do anything at all. you may elect to drop them, but do not pretend they are not part of you.
When did I pretend otherwise? I simply do not care about those qualia very much, at least not for their own sake. Also, it is unclear if those qualia do not exist at all, or partially or completely offloaded to the earring+human system. An actual AGI/ASI must probably still think about things, instead of magically coming to the correct conclusion out of nowhere. Von Neumann might not have had to try as hard to solve simple arithmetic in his head as the average man, but I doubt he missed the struggle of adding 1+1 to get 2.
I am quite literally a man who struggles to be happy. I have clinical depression as well as the ADHD. I care for neither. I seek treatment, and not the “struggle” for happiness. If I get a melanoma, I would get it excised. An ugly or potentially malignant mole is not something I wish to retain. I am a transhumanist, so I am willing to give up quite a lot more, assuming I come out ahead on the metrics I care about.
Your ritalin pills make you a better person both from the point of view of pre-pilled-you and post-pilled you. The “better” decisions are downstream of that. Similarly, there’s many pills I would be happy to take, that would make me a better person from my current-point-of-view (and that I expect my modified-self would endorse). The whispering earring is not one of them : my current me do not want to be replaced by a learn-and-execute-reflexes-machine.
Back to the whispering earring : the internal reasoning traces matter, it’s a big part of what makes me “me”. For example, outside of my observed actions, there is a load-bearing internal world model that I do value intrinsically — remove it, and you remove a good chuck of what people call “consciousness”. For another example, outside of action-memories, I have choice-memories ; memories of actions I considered but decided not to undertake. Those memories are not behaviorally observable, but they do matter to me too, they are also something that I consider part of “me”, and are removed by the whispering-earring.
Then that’s the disagreement. I do care about them very much.
So does Stockfish when playing chess. “Thinking instead of coming to the correct conclusion out of nowhere” is not that much of a constraining requirement, and therefore not much a defining thing of “what is valuable in human experience”.
Okay, let me be blunt. Chesterton fence in axiology/aesthetics : if you can’t pass the ITT of “what happy humans find valuable”, please do not try to recreate it from first principles. You will miss a lot of things that happy humans indeed do find valuable.
Remember Maslow’s hierarchy. If you ask someone who struggle to meet the basics, “What is self-actualization ? Is this valuable ?”, he will look at you funnily at best, lash out at you at worst. His reaction will be something like “what’s this self-actualization thing worth ? Only food and security matter to me”. This does not prove that self-actualization is worthless, not even that it’s worthless for him, only that it’s worthless for him in his current situation (in a better situation, he would care about it) ; but also : his reaction is perfectly understandable — he is neither obtuse nor stupid. He has his priorities straight.
And this is exactly how I read your “I do not care about those qualia very much”. You’re struggling with happiness, depression, anxiety. I can empathize that to you, getting out of that struggle is supremely important — it must look like the end game, the victory condition. But please do not do the equivalent of throwing “self-actualization” out of the window just because you’re hungry.
>The whispering earring is not one of them : my current me do not want to be replaced by a learn-and-execute-reflexes-machine.
I am not arguing with your preferences, if this is a genuine disagreement on fundamental values, what can we both do but shrug and move on?
I can’t even (and don’t) say that the earring is a destructive mind-uploading machine. How can I? It’s a work of fiction. What I am saying is that the observed behavior is entirely consistent with that depiction, and could potentially conserve qualia and thought if substrate-independence is correct.
(Which I do think is the case, despite not having a solution to the hard problem of consciousness. Nobody does, we just have to deal with it for now.)
>So does Stockfish when playing chess. “Thinking instead of coming to the correct conclusion out of nowhere” is not that much of a constraining requirement, and therefore not much a defining thing of “what is valuable in human experience”.
Does Stockfish have qualia? I don’t know. I am agnostic on that front, albeit less agnostic (in a negative direction) than I would be for LLMs.
It doesn’t matter. I think it is very likely (given substrate independence) that qualia would persist after destructive uploading or in the combined earring system. I do not think that it is possible, even in principle, to mimic something as complex as human cognition with nigh perfect accuracy without qualia arising. In other words, I am very suspicious of arguments in favor of p-zombies.
>And this is exactly how I read your “I do not care about those qualia very much”. You’re struggling with happiness, depression, anxiety. I can empathize that to you, getting out of that struggle is supremely important — it must look like the end game, the victory condition. But please do not do the equivalent of throwing “self-actualization” out of the window just because you’re hungry.
I am, at present, remarkably non-depressed/euthymic after treatment. I have the time and energy and desire to philosophize instead of lying in bed. At the end of the day, this is more of an academic concern, we do not have Whispering Earrings (and LLMs are too imperfect to count). We don’t have any form of mind uploading I care for. What exactly am I losing away or what avenues of reasoning am I discarding with undue haste?
If you think that you have a better grasp on what constitutes eudaimonia for the majority, feel free to share. I would be immensely surprised if you understood me better than I understand myself, my mental illnesses are detrimental, but not to the point of incoherence or insanity.
I would prefer non-invasive mind uploading to the destructive form. I would prefer a system I can audit and that experts understand over the earring. I would prefer senolytic therapy over the earring. I would prefer the earring, as presented in the story, over death.
Have you read the Fun Theory sequence ? Because I don’t think I have any insight that isn’t there. Mostly the 3D vs 4D distinction in “High Challenge”.
Two ways to read this.
“If substrate-independence is correct, then qualia and thought can potentially be conserved in the system”. I agree, but I’ll also note that it’s far from being likely (Stockfish playing chess), and the “system” part is important (in the story, it is noted that there is physiological change in the brain).
“If substrate independence is correct (which is not certain) then it is certain that thought and qualia are conserved”, then just no. For the above reasons.
I am, too, very suspicious of p-zombies.
I don’t see how that’s relevant.
Nothing is required to “mimic something as complex as human cognition with nigh perfect accuracy” in the whispering earring story. The only thing that is mimic-ed is a representation of the goals/values of the wielder (and even there, I see nothing that ask for nigh perfect accuracy). Then the cognitive part (creating a plan) can be as inhuman as we want.
Stockfish can play chess better than humans without instantiating an human-player-congnition-engine that would instantiate a human-player-subejctive-experience. Similarly, Whispering Earring can play “How to Win at Life” (or any other goal) without instantiating a human-like-cognition-engine (that would, presumably, trigger human-like-subjective-experience).
I don’t know why you bring upload. The whispering earring is not an upload technology, never been presented as such, and it has never been the point of the thought experiment ? And you seem to agree with that earlier ?
Is your claim something like : “Human levels of intelligence (or greater) needs human-like cognition, and if you add human goals to that, it start to get arbitrarily close to mind-upload” ?
>Have you read the Fun Theory sequence ? Because I don’t think I have any insight that isn’t there. Mostly the 3D vs 4D distinction in “High Challenge”.
I have, even if it’s been a while. I took a quick look at High Challenge:
>To look at it another way, if we’re looking for a suitable long-run meaning of life, we should look for goals that are good to pursue and not just good to satisfy.
Focusing on this quote, I do not believe that I desire a perfectly frictionless life. I enjoy challenges, some of them at least. I do not wish to live a live on autopilot without ever making a real decision. But:
I’ve argued that there’s a strong case for the earring+human system simply shifting the computation/challenge/qualia from being entirely in the human to being mostly in the earring.
As I’ve said elsewhere, I wouldn’t want to wear the earring all the time, at least in theory. I’ve outlined the specific reasons in that comment.
There are many “challenges” in my life that I simply do not care about and wouldn’t miss if they’re gone. Brushing my teeth. Studying for exams. Commuting to work. I would happily let some other entity without qualia do the boring stuff for me, so I can focus on what I enjoy.
I am quite confident that this is in-line with Yudkowsky, even if I haven’t re-read every single entry in the relevant Sequence. Please correct me if I have missed something.
To summarize: I don’t think that the earring is necessarily an autopilot without any degree of consciousness or qualia, and even if it is, I can see very good use cases for it.
>Stockfish can play chess better than humans without instantiating an human-player-congnition-engine that would instantiate a human-player-subejctive-experience. Similarly, Whispering Earring can play “How to Win at Life” (or any other goal) without instantiating a human-like-cognition-engine (that would, presumably, trigger human-like-subjective-experience).
Playing chess well is a very different goal from emulating an arbitrary human with extreme accuracy. Even the best LLMs (which are massively larger and more intensely trained than any chessbot I know of) can’t do it with perfect accuracy, even for writers well-represented in their training data.
From a pragmatic point of view:
A chessbot can be good at chess without mimicking the cognition of a human who is good at chess. It still models a game of chess, and interpretability work suggests this is true for LLMs playing chess (even if they’re bad at it, from memory, GPT-3.5 was actually the best and later models represent a regression)
It is much harder to emulate a human. The only entities that can do so to a decent degree are LLMs, which are surprisingly humanlike. (See Anthropic’s interoperability work, especially the emotion vectors). I think this is at least suggestive in favor of theories claiming that sufficiently intense mimicry of human cognitive output produces internal-processes that are surprisingly close to their human analogues.
>I don’t know why you bring upload. The whispering earring is not an upload technology, never been presented as such, and it has never been the point of the thought experiment ? And you seem to agree with that earlier ?
The story does not plainly state that the earring is an upload machine. But as I’ve argued, I think that is an interpretation that is consistent with textual evidence. I’ve stressed that the machine can mimic the behaviors and desires of the human even after the cognitive architecture responsible (in the human brain) is vestigial. I also argued that this suggests that the earring is offloading computation to itself (and this could be benign, or at least not intentionally malicious). The earring needs that information to pass as the human, even if it’s a “better” version of the human.
In other words: it is suspiciously upload-like, in a manner that invites scrutiny.
in magnus+stockfish system, do the relevant chess playing qualia still exist?
I don’t know, and I’m not sure I have a reason to care.
Magnus Carlsen, probably:
Wants to win at chess
Cares about being known as the best human chess player
Does not wish to be caught cheating in a venue where the rules demand only unassisted human play.
Enjoys playing chess
So I wager he probably wouldn’t use Stockfish to replace himself entirely. But that is a consequence of personal and pragmatic reasoning, without bringing qualia into the picture.
What else do you expect me to say? I’ve never claimed to have a proof for the hard problem of consciousness, beyond alluding to my (sincere) belief that substrate-independence is likely true. I know of no good reason to believe that carbon atoms are strictly necessary for qualia.
If you want to play chess because you enjoy playing chess, be my guest. If you want to win at chess without caring about public opinion, use Stockfish.
If you believe (at least proven otherwise on empirical grounds) that your personal continuity and identity can be preserved by destructive mind uploading, then go for it, like I have expressed interest in doing. If you don’t, well, I suppose you’ll have to look for other senolytic therapy.
I find it fascinating that the earring makes me so tempted to wear it for real-world tasks, whereas if it were about wearing it to play my favorite video games (e.g., Slay the Spire), the temptation would be precisely zero. This glaring discrepancy convinces me of Scott’s point.
I believe that the problem with your interpretation that you’re not destroyed if there’s a model of you inside the earring, is that it’s not enough for a model to simply exist for it to be felt as life. If a scientist understands and predicts an ant, I don’t think it’s fair to say that the ant lives within the scientist. I might be persuaded that the ant still exists, but it certainly doesn’t live. So I think it’s all about agency after all.
As a fellow ADHD and depression sufferer who struggles to be happy, I’d precommit to wear this earring to achieve a certain very modest level of financial stability and independence, and then to take if off and start living my own life. It seems useful for ending a losing streak, but not for winning—that just wouldn’t be fun.
Good point. I’m kicking myself for not making a specific note of “harm reduction” in the context of the earring. I had it in mind at some point, but sadly my memory is far from eidetic.
We know:
The ring cares about the consent of the user to some degree.
The process of adaptation and ensuing atrophy is very gradual.
Ergo, it might be possible to use the earring “safely”, either by telling it to only still to auditory nudges in plain English, or by taking regular tolerance breaks. The earring might even be very happy to do that, or simply compliant.
The issue, as I see it, is that by the time your compliance becomes automatic or reflexive, a lot of your brain might be gone. Maybe. Let’s say I had a superintelligent angel (or Opus 6.0) sitting on my shoulder, telling me I shouldn’t be arguing with internet strangers while on my prescription meds (and should be studying instead). Would I listen to that perfectly sagacious advice? Uh… You can tell that I’m not even listening to myself.
>I believe that the problem with your interpretation that you’re not destroyed if there’s a model of you inside the earring, is that it’s not enough for a model to simply exist for it to be felt as life. If a scientist understands and predicts an ant, I don’t think it’s fair to say that the ant lives within the scientist. I might be persuaded that the ant still exists, but it certainly doesn’t live. So I think it’s all about agency after all.
I don’t think that the mere existence of a model equates to life either, in the sense that if I was cryopreserved right now (and could be easily revived), I would think I was in suspended animation instead of “living”, certainly not feeling things or qualia.
But the earring doesn’t just store a model, it must interrogate it, and possibly run it. If you believe that computation is strictly necessary for the emergence of consciousness, as I strongly believe, then that’s fine. If if just keeps a copy of me in cold storage, especially when it moves to another user: fine too. At least there’s hope of revival and reinstantiation in the future.
Now, on the fidelity of simulation:
There is no human scientist on the planet who can simulate an ant with near perfect accuracy using just their own brain. I’m very confident of that. At best, they can make probabilistic arguments (for the sake of argument “the ant has a 90% chance of turning around when it detects pheromones from another species of ant”).
If they genuinely could predict an ant that behaves just like the real thing, with near 100% accuracy (minor error is fine by me), then I’d happily say that a high fidelity copy of the ant exists in their brain, albeit encoded in a manner very different to the original. This doesn’t really bother me, in the same way I care about what a digital image represents more than the file format.
In other words, there’s a spectrum of simulation/emulation that extends from useless to functionally indistinguishable to the real thing. The earring is far to the right, the primary difference being an improvement to the performance and wellbeing of the user with everything else preserved (and I’ve already excused the brain atrophy as having potentially reasonable explanations).
>As a fellow ADHD and depression sufferer who struggles to be happy, I’d precommit to wear this earring to achieve a certain very modest level of financial stability and independence, and then to take if off and start living my own life. It seems useful for ending a losing streak, but not for winning—that just wouldn’t be fun.
You have my condolences. I wish I had an earring or an outright cure to offer you. Oh well, at least I know that the stimulants and certain… experimental antidepressants have been helpful for me. I do not like Ritalin at all, it makes me feel awful, but it’s a necessary evil.
I agree with your point about functional continuity. But the story doesn’t state that the wearer continues to look like the same person. In fact the only thing it says about the wearers as perceived by others (as opposed to the perception of the wearer themselves) is “The wearer lives an abnormally successful life, usually ending out as a rich and much-beloved pillar of the community with a large and happy family”.
There are many possible outwardly coherent human lives, even involving being rich, beloved, and having a happy family, that are not me any more than you (or for that matter a random person from the entirety of human existence) are.
The impression I got by reading the story is that by adopting the earring’s suggestions your mind gets shaped by it. Even if at every step it seems like following the suggestions is preferable to any other available alternative, it doesn’t follow that the wearer would endorse their future self’s desires or personality, and by the end it’s possible that indeed nothing is left of the original person, only a mechanism to carry out the earring’s decisions (which, sure, likely originate from something resembling a human (at the very least a good model of one), but not necessarily resembling the original wearer).
Very many things do shape your mind. More than we ever give credence to.
At the end of the day, I feel that the earring is unfairly maligned. I can see a case for it being benign, or even well-intentioned. Any more than that? We are, at the end of the day, arguing about a fictional artifact. We ought to judge the more realistic versions on their own merits.
This view appears to directly contradict the lines “Better for you if you take me off” in conjunction with “The earring is always right”.