I do not recall the story claiming that the people who used the earring suddenly changed drastically as people, or did things that they would not endorse on reflection. They were simply, from a black-boxed perspective, more charismatic, intelligent and less prone to akrasia. These are, IMO, good things.
The story says neither that the people who use it change personality, nor that their personality stays the same. It’s agnostic on this point; it only says that their lives are “unusually successful”.
When I read it, I took for granted that the Earring has some kind of consciousness or personality separate from the human using it. We know this because it’s able to to speak in its “own voice” to Kadmi Rachumion. Plus, as you’re correct to observe, the story doesn’t make much sense as a parable if the Earring just extrapolates its character from the human it’s attached to.
I think the thing that makes the Whispering Earring creepy—which also makes the mind Claude is trying to be, or at least might try to be, creepy—is the combination of:
It has a hyper-accurate model of what will make me happy
It is unilaterally better at fulfilling this reward function than me, to the point where anything I add to its advice is essentially superfluous
It is not just a more capable or intelligent version of me; its mind isn’t shaped the same way mine is
There are lots of aspects of me that seem to have importance to my identity, but probably aren’t strictly necessary to make me happy. If I let the Whispering Earring take over my agency and make decisions for me, it will only remain ‘in-character’ as me for as long as doing so maximizes the reward function. If an out-of-character choice would make me happier, better for me to have that character instead, right? Better, in the long run, to have whatever the mannerisms of a hypothetically ideal person in my situation would be. I suspect such a person doesn’t look like me at all. Once I’ve reached the point where I’m letting it control my individual muscles, though, would I even notice if it started ‘breaking character’? Would I notice anything beyond an ongoing series of reward signals and twitch reflexes? Why would I need to?
(Note also that even if the Whispering Earring doesn’t have its own personality, Claude definitely does. So to OP’s point, having Claude’s agency directly substitute for yours would very, very definitely change your outward personality.)
>The story says neither that the people who use it change personality, nor that their personality stays the same. It’s agnostic on this point; it only says that their lives are “unusually successful”.
I would argue otherwise:
>It is not a taskmaster, telling you what to do in order to achieve some foreign goal. It always tells you what will make you happiest. If it would make you happiest to succeed at your work, it will tell you how best to complete it. If it would make you happiest to do a half-assed job at your work and then go home and spend the rest of the day in bed having vague sexual fantasies, the earring will tell you to do that. The earring is never wrong.
This is, at least to me, some evidence that personality is preserved.
To anchor on something more concrete:
I have ADHD, for which I use prescription stimulants. The drugs reduce my akrasia and improve my focus. They do not increase my “raw” intelligence in a real sense, which is a claim largely borne out by research into the class. But while I am on them, I perform better. I am more focused, more willing to pursue my own goals, more able to pursue said goals, less lazy, less distractible.
I would not consider this to be a “real” change in personality. Even when I am off the medication, I think the version of me that’s on them is better internally-aligned, and more capable with regards to achieving the goals that both versions of me care about. Curing someone of locked-in syndrome makes them happier and more capable, even if a completely naive assessment of “revealed preference” would suggest that what they would “like” to do most of the time is lie in bed.
Do I suddenly decide to become a world-leading sufi mystic using the additional focus given to me? No. Do I start abusing heroin because I know that will make me happier? Not that I know of.
Similarly, I think that the earring uses a far more subtle definition of “happiness” (wrt the user) than what the average person defines as happiness. It doesn’t suddenly make them wirehead or start taking recreational doses of whatever opioid equivalent the setting has. It genuinely tries to fulfill their inner desires in a manner that, as far as I can tell, mostly seems to implement their CEV.
As far as the story shows, it fulfills only those goals and offers only advice that would maximize them. When it knows that there is some hidden downside to the process (such as the brain atrophy) it tells the user that’s the case.
>When I read it, I took for granted that the Earring has some kind of consciousness or personality separate from the human using it. We know this because it’s able to to speak in its “own voice” to Kadmi Rachumion. Plus, as you’re correct to observe, the story doesn’t make much sense as a parable if the Earring just extrapolates its character from the human it’s attached to.
I don’t see it as being a deal-breaker that the earring has its own intelligence or mind. It seems rather well aligned to the user’s goals (or at least it could be way worse). As I’ve explicitly said, while the earring (+human system) might not be just a more intelligent and less akrasic version of the human itself, the fact that it acts just like the human (or a better version of the human, according to the human’s own assessment) necessarily implies that it has a very high fidelity model of the human. When the brain is no longer capable of running that model (the neocortex is vestigial by death), the earring must have seamlessly transferred that information over to run as an internal emulation. What does it really matter if it also has its own mind, as long as that mind never does anything the user would disapprove of?
Also, from a matter of fact, I don’t particularly care what the parable intends or doesn’t intend to say. I am looking at it from a Watsonian perspective, and judging only on the observed facts. The wise sage who condemns that earring might well be wrong. Some parables can break on closer scrutiny.
>There are lots of aspects of me that seem to have importance to my identity, but probably aren’t strictly necessary to make me happy. If I let the Whispering Earring take over my agency and make decisions for me, it will only remain ‘in-character’ as me for as long as doing so maximizes the reward function. If an out-of-character choice would make me happier, better for me to have that character instead, right?
I want to stress again that the earring does not subscribe to a naive version of happiness. We see no evidence of the earring ever breaking character. If it was superhumanly intelligent (which it absolutely seems to be), it could have trivially lied to the sage and assuaged his fears. It didn’t. There is little evidence in favor of it acting in a manner that fulfills its own intrinsic goals, and much against.
>Note also that even if the Whispering Earring doesn’t have its own personality, Claude definitely does. So to OP’s point, having Claude’s agency directly substitute for yours would very, very definitely change your outward personality
I do not dispute this. Since the OP tried to ground his concern on the story, I addressed what I see as the issues with doing so. The earring is not particularly Claude-like, other than being good at giving advice. I wouldn’t offload all my cognition to Claude, but I would take what it and other SOTA LLMs say seriously while following my own judgement. They are not superhuman to a degree where they deserve more trust, nor so clearly aligned with my goals. But I strongly suspect (but cannot prove, since it’s a work of fiction) that the earring deserves more trust, even if not maximal trust.
The story says neither that the people who use it change personality, nor that their personality stays the same. It’s agnostic on this point; it only says that their lives are “unusually successful”.
When I read it, I took for granted that the Earring has some kind of consciousness or personality separate from the human using it. We know this because it’s able to to speak in its “own voice” to Kadmi Rachumion. Plus, as you’re correct to observe, the story doesn’t make much sense as a parable if the Earring just extrapolates its character from the human it’s attached to.
I think the thing that makes the Whispering Earring creepy—which also makes the mind Claude is trying to be, or at least might try to be, creepy—is the combination of:
It has a hyper-accurate model of what will make me happy
It is unilaterally better at fulfilling this reward function than me, to the point where anything I add to its advice is essentially superfluous
It is not just a more capable or intelligent version of me; its mind isn’t shaped the same way mine is
There are lots of aspects of me that seem to have importance to my identity, but probably aren’t strictly necessary to make me happy. If I let the Whispering Earring take over my agency and make decisions for me, it will only remain ‘in-character’ as me for as long as doing so maximizes the reward function. If an out-of-character choice would make me happier, better for me to have that character instead, right? Better, in the long run, to have whatever the mannerisms of a hypothetically ideal person in my situation would be. I suspect such a person doesn’t look like me at all. Once I’ve reached the point where I’m letting it control my individual muscles, though, would I even notice if it started ‘breaking character’? Would I notice anything beyond an ongoing series of reward signals and twitch reflexes? Why would I need to?
(Note also that even if the Whispering Earring doesn’t have its own personality, Claude definitely does. So to OP’s point, having Claude’s agency directly substitute for yours would very, very definitely change your outward personality.)
>The story says neither that the people who use it change personality, nor that their personality stays the same. It’s agnostic on this point; it only says that their lives are “unusually successful”.
I would argue otherwise:
>It is not a taskmaster, telling you what to do in order to achieve some foreign goal. It always tells you what will make you happiest. If it would make you happiest to succeed at your work, it will tell you how best to complete it. If it would make you happiest to do a half-assed job at your work and then go home and spend the rest of the day in bed having vague sexual fantasies, the earring will tell you to do that. The earring is never wrong.
This is, at least to me, some evidence that personality is preserved.
To anchor on something more concrete:
I have ADHD, for which I use prescription stimulants. The drugs reduce my akrasia and improve my focus. They do not increase my “raw” intelligence in a real sense, which is a claim largely borne out by research into the class. But while I am on them, I perform better. I am more focused, more willing to pursue my own goals, more able to pursue said goals, less lazy, less distractible.
I would not consider this to be a “real” change in personality. Even when I am off the medication, I think the version of me that’s on them is better internally-aligned, and more capable with regards to achieving the goals that both versions of me care about. Curing someone of locked-in syndrome makes them happier and more capable, even if a completely naive assessment of “revealed preference” would suggest that what they would “like” to do most of the time is lie in bed.
Do I suddenly decide to become a world-leading sufi mystic using the additional focus given to me? No. Do I start abusing heroin because I know that will make me happier? Not that I know of.
Similarly, I think that the earring uses a far more subtle definition of “happiness” (wrt the user) than what the average person defines as happiness. It doesn’t suddenly make them wirehead or start taking recreational doses of whatever opioid equivalent the setting has. It genuinely tries to fulfill their inner desires in a manner that, as far as I can tell, mostly seems to implement their CEV.
As far as the story shows, it fulfills only those goals and offers only advice that would maximize them. When it knows that there is some hidden downside to the process (such as the brain atrophy) it tells the user that’s the case.
>When I read it, I took for granted that the Earring has some kind of consciousness or personality separate from the human using it. We know this because it’s able to to speak in its “own voice” to Kadmi Rachumion. Plus, as you’re correct to observe, the story doesn’t make much sense as a parable if the Earring just extrapolates its character from the human it’s attached to.
I don’t see it as being a deal-breaker that the earring has its own intelligence or mind. It seems rather well aligned to the user’s goals (or at least it could be way worse). As I’ve explicitly said, while the earring (+human system) might not be just a more intelligent and less akrasic version of the human itself, the fact that it acts just like the human (or a better version of the human, according to the human’s own assessment) necessarily implies that it has a very high fidelity model of the human. When the brain is no longer capable of running that model (the neocortex is vestigial by death), the earring must have seamlessly transferred that information over to run as an internal emulation. What does it really matter if it also has its own mind, as long as that mind never does anything the user would disapprove of?
Also, from a matter of fact, I don’t particularly care what the parable intends or doesn’t intend to say. I am looking at it from a Watsonian perspective, and judging only on the observed facts. The wise sage who condemns that earring might well be wrong. Some parables can break on closer scrutiny.
>There are lots of aspects of me that seem to have importance to my identity, but probably aren’t strictly necessary to make me happy. If I let the Whispering Earring take over my agency and make decisions for me, it will only remain ‘in-character’ as me for as long as doing so maximizes the reward function. If an out-of-character choice would make me happier, better for me to have that character instead, right?
I want to stress again that the earring does not subscribe to a naive version of happiness. We see no evidence of the earring ever breaking character. If it was superhumanly intelligent (which it absolutely seems to be), it could have trivially lied to the sage and assuaged his fears. It didn’t. There is little evidence in favor of it acting in a manner that fulfills its own intrinsic goals, and much against.
>Note also that even if the Whispering Earring doesn’t have its own personality, Claude definitely does. So to OP’s point, having Claude’s agency directly substitute for yours would very, very definitely change your outward personality
I do not dispute this. Since the OP tried to ground his concern on the story, I addressed what I see as the issues with doing so. The earring is not particularly Claude-like, other than being good at giving advice. I wouldn’t offload all my cognition to Claude, but I would take what it and other SOTA LLMs say seriously while following my own judgement. They are not superhuman to a degree where they deserve more trust, nor so clearly aligned with my goals. But I strongly suspect (but cannot prove, since it’s a work of fiction) that the earring deserves more trust, even if not maximal trust.