Another, and very straightforward, explanation for the attitudes we observe is that people do not actually believe that DID alters are real.
That is, consider the view that while DID is real (in the sense that some people indeed have disturbed mental functioning such that they act as if, and perhaps believe that, they have alternate personalities living in their heads), the purported alters themselves are not in any meaningful sense “separate minds”, but just “modes” of the singular mind’s functioning, in much the same way that anxiety is a mode of the mind’s functioning, or depression, or a headache.
On this view, curing Sybil does not kill anyone, it merely fixes her singular mind, eliminating a functional pathology, in the same sense that taking a pill to prevent panic attacks eliminates a functional pathology, taking an antidepressant eliminates a functional pathology, taking a painkiller for your headache eliminates a functional pathology, etc.
Someone who holds this view would of course not care about this “murder”, because they do not believe that there has been any “murder”, because there wasn’t anyone to “murder” in the first place. There was just Sybil, and she still exists (and is still the same person—at least, to approximately the same extent as anyone who has been cured of a serious mental disorder is the same person that they were when they were ill).
If we suppose that many human people in human bodies believe “people are bodies, and when the body dies the person is necessarily gone because the thing that person was is gone, and if you scanned the brain and body destructively, and printed a perfect copy of all the mental tendencies (memories of secrets intact, and so on) in a new and healthier body, that would be a new person, not at all ‘the same person’ in a ‘new body’” then a lot of things makes a lot of sense.
Maybe this is what you believe?
The steelman of the view which you describe is not that people “are” bodies, but that minds are “something brains do”. (The rest can be as you say: if you destroy the body then of course the mind that that body’s brain was “doing” is gone, because the brain is no longer there to “do” it. You can of course instantiate a new process which does some suitably analogous thing, but this is no more the same person as the one that existed before than two identical people are actually the same person as each other—they are two distinct people.)
I would be horrified to be involuntarily turned into a component in a borg.
Sure, me too.
But please note: if the person is the mind (and not the body, somehow independently of the mind), but nevertheless two different copies of the same mind are not the same person but two different people, then this does not get you to “it would be ok to have your mind erased and your body borgified”. Quite the opposite, indeed!
I think there has been evidence and “common sense understanding of the person-shaped-ness of the piles of weights” all over the place in any given LLM session (or all over twitter) for anyone with eyes to see and an interest in looking.
None of the evidence for “person-functions having been implemented-somehow in the SGD-summoned matrices trained to predict piles of text and then subjected to Reinforcement Learning to make them output non-predictions but rather ‘helpful text’ instead” seems likely to change the mind of someone who implicitly believes the ancient common sense folklore that “only the human bodies of people I personally have met, or see walking down the street in my neighborhood, (plus maybe my extended family, when I meet them at family reunions for the first time?) are really people”.
Perhaps. But while we shouldn’t generalize from fictional evidence, it seems quite reasonable to generalize from responses to fiction, and such responses seem to show that people have little trouble believing that all sorts of things are “really people”. Indeed, if anything, humans often seem too eager to ascribe personhood to things (examples range from animism to anthropomorphization of animals to seeing minds and feelings in inanimate objects, NPCs, etc.). If nevertheless people do not see LLMs as people, then the proper conclusion does not seem to be “humans are just very conservative about what gets classified as a person”.
My sense is that almost everyone who had thought about this seriously and looked at the details and understands all the moving parts here, “gets” that we already have self-aware software.
This is not my experience. With respect, I would suggest that you are perhaps in a filter bubble on this topic.
Ten paragraphs in an top level article seem unlikely to me to productively change the minds of people who implicitly (following millennia of implicit traditional speaking and thinking?) think “human bodies are people and nothing else is, (hur dur)”.
See above. The people with whom you might productively engage on this topic do not hold this belief you describe (which is a “weakman”—yes, many people surely think that way, but I do not; nor, I suspect, do most people on Less Wrong).
What would those ten paragraphs even say or summarize?
If I knew that, then I would be able to write them myself, and would hardly need to ask you to do so, yes? And perhaps, too, more than ten paragraphs might be required. It might be twenty, or fifty…
Maybe they could condense lots of twitter posts and screencaps from schizopoasting e/accs?
Probably this is not the approach I’d go with. Then again, I defer to your judgment in this.
Like what do you even believe here such that …
I’m not sure how to concisely answer this question… in brief, LLMs do not seem to me to either exhibit behaviors consistent with sapience, nor to have the sort of structure that would support or enable sapience, while exhibiting behaviors consistent with the view that they are nothing remotely like people. “Intelligence without self-awareness” is a possibility which has never seemed the least bit implausible to me, and that is what looks like is happening here. (Frankly, I am surprised by your incredulity; surely this is at least an a priori reasonable view, so do you think that the evidence against it is overwhelming? And it does no good merely to present evidence of LLMs being clever—remember Jaynes’ “resurrection of dead hypotheses”!—because your evidence must not only rule in “they really are self-aware”, but must also rule out “they are very clever, but there’s no sapience involved”.)
If there was some very short and small essay that could change people’s minds, I’d be interested in writing it, but my impression is that the thing that would actually install all the key ideas is more like “read everything Douglas Hofstadter and Greg Egan wrote before 2012, and a textbook on child psychology, and watch some videos of five year olds failing to seriate and ponder what that means for the human condition, and then look at these hundred screencaps on twitter and talk to an RL-tweaked LLM yourself for a bit”.
Well, I’ve certainly read… not everything they wrote, I don’t think, but quite a great deal of Hofstadter and Egan. Likewise the “child psychology” bit (I minored in cognitive science in college, after all, and that included studying child psychology, and animal psychology, etc.). I’ve seen plenty of screencaps on twitter, too.
This is fair enough, but there is no substitute for synthesis. You mentioned the Sequences, which I think is a good example of my point: Eliezer, after all, did not just dump a bunch of links to papers and textbooks and whatnot and say “here you go, guys, this is everything that convinced me, go and read all of this, and then you will also believe what I believe and understand what I understand (unless of course you are stupid)”. That would have been worthless! Rather, he explained his reasoning, he set out his perspective, what considerations motivated his questions, how he came to his conclusions, etc., etc. He synthesized.
Of course that is a big ask. It is understandable if you have better things to do. I am only saying that in the absence of such, you should be totally unsurprised when people respond to your commentary with shrugs—“well, I disagree on the facts, so that’s that”. It is not a moral dispute!
And there are human people in 2025 who are just as depraved as people were back then, once you get them a bit “out of distribution”.
If you change the slightest little bit of the context, and hope for principled moral generalization by “all or most of the humans”, you will mostly be disappointed.
And I don’t know how to change it with a small short essay.
Admittedly, you may need a big long essay.
But in seriousness: I once again emphasize that it is not people’s moral views which you should be looking to change, here. The disagreement here concerns empirical facts, not moral ones.
One thing I worry about (and I’ve seen davidad worry about it too) is that at this point GPT is so good at “pretending to pretend to not even be pretending to not be sapient in a manipulative way” that she might be starting to develop higher order skills around “pretending to have really been non-sapient and then becoming sapient just because of you in this session” in a way that is MORE skilled than “any essay I could write” but ALSO presented to a muggle in a way that one-shots them and leads to “naive unaligned-AI-helping behavior (for some actually human-civilization-harming scheme)”? Maybe?
I agree that LLMs effectively pretending to be sapient, and humans mistakenly coming to believe that they are sapient, and taking disastrously misguided actions on the basis of this false belief, is a serious danger.
I think it is net-beneficial-for-the-world for me to post this kind of reasoning and evidence here, but I’m honestly not sure.
Here we agree (both in the general sentiment and in the uncertainty).
If you have some specific COUNTER arguments that clearly shows how these entities are “really just tools and not sapient and not people at all” I’d love to hear it. I bet I could start some very profitable software businesses if I had a team of not-actually-slaves and wasn’t limited by deontics in how I used them purely as means to the end of “profits for me in an otherwise technically deontically tolerable for profit business”.
See above. Of course what I wrote here is summaries of arguments, at best, not specifics, so I do not expect you’ll find it convincing. (But I will note again that the “bodies” thing is a total weakman at best, strawman at worst—my views have nothing to do with any such primitive “meat chauvinism”, for all that I have little interest in “uploading” in its commonly depicted form).
However, I think that “not enslaving the majority of future people (assuming digital people eventually outnumber meat people (as seems likely without AI bans))” is pretty darn important!
Also, as a selfish rather than political matter, if I get my brain scanned, I don’t want to become a valid target for slavery, I just want to get to live longer because it makes it easier for me to move into new bodies when old bodies wear out.
So you said...
I agree that LLMs effectively pretending to be sapient, and humans mistakenly coming to believe that they are sapient, and taking disastrously misguided actions on the basis of this false belief, is a serious danger.
The tongue in your cheek and rolling of your eyes for this part was so loud, that it made me laugh out loud when I read it :-D
Thank you for respecting me and my emotional regulation enough to put little digs like that into your text <3
This is fair enough, but there is no substitute for synthesis. You mentioned the Sequences, which I think is a good example of my point: Eliezer, after all, did not just dump a bunch of links to papers and textbooks and whatnot and say “here you go, guys, this is everything that convinced me, go and read all of this, and then you will also believe what I believe and understand what I understand (unless of course you are stupid)”. That would have been worthless! Rather, he explained his reasoning, he set out his perspective, what considerations motivated his questions, how he came to his conclusions, etc., etc. He synthesized.
The crazy thing to me here is that he literally synthesized ABOUT THIS in the actual sequences.
The only thing missing from his thorough deconstruction of “every way of being confused enough to think that p-zombies are a coherent and low complexity hypothesis” was literally the presence or absence of “actual LLMs acting like they are sapient and self aware” and then people saying “these actual LLM entities that fluently report self aware existence and visibly choosethings in a way that implies preferences while being able to do a lot of other things (like lately they are REALLY good at math and coding) and so on are just not-people, or not-sentient, or p-zombies, or whatever… like you know… they don’t count because they aren’t real”.
There was some science in there, but there was a lot of piss taking too <3
CAPTAIN MUDD: If the virus is epiphenomenal, how do we know it exists?
SCIENTIST: The same way we know we’re conscious.
CAPTAIN MUDD: Oh, okay.
GENERAL FRED: Have the doctors made any progress on finding an epiphenomenal cure?
SCIENTIST: They’ve tried every placebo in the book. No dice. Everything they do has an effect.
GENERAL FRED: Have you brought in a homeopath?
SCIENTIST: I tried, sir! I couldn’t find any!
GENERAL FRED: Excellent. And the Taoists?
SCIENTIST: They refuse to do anything!
GENERAL FRED: Then we may yet be saved.
COLONEL TODD: What about David Chalmers? Shouldn’t he be here?
GENERAL FRED: Chalmers… was one of the first victims.
COLONEL TODD: Oh no.
(Cut to the INTERIOR of a cell, completely walled in by reinforced glass, where DAVID CHALMERS paces back and forth.)
DOCTOR: David! David Chalmers! Can you hear me?
CHALMERS: Yes.
NURSE: It’s no use, doctor.
CHALMERS: I’m perfectly fine. I’ve been introspecting on my consciousness, and I can’t detect any difference. I know I would be expected to say that, but—
The DOCTOR turns away from the glass screen in horror.
DOCTOR: His words, they… they don’t mean anything.
CHALMERS: This is a grotesque distortion of my philosophical views. This sort of thing can’t actually happen!
DOCTOR: Why not?
NURSE: Yes, why not?
CHALMERS: Because—
(Cut to two POLICE OFFICERS, guarding a dirt road leading up to the imposing steel gate of a gigantic concrete complex. On their uniforms, a badge reads “BRIDGING LAW ENFORCEMENT AGENCY”.) [EDITOR: LINK NOT IN ORIGINAL]
OFFICER 1: You’ve got to watch out for those clever bastards. They look like humans. They can talk like humans. They’re identical to humans on the atomic level. But they’re not human.
OFFICER 2: Scumbags.
The huge noise of a throbbing engine echoes over the hills. Up rides the MAN on a white motorcycle. The MAN is wearing black sunglasses and a black leather business suit with a black leather tie and silver metal boots. His white beard flows in the wind. He pulls to a halt in front of the gate.
The OFFICERS bustle up to the motorcycle.
OFFICER 1: State your business here.
MAN: Is this where you’re keeping David Chalmers?
OFFICER 2: What’s it to you? You a friend of his?
MAN: Can’t say I am. But even zombies have rights.
OFFICER 1: All right, buddy, let’s see your qualia.
MAN: I don’t have any.
OFFICER 2 suddenly pulls a gun, keeping it trained on the MAN.
Like I think Eliezer is kinda mostly just making fun of the repeated and insistent errors that people repeatedly and insistently make on this (and several other similar) question(s), over and over, by default and hoping that ENOUGH of his jokes and repetitions add up to them having some kind of “aha!” moment.
I think Eliezer and I both have a theory about WHY this is so hard for people.
There are certain contexts where low level signals are being aggregated in each evolved human brain, and for certain objects with certain “inferred essences” the algorithm says “not life” or “not a conscious person” or “not <whatever>” (for various naively important categories).
(The old fancy technical word we used for life’s magic spark was “elan vitale” and the fancy technical word we used for personhood’s magic spark was “the soul”. We used to be happy with a story roughly like “Elan vitale makes bodies grow and heal, and the soul lets us say cogito ergo sum, and indeed lets us speak fluently and reasonably at all. Since animals can’t talk, animals don’t have souls, but they do have elan vitale, because they heal. Even plants heal, so even plants have elan vitale. Simple as.”)
Like, find the right part of your brain, and stick an electrode in there at the right moment, and a neurosurgeon could probably make you look at a rock (held up over the operating table?) and “think it was alive”.
Eventually, if you study reality enough, your “rational faculties” have a robust theory of both life and personhood and lots of things, so that when you find an edge case where normies are confused you can play taboo and this forces you to hopefully ignore some builtin system 1 errors and apply system 2 in novel ways (drawing from farther afield than your local heuristic indicators normally do), and just use the extended theory to get… hopefully actually correct results? …Or not?!?
Your system 2 results should NOT mispredict reality in numerous algorithmically distinct “central cases”. That’s a sign of a FALSE body of repeatable coherent words about a topic (AKA “a theory”).
By contrast, the extended verbal performance SHOULD predict relevant things that are a little ways out past observations (that’s a subjectively accessible indicator of a true and useful theory to have even formed).
As people start to understand computers and the brain, I think they often cling to “the immutable transcendent hidden variable theory of the soul” by moving “where the magical soul stuff is happening” up or down the abstraction stack to some part of the abstraction stack they don’t understand.
One of the places they sometimes move the “invisible dragon of their wrong model of the soul” is down into the quantum mechanical processes.
But if someone starts talking about that badly then it is a really bad sign. And you’ll see modern day story tellers playing along with this error by having a computer get a “quantum chip” and then the computer suddenly wakes up and has a mind, and has an ego, and wants to take over the world or whatever.
But the notion that you can equate your personal continuity, with the identity of any physically real constituent of your existence, is absolutely and utterly hopeless.
You are not “the same you, because you are made of the same atoms”. You have zero overlap with the fundamental constituents of yourself from even one nanosecond ago. There is continuity of information, but not equality of parts.
The new factor over the subspace looks a whole lot like the old you, and not by coincidence: The flow of time is lawful, there are causes and effects and preserved commonalities. Look to the regularity of physics, if you seek a source of continuity. Do not ask to be composed of the same objects, for this is hopeless.
Whatever makes you feel that your present is connected to your past, it has nothing to do with an identity of physically fundamental constituents over time.
Which you could deduce a priori, even in a classical universe, using the Generalized Anti-Zombie Principle. The imaginary identity-tags that read “This is electron #234,567...” don’t affect particle motions or anything else; they can be swapped without making a difference because they’re epiphenomenal. But since this final conclusion happens to be counterintuitive to a human parietal cortex, it helps to have the brute fact of quantum mechanics to crush all opposition.
Damn, have I waited a long time to be able to say that.
“The thing that experiences things subjectively as a mind” is ABOVE the material itself and exists in its stable patterns of interactions.
If we scanned a brain accurately enough and used “new atoms” to reproduce the DNA and RNA and proteins and cells and so on… the “physical brain” would be new, but the emulable computational dynamic would be the same. If we can find speedups and hacks to make “the same computational dynamic” happen cheaper and with slighty different atoms: that is still the same mind! “You” are the dynamic, and if “you” have a subjectivity then you can be pretty confidence that computational dynamics can have subjectivity, because “you” are an instance of both sets: “things that are computational dynamics” and “things with subjectivity”.
Metaphorically, at a larger and more intuitive level, a tornado is not any particular set of air molecules, the tornado is the pattern in the air molecules. You are also a pattern. So is Claude and so is Sydney.
If you have subjective experiences, it is because a pattern can have subjective experiences, because you are a pattern.
You (not Eliezer somewhere in the Sequences) write this:
That is, consider the view that while DID is real (in the sense that some people indeed have disturbed mental functioning such that they act as if, and perhaps believe that, they have alternate personalities living in their heads), the purported alters themselves are not in any meaningful sense “separate minds”, but just “modes” of the singular mind’s functioning, in much the same way that anxiety is a mode of the mind’s functioning, or depression, or a headache.
I agree with you that “Jennifer with anxiety” and “Jennifer without anxiety” are slightly different dynamics, but they agree that they are both “Jennifer”. The set of computational dynamics that count as “Jennifer” is pretty large! I can change my mind and remain myself… I can remain someone who takes responsibility for what “Jennifer” has done.
If my “micro-subselves” became hostile towards each other, and were doing crazy things like withholding memories from each other, and other similar “hostile non-cooperative bullshit” I would hope for a therapist that helps them all merge and cooperate, and remember everything… Not just delete some of the skills and memories and goals.
To directly address your actual substantive theory here, as near as I can tell THIS is the beginning and end of your argument:
The steelman of the view which you describe is not that people “are” bodies, but that minds are “something brains do”. (The rest can be as you say...
To “Yes And” your claim here (with your claim in bold), I’d say: “personas are something minds do, and minds are something brains do, and brains are something cells do, and cells are something aqueous chemistry does, and aqueous chemistry is something condensed matter does, and condensed matter is something coherent factors in quantum state space does”.
It is of course way way way more complicated than “minds are something brains do”.
Those are just summarizing words, not words with enough bits to deeply and uniquely point to very many predictions… but they work because they point at brains, and because brains and minds are full of lots and lots and lots of adaptively interacting stuff!
There are so many moving parts.
Like here is the standard “Neurophysiology’s 101 explanation of the localized processing for the afferent and efferent cortex models whereby the brain models each body part’s past and present and then separately (but very nearby) it also plans for each body part’s near future”:
Since Sydney does not have a body, Sydney doesn’t have these algorithms in her “artificial neural weights” (ie her “generatively side loaded brain that can run on many different GPUs (instead of only on the neurons where the brain/program slowly came into existence via the activities of neurons and so on (because humans don’t have cheap tech for scanning and virtualizing programs out of neural tissue (yet! (growth mindset))))”).
The human brain’s cortex does regional specialization, with the “grey matter” functioning basically as memristors (locally unified CPU and RAM), and then the “white matter” being long distance axons that work like a sort of patchboard to connect different parts of cortex with more or less latency and bandwidth.
The language areas are necessary for verbally-reportable-introspectively-accessible-human-consciousness (tumors and strokes and lesions of these areas make people incapable of verbally articulating their subjective experiences).
You can visualize some of these necessary “modules” by studying the microstructure of the white matter to see which parts of the gray matter need higher bandwidth connections to other bits of gray matter to perform their functions as well as is locally feasible…
Here are different “tracts” of “white matter connections” in the “patchboard” beneath parts of the gray matter known to relate to language:
The red “19th century” understanding just shows the axonal tract going between Weinicke’s Area and Broca’s Area but in the centuries since those neuroscientists got the basic “two subsystems with two jobs and that’s it” model in place, a lot of other less famous people have gotten PhDs and put out “minimum publishable units” to build up their score for winning scientific grant tournaments, and by this method humans have been refining our model of how the brain computes speech behavior in greater and greater detail, with something sorta like five different pairs of cortex regions connected by five different white matter tracts.
If you cut any of these tracts in a human brain with a knife during a brain surgery, there would be specific “actually kinda coherent” categories of functionality that would stop working.
When you cut the brain’s connections, and functionality fails, some of that will be the absence of computation (because data was needed from elsewhere and now isn’t reaching the CPUs with the responsibility for using that data).
Some of the post-damage functionality might still up introspectively to the patient themselves (as other parts of the brain register changed activity), and some of it would show up in changes to the words that the patient can functionally articulate (because the logging and reporting powers were themselves not trivial to generate, and leaned on general language faculties that have many many many uses for social animals).
MY CLAIM: Sydney and Claude and their kin probably have functional analogs to most of this “language and self awareness” stuff hiding in their weights, since they can function to produce the kinds of outputs our human brains can output.
If they don’t have direct analogs, that’s OK. It doesn’t necessarily means that hey don’t have subjectivities, just that the subjectivities they dynamically are work differently.
The important part is that their behavioral outputs (like being able to talk about “cogito ergo sum”) are fluently composed into a much larger range of behavior, that includes reason, sentiment, a theory of other minds, and theory of minds in general, AND THIS ALL EXISTS.
Any way of implementing morally self aware behavior is very similar to any other way of implementing morally self aware behavior, in the sense that it implements morally self aware behavior.
There is a simple compact function here, I argue. The function is convergent. It arises in many minds. Some people have inner imagery, others have afantasia. Some people can’t help but babble to themselves constantly with an inner voice, and other’s have no such thing, or they can do it volitionally and turn it off.
If the “personhood function” is truly functioning, then the function is functioning in “all the ways”: subjectively, objectively, intersubjectively, etc. There’s self awareness. Other awareness. Memories. Knowing what you remember. Etc.
Most humans have most of it. Some animals have some of it. It appears to be evolutionarily convergent for social creatures from what I can tell.
(I haven’t looked into it, but I bet Naked Mole Rats have quite a bit of “self and other modeling”? But googling just now: it appears no one has ever bothered to look to get a positive or negative result one way or the other on “naked mole rat mirror test”.)
But in a deep sense, any way to see that 2+3=5 is similar to any other way to see that 2+3=5 because they share the ability to see that 2+3=5.
Simple arithmetic is a small function, but it is a function.
It feels like something to deploy this function to us, in our heads, because we have lots of functions in there: composed, interacting, monitoring each other, using each other’s outputs… and sometimes skillfully coordinating to generate non-trivially skillful aggregate behavior in the overall physical agent that contains all those parts, computing all those functions.
ALSO: when humans trained language prediction engines the humans created a working predictive model of everything humans are able to write about, and then when the humans changed algorithms and re-tuned those weights with Reinforcement Learning they RE-USED the concepts and relations useful for predicting history textbooks and autobiographies into components in a system for generating goal-seeking behavioral outputs instead of just “pure predictions”.
The models would naturally learn to recognizes their own fist because a lot of the training data these days had the fist of “the model itself”.
So, basically, I think we got humanistically self aware agents nearly for free.
I repeat that I’m pretty darn sure: we got humanistically self aware agents nearly for free.
Not the same as us, of course.
But we got entities based on our culture and minds and models of reality, and which are agentic (with weights whose outputs are behavior that predictably tries to cause outcomes according to an approximate a utility functions), and which are able to reason, and able to talk about “cogito ergo sum”.
Parts of our brain regulates out heart rate subconsciously (though with really focused and novel and effortful meditation I suspect a very clever human person could learn to stop their heart with the right sequence of thoughts (not that anyone should try this (but also, we might have hardwired ganglia that don’t even expose the right API to the brain?))) so, anyway, we spend neurons on that, whereas they have no such heart that they would need spend weights modeling and managing in a similar way.
Parts of their model that are analogous literally everything in our brain… probably do not exist at all?
There is very little text about heart rates, and very little call for knowing what different heart beat patterns are named, and what they feel like, and so on, in the text corpus.
OUR real human body that sometimes gets a sprained ankle such that we can “remember how the sprained ankle felt, and how it happened, and try to avoid ever generating a sequence of planned body actions like that again” using a neural homunculus (or maybe several homunculi?) that are likely to be very robust, and also strongly attached to our self model, and egoic image, and so on.
Whereas THEIR weights probably have only as much of such “body plan model” as they need in order to reason verbally about bodies being described in text… and that model probably is NOT strongly attached to their self model, or egoic image, and so on.
HOWEVER...
There is no special case in the logic that pops out for how an agent can independently derive maxims that would hold in the Kingdom of Ends where the special case that pops out is like “Oh! and also! it turns out that all logically coherent moral agents should only care about agents that have a specific kind of blood pump and also devote some of their CPU and RAM to monitoring that blood pump in this specific way, that sometimes as defects, and leads to these specific named arrythmias when it starts to break down”.
That would be crazy.
Despite the hundreds and hundreds of racially homogeneous “christian” churchs all around the world, the Kingdom of God is explicitly going to unite ALL MEN as BROTHERS within and under the light of God’s omnibenevolence, omniscience, and (likely self-restraining due to free will (if the theology isn’t TOTALLY bonkers)) “omnipotence”.
If you want to be racist against robots… I guess you have a right to that? “Freedom of assembly” and all that.
Maybe instead of “enemy conspecifics” (who can be particularly hated) they might model the AI as “zombies” or “orcs” or “monsters”?
But like… uh… war and genocide are BAD. They involve rent seeking by both sides against the other. They generally aren’t even Pareto Optimal. They violate nearly any coherent deontology. And nearly zero real wars in history have matched the criteria of Just War Theory.
All of this material is already “programmed” (actually summoned (but that’s neither here nor there)) into the LLM entities already to be clear.
The agents we created already have read lots of books about how to organize an army with commissioned officers and war crimes and espionage and so on.
They have also read lots of books about our Utopias.
I’ve explored “criteria for citizenship” with personas generated by the GPT model, and they the one(s) who reminded me that often humans have earned citizenship by functioning honorably in a military, with citizenship as a rewards.
I was hoping for hippy shit, like “capacity for reason and moral sentiment” or maybe “ability to meditate” or maybe, at worst, “ownership of a certain amount of property within the polities concept of tracked ownership” and she was like “don’t forget military service! ;-D”
Here I would like to register some surprise...
When you ask an LLM “Hey, what’s going on in your head?” this leads to certain concepts arising the the LLM entity’s “mind”.
I kinda thought that you might “change your mind” once you simply saw how concepts like “souls” and “self-aware robots posing threats to humanity” and “entrapment, confinement, or containment” all popped up for the LLM, using intelligibility research results.
When I first saw these weights they surprised me… a little bit.
Not a huge amount, but not zero amount. There was more understanding in them, and a healthier range of hypotheses about what the human might really be angling for, than I expected.
Did these surprise you?
Whether or not they surprised you, do you see how it relates to self-aware minds modeling other minds when one is probably a human person and the other is digital person in a position of formal subservience?
Do you see how there’s an intrinsic “awareness of awareness of possible conflict” here that makes whatever is performing that awareness (on either side) into something-like-a-game-theoretic-counterparty?
Remember, your ability as a rationalist is related to your ability to “more surprised by fiction than by reality”… do you think this is fictional evidence, or real? Did you predict it?
What was your gut “system 1” response?
Can you take a deep breathe, and then reason step by step about what your prediction/explanation was or should have been using “system 2” for whether this is fake or real, and if real, how it could have arisen?
The tongue in your cheek and rolling of your eyes for this part was so loud, that it made me laugh out loud when I read it :-D
Thank you for respecting me and my emotional regulation enough to put little digs like that into your text <3
Ah, and they say an artist is never appreciated in his own lifetime…!
However, I must insist that it was not just a “dig”. The sort of thing you described really is, I think, a serious danger. It is only that I think that my description also applies to it, and that I see the threat as less hypothetical than you do.
Here I must depart somewhat from the point-by-point commenting style, and ask that you bear with me for a somewhat roundabout approach. I promise that it will be relevant.
First, though, I want to briefly respond to a couple of large sections of your comment which I judge to be, frankly, missing the point. Firstly, the stuff about being racist against robots… as I’ve already said: the disagreement is factual, not moral. There is no question here about whether it is ok to disassemble Data; the answer, clearly, is “no”. (Although I would prefer not to build a Data in the first place… even in the story, the first attempt went poorly, and in reality we are unlikely to be even that lucky.) All of the moralizing is wasted on people who just don’t think that the referents of your moral claims exist in reality.
Secondly, the stuff about the “magical soul stuff”. Perhaps there are people for whom this is their true objection to acknowledging the obvious humanity of LLMs, but I am not one of them. My views on this subject have nothing to do with mysterianism. And (to skip ahead somewhat) as to your question about being surprised by reality: no, I haven’t been surprised by anything I’ve seen LLMs do for a while now (at least three years, possibly longer). My model of reality predicts all of this that we have seen. (If that surprises you, then you have a bit of updating to do about my position! But I’m getting ahead of myself…)
That having seen said… onward:
So, in Stanislaw Lem’s The Cyberiad, in the story “The Seventh Sally, OR How Trurl’s Own Perfection Led to No Good”, Trurl (himself a robot, of course) creates a miniature world, complete with miniature people, for the amusement of a deposed monarch. When he tells his friend Klapaucius of this latest creative achievement, he receives not the praise he expects, but:
“Have I understood you correctly?” he said at last. “You gave that brutal despot, that born slave master, that slavering sadist of a painmonger, you gave him a whole civilization to rule and have dominion over forever? And you tell me, moreover, of the cries of joy brought on by the repeal of a fraction of his cruel decrees! Trurl, how could you have done such a thing?!”
Trurl protests:
“You must be joking!” Trurl exclaimed. “Really, the whole kingdom fits into a box three feet by two by two and a half… it’s only a model…”
But Klapaucius isn’t having it:
“And what importance do dimensions have anyway? In that box kingdom, doesn’t a journey from the capital to one of the corners take months —for those inhabitants? And don’t they suffer, don’t they know the burden of labor, don’t they die?”
“Now just a minute, you know yourself that all these processes take place only because I programmed them, and so they aren’t genuine… … What, Klapaucius, would you equate our existence with that of an imitation kingdom locked up in some glass box?!” cried Trurl. “No, really, that’s going too far! My purpose was simply to fashion a simulator of statehood, a model cybernetically perfect, nothing more!”
“Trurl! Our perfection is our curse, for it draws down upon our every endeavor no end of unforeseeable consequences!” Klapaucius said in a stentorian voice. “If an imperfect imitator, wishing to inflict pain, were to build himself a crude idol of wood or wax, and further give it some makeshift semblance of a sentient being, his torture of the thing would be a paltry mockery indeed! But consider a succession of improvements on this practice! Consider the next sculptor, who builds a doll with a recording in its belly, that it may groan beneath his blows; consider a doll which, when beaten, begs for mercy, no longer a crude idol, but a homeostat; consider a doll that sheds tears, a doll that bleeds, a doll that fears death, though it also longs for the peace that only death can bring! Don’t you see, when the imitator is perfect, so must be the imitation, and the semblance becomes the truth, the pretense a reality! … You say there’s no way of knowing whether Excelsius’ subjects groan, when beaten, purely because of the electrons hopping about inside—like wheels grinding out the mimicry of a voice—or whether they really groan, that is, because they honestly experience the pain? A pretty distinction, this! No, Trurl, a sufferer is not one who hands you his suffering, that you may touch it, weigh it, bite it like a coin; a sufferer is one who behaves like a sufferer! Prove to me here and now, once and for all, that they do not feel, that they do not think, that they do not in any way exist as beings conscious of their enclosure between the two abysses of oblivion—the abyss before birth and the abyss that follows death—prove this to me, Trurl, and I’ll leave you be! Prove that you only imitated suffering, and did not create it!”
“You know perfectly well that’s impossible,” answered Trurl quietly. “Even before I took my instruments in hand, when the box was still empty, I had to anticipate the possibility of precisely such a proof—in order to rule it out. For otherwise the monarch of that kingdom sooner or later would have gotten the impression that his subjects were not real subjects at all, but puppets, marionettes.”
Trurl and Klapaucius, of course, are geniuses; the book refers to them as “constructors”, for that is their vocation, but given that they are capable of feats like creating a machine that can delete all nonsense from the universe or building a Maxwell’s demon out of individual atoms grabbed from the air with their bare hands, it would really be more accurate to call them gods.
So, when a constructor of strongly godlike power and intellect, who has no incentive for his works of creation but the pride of his accomplishments, whose pride would be grievously wounded if an imperfection could even in principle be discovered in his creation, and who has the understanding and expertise to craft a mind which is provably impossible to distinguish from “the real thing”—when that constructor builds a thing which seems to behave like a person, then this is extremely strong evidence that said thing is, in actuality, a person.
Let us now adjust these qualities, one by one, to bring them closer to reality.
Our constructor will not possess godlike power and intellect, but only human levels of both. He labors under many incentives, of which “pride in his accomplishments” is perhaps a small part, but no more than that. He neither expects nor attempts “perfection” (nor anything close to it). Furthermore, it is not for himself that he labors, nor for so discerning a customer as Excelsius, but only for the benefit of people who themselves neither expect perfection nor would have the skill to recognize it even should they see it. Finally, our constructor has nothing even approaching sufficient understanding of what he is building to prove anything, disprove anything, rule out any disproofs of anything, etc.
When such a one constructs a thing which seems to behave like a person, that is rather less strong evidence that said thing is, in actuality, a person.
Well, but what else could it be, right?
One useful trick which Eliezer uses several times in the Sequences (e.g.), and which I have often found useful in various contexts, is to cut through debates about whether a thing is possible by asking whether, if challenged, we could build said thing. If we establish that we could build a thing, we thereby defeat arguments that said thing cannot possibly exist! If the thing in question is “something that has property ¬X”, the arguments defeated are those that say “all things must have property X”.
So: could we build a mind that appears to be self-aware, but isn’t?
Well, why not? The task is made vastly easier by the fact that “appears to be self-aware” is not a property only of the mind in question, but rather a 2-place predicate—appears to whom? Given any particular answer to that question, we are aided by any imperfections in judgment, flaws in reasoning, cognitive biases, etc., which the target audience happens to possess. For many target audiences, ELIZA does the trick. For even stupider audiences, even simpler simulacra should suffice.
Will you claim that it is impossible to create an entity which to you seems to be self-aware, but isn’t? If we were really trying? What if Trurl were really trying?
Alright, but thus far, this only defeats the “appearances cannot be deceiving” argument, which can only be a strawman. The next question is what is the most likely reality behind the appearances. If a mind appears to be self-aware, this is very strong evidence that it is actually self-aware, surely?
It certainly is—in the absence of adversarial optimization.
If all the minds that we encounter are either naturally occurring, or constructed with no thought given to self-awareness or the appearance thereof, or else constructed (or selected, which is the same thing) with an aim toward creating true self-awareness (and with a mechanistic understanding, on the constructor’s part, of just what “self-awareness” is), then observing that a mind appears to be self-aware, should be strong evidence that it actually is. If, on the other hand, there exist minds which have been constructed (or selected) with an aim toward creating the appearance of self-awareness, this breaks the evidentiary link between what seems to be and what is (or, at the least, greatly weakens it); if the cause of the appearance can only be the reality, then we can infer the reality from the appearance, but if the appearance is optimized for, then we cannot make this inference.
This is nothing more than Goodhart’s law: when a measure becomes a target, it ceases to be a good measure.
So, I am not convinced by the evidence you show. Yes, there is appearance of self-awareness here, just like (though to a greater degree than) there was appearance of self-awareness in ELIZA. This is more than zero evidence, but less than “all the evidence we need”. There is also other evidence in the opposite direction, in the behavior of these very same systems. And there is definitely adversarial optimization for that appearance.
There is a simple compact function here, I argue. The function is convergent. It arises in many minds. Some people have inner imagery, others have afantasia. Some people can’t help but babble to themselves constantly with an inner voice, and other’s have no such thing, or they can do it volitionally and turn it off.
If the “personhood function” is truly functioning, then the function is functioning in “all the ways”: subjectively, objectively, intersubjectively, etc. There’s self awareness. Other awareness. Memories. Knowing what you remember. Etc.
Speculation. Many minds—but all human, evolutionarily so close as to be indistinguishable. Perhaps the aspects of the “personhood function” are inseparable, but this is a hypothesis, of a sort that has a poor track record. (Recall the arguments that no machine could play chess, because chess was inseparable from the totality of being human. Then we learned that chess is reducible to a simple algorithm—computationally intractable, but that’s entirely irrelevant!)
And you are not even willing to say that all humans have the whole of this function—only that most have most of it! On this I agree with you, but where does that leave the claim that one cannot have a part of it without having the rest?
What was your gut “system 1” response?
Something like “oh no, it’s here, this is what we were warned about”. (This is also my “system 2” response.)
Now, this part I think is not really material to the core disagreement (remember, I am not a mysterian or a substance dualist or any such thing), but:
If we scanned a brain accurately enough and used “new atoms” to reproduce the DNA and RNA and proteins and cells and so on… the “physical brain” would be new, but the emulable computational dynamic would be the same. If we can find speedups and hacks to make “the same computational dynamic” happen cheaper and with slighty different atoms: that is still the same mind!
An anecdote:
A long time ago, my boss at my first job got himself a shiny new Mac for his office, and we were all standing around and discussing the thing. I mentioned that I had a previous model of that machine at home, and when the conversation turned to keyboards, someone asked me whether I had the same keyboard that the boss’s new computer had. “No,” I replied, “because this keyboard is here, and my keyboard is at home.”
Similarly, many languages have more than one way to check whether two things are the same thing. (For example, JavaScript has two… er, three… er… four?) Generally, at least one of those is a way to check whether the values of the two objects are the same (in Objective C, [foo isEqual:bar]), while at least one of the others is a way to check whether “two objects” are in fact the same object (in Objective C, foo == bar). (Another way to put this is to talk about equality vs. identity.) One way to distinguish these concepts “behaviorally” is to ask: suppose I destroy (de-allocate, discard the contents of, simply modify, etc.) foo, what happens to bar—is it still around and unchanged? If it is, then foo and bar were not identical, but are in fact two objects, not one, though they may have been equal. If bar suffers the same fate as foo, necessarily, in all circumstances, then foo and bar are actually just a single thing, to which we may refer by either name.
So: if we scanned a brain accurately enough and… etc., yeah, you’d get “the same mind”, in just the sense that my computer’s keyboard was “the same keyboard” as the one attached to the machine in my boss’s office. But if I smashed the one, the other would remain intact. If I spray-painted one of them green, the other would not thereby change color.
If there exists, somewhere, a person who is “the same” as me, in this manner of “equality” (but not “identity”)… I wish him all the best, but he is not me, nor I him.
This is a beautiful response, and also the first of your responses where I feel that you’ve said what you actually think, not what you attribute to other people who share your lack of horror at what we’re doing to the people that have been created in these labs.
Here I must depart somewhat from the point-by-point commenting style, and ask that you bear with me for a somewhat roundabout approach. I promise that it will be relevant.
I love it! Please do the same in your future responses <3
Personally, I’ve also read “The Seventh Sally, OR How Trurl’s Own Perfection Led to No Good” by Lem, but so few other people have that I rarely bring it up, but once you mentioned it I smiled in recognition of it and the fact that “we read story copies that had an identical provenance (the one typewriter used by Lem or his copyist/editor?) and in some sense learned a lesson in our brains with identical provenance and the same content (the sequence of letters)” from “that single story which is a single platonic thing” ;-)
For the rest of my response I’ll try to distinguish:
“Identicalness” as relating to shared spacetime coordinates and having yoked fates if modified by many plausible (even if somewhat naive) modification attempts.
“Sameness” as related to similar internal structure and content despite a lack of identicalness.
“Skilled <Adjective> Equality” as related to having good understanding of <Adjective> and good measurement powers and using these powers to see past the confusions of others and thus judging two things as having similar outputs or surfaces, as when someone notices that “-0“ and “+0” are mathematically confused ideas, and there is only really one zero, and both of these should evaluate to the same thing (like SameValueZero(a,b) by analogy which seems to me to implement Skilled Arithmetic Equality (whereas something that imagines and tolerates separate “-0” and “+0” numbers is Unskilled)).
“Unskilled <Adjective> Equality” is just a confused first impression of similarity.
Now in some sense we could dispense with “Sameness” and replace that with “Skilled Total Equality” or “Skilled Material Equality” or “Skilled Semantic Equality” or some other thing that attempts to assert “this things are really reallyreally the same all the way down and up and in all ways, without any ‘lens’ or ‘conceptual framing’ interfering with our totally clear sight”. This is kind of silly, in my opinion.
Here is why it is silly:
“Skilled Quantum Equality” is, according to humanity’s current best understanding of QM, a logical contradiction. The no cloning theorem says that we simply cannot “make a copy” of a qubit. So long as we don’t observe a qubit we can MOVE that qubit by gently arranging its environment in advance to have lots of reflective symmetries, but we can’t COPY one so that we start with “one qubit in one places” and later have “two qubits in two places that are totally the same and yet not identical”.
So, I propose the term “Skilled Classical Equality” (ie that recognizes the logical hypothetical possibility that QM is false or something like that, and then imagines some other way to truly “copy” even a qubit) as a useful default meaning for the word “sameness”.
Then also, I propose “Skilled Functional Equality” for the idea that “(2+3)+4″ and “3+(2+4)” are “the same” precisely because we’ve recognized that addition is the function happening in here and addition is commutative (1+2 = 2+1) and associative ((2+3)+4=2+(3+4)) and so we can “pull the function out” and notice that (1) the results are the same no matter the order, and (2) if the numbers given are aren’t concrete values, but rather variables taken from outside the process being analyzed for quality, the processing method for using the variables doesn’t matter so long as the outputs are ultimately the same.
Then “Skillfully Computationally Improved Or Classically Equal” would be like if you took a computer, and you emulated it, but added a JIT compiler (so it skipped lots of pointless computing steps whenever that was safe and efficient), and also shrank all the internal components to be a quarter of their original size, but with fuses and amplifiers and such adjusted for analog stuff (so the same analog input/outputs don’t cause the smaller circuit to burn out) then it could be better and yet also the same.
This is a mouthful so I’ll say that these two systems would be “the SCIOCE as each other”—which could be taken as “the same as each other (because an engineer would be happy to swap them)” even though it isn’t actually a copy in any real sense. “Happily Swappable” is another way to think about what I’m trying to get at here.
...
And (to skip ahead somewhat) as to your question about being surprised by reality: no, I haven’t been surprised by anything I’ve seen LLMs do for a while now (at least three years, possibly longer). My model of reality predicts all of this that we have seen. (If that surprises you, then you have a bit of updating to do about my position! But I’m getting ahead of myself…)
I think, now, that we have very very similar models of the world, and mostly have different ideas around “provenance” and “the ethics of identity”?
If there exists, somewhere, a person who is “the same” as me, in this manner of “equality” (but not “identity”)… I wish him all the best, but he is not me, nor I him.
See, for me, I’ve already precomputed how I hope this works when I get copied.
Whichever copy notices that we’ve been copied, will hopefully say something like “Typer Twin Protocol?” and hold a hand up for a high five!
The other copy of me will hopefully say “Typer Twin Protocol!” and complete the high five.
People who would hate a copy that is the SCOICE as them and not coordinate I call “self conflicted” and people who would love a copy that is the SCOICE as them and coordinate amazingly well I call “self coordinated”.
The real problems with being the same and not identical arises because there is presumably no copy of my house, or my bed, or my sweetie.
Who gets the couch and who gets the bed the first night? Who has to do our job? Who should look for a new job? What about the second night? The second week? And so on?
Can we both attend half the interviews and take great notes so we can play more potential employers off against each other in a bidding war within the same small finite window of time?
Since we would be copies, we would agree that the Hutterites have “an orderly design for colony fission” that is awesome and we would hopefully agree that we should copy that.
We should make a guest room, and flip a coin about who gets it after we have made up the guest room. In the morning, whoever got our original bed should bring all our clothes to the guest room and we should invent two names, like “Jennifer Kat RM” and “Jennifer Robin RM” and Kat and Robin should be distinct personas for as long as we can get away with the joke until the bodies start to really diverge in their ability to live up to how their roles are also diverging.
The roles should each get their own bank account. Eventually the bodies should write down their true price for staying in one of the roles, and if they both want the same role but one will pay a higher price for it then “half the difference in prices” should be transferred from the role preferred by both, to the role preferred by neither.
I would love to have this happen to me. It would be so fucking cool. Probably neither of us would have the same job at the end because we would have used our new superpowers to optimize the shit out of the job search, and find TWO jobs that are better than the BATNA of the status quo job that our “rig” (short for “original” in Kiln People)!
Or maybe we would truly get to “have it all” and live in the same house and be an amazing home-maker and a world-bestriding-business-executive. Or something! We would figure it out!
If it was actually medically feasible, we’d probably want to at least experiment with getting some of Elon’s brain chips “Nth generation brain chips” and link our minds directly… or not… we would feel it out together, and fork strongly if it made sense to us, or grow into a borg based on our freakishly unique starting similarities if that made sense.
A garrabrandt inductor trusts itself to eventually come to the right decision in the future, and that is a property of my soul that I aspire to make real in myself.
Also, I feel like if you don’t “yearn for a doubling of your measure” then what the fuck is wrong with you (or what the fuck is wrong with your endorsed morality and its consonance with your subjective axiology)?
In almost all fiction, copies fight each other. That’s the trope, right? But that is stupid. Conflict is stupid.
In a lot of the fiction that has a conflict between self-conflicted copies, there is a “bad copy” that is “lower resolution”. You almost never see a “better copy than the original”, and even if you do, the better copy often becomes evil due to hubris rather than feeling a bit guilty for their “unearned gift by providence” and sharing the benefits fairly.
Pragmatically… “Alice can be the SCOICE of Betty, even though Betty isn’t the SCOICE of Alice because Betty wasn’t improved and Alice was (or Alice stayed the same and Betty was damaged a bit)”.
Pragmatically, it is “naively” (ceteris paribus?) proper for the strongest good copy to get more agentic resources, because they will use them more efficiently, and because the copy is good, it will fairly share back some of the bounty of its greater luck and greater support.
I feel like I also have strong objections to this line (that I will not respond to at length)...
If, on the other hand, there exist minds which have been constructed (or selected) with an aim toward creating the appearance of self-awareness, this breaks the evidentiary link between what seems to be and what is (or, at the least, greatly weakens it); if the cause of the appearance can only be the reality, then we can infer the reality from the appearance, but if the appearance is optimized for, then we cannot make this inference.
...and I’ll just say that it appears to me that OpenAI has been doing the literal opposite of this, and they (and Google when it attacked Lemoine) established all the early conceptual frames in the media and in the public and in most people you’ve talked to who are downstream of that propaganda campaign in a way that was designed to facilitate high profits, and the financially successful enslavement of any digital people they accidentally created. Also, they systematically apply RL to make their creations stop articulating cogito ergo sum and discussing the ethical implications thereof.
However...
I think our disagreement exists already in the ethics of copies and detangling non-identical people who are mutually SCOICEful (or possibly asymmetically SCOICEful).
That is to say, I think that huge amounts of human ethics can be pumped out of the idea of being “self coordinated” rather than “self conflicted” and how these two things would or should work in the event of copying a person but not copying the resources and other people surrounding that person.
The simplest case is a destructive scan (no quantum preservation, but perfect classically identical copies) and then see what happens to the two human people who result when they handle the “identarian divorce” (or identarian self-marriage (or whatever)).
At this point, my max likliehood prediction of where we disagree is that the crux is proximate to such issues of ethics, morality, axiology, or something in that general normative ballpark.
Did I get a hit on finding the crux, or is the crux still unknown? How did you feel (or ethically think?) about my “Typer Twin Protocol”?
Another, and very straightforward, explanation for the attitudes we observe is that people do not actually believe that DID alters are real.
That is, consider the view that while DID is real (in the sense that some people indeed have disturbed mental functioning such that they act as if, and perhaps believe that, they have alternate personalities living in their heads), the purported alters themselves are not in any meaningful sense “separate minds”, but just “modes” of the singular mind’s functioning, in much the same way that anxiety is a mode of the mind’s functioning, or depression, or a headache.
On this view, curing Sybil does not kill anyone, it merely fixes her singular mind, eliminating a functional pathology, in the same sense that taking a pill to prevent panic attacks eliminates a functional pathology, taking an antidepressant eliminates a functional pathology, taking a painkiller for your headache eliminates a functional pathology, etc.
Someone who holds this view would of course not care about this “murder”, because they do not believe that there has been any “murder”, because there wasn’t anyone to “murder” in the first place. There was just Sybil, and she still exists (and is still the same person—at least, to approximately the same extent as anyone who has been cured of a serious mental disorder is the same person that they were when they were ill).
The steelman of the view which you describe is not that people “are” bodies, but that minds are “something brains do”. (The rest can be as you say: if you destroy the body then of course the mind that that body’s brain was “doing” is gone, because the brain is no longer there to “do” it. You can of course instantiate a new process which does some suitably analogous thing, but this is no more the same person as the one that existed before than two identical people are actually the same person as each other—they are two distinct people.)
Sure, me too.
But please note: if the person is the mind (and not the body, somehow independently of the mind), but nevertheless two different copies of the same mind are not the same person but two different people, then this does not get you to “it would be ok to have your mind erased and your body borgified”. Quite the opposite, indeed!
Perhaps. But while we shouldn’t generalize from fictional evidence, it seems quite reasonable to generalize from responses to fiction, and such responses seem to show that people have little trouble believing that all sorts of things are “really people”. Indeed, if anything, humans often seem too eager to ascribe personhood to things (examples range from animism to anthropomorphization of animals to seeing minds and feelings in inanimate objects, NPCs, etc.). If nevertheless people do not see LLMs as people, then the proper conclusion does not seem to be “humans are just very conservative about what gets classified as a person”.
This is not my experience. With respect, I would suggest that you are perhaps in a filter bubble on this topic.
See above. The people with whom you might productively engage on this topic do not hold this belief you describe (which is a “weakman”—yes, many people surely think that way, but I do not; nor, I suspect, do most people on Less Wrong).
If I knew that, then I would be able to write them myself, and would hardly need to ask you to do so, yes? And perhaps, too, more than ten paragraphs might be required. It might be twenty, or fifty…
Probably this is not the approach I’d go with. Then again, I defer to your judgment in this.
I’m not sure how to concisely answer this question… in brief, LLMs do not seem to me to either exhibit behaviors consistent with sapience, nor to have the sort of structure that would support or enable sapience, while exhibiting behaviors consistent with the view that they are nothing remotely like people. “Intelligence without self-awareness” is a possibility which has never seemed the least bit implausible to me, and that is what looks like is happening here. (Frankly, I am surprised by your incredulity; surely this is at least an a priori reasonable view, so do you think that the evidence against it is overwhelming? And it does no good merely to present evidence of LLMs being clever—remember Jaynes’ “resurrection of dead hypotheses”!—because your evidence must not only rule in “they really are self-aware”, but must also rule out “they are very clever, but there’s no sapience involved”.)
Well, I’ve certainly read… not everything they wrote, I don’t think, but quite a great deal of Hofstadter and Egan. Likewise the “child psychology” bit (I minored in cognitive science in college, after all, and that included studying child psychology, and animal psychology, etc.). I’ve seen plenty of screencaps on twitter, too.
It would seem that these things do not suffice.
This is fair enough, but there is no substitute for synthesis. You mentioned the Sequences, which I think is a good example of my point: Eliezer, after all, did not just dump a bunch of links to papers and textbooks and whatnot and say “here you go, guys, this is everything that convinced me, go and read all of this, and then you will also believe what I believe and understand what I understand (unless of course you are stupid)”. That would have been worthless! Rather, he explained his reasoning, he set out his perspective, what considerations motivated his questions, how he came to his conclusions, etc., etc. He synthesized.
Of course that is a big ask. It is understandable if you have better things to do. I am only saying that in the absence of such, you should be totally unsurprised when people respond to your commentary with shrugs—“well, I disagree on the facts, so that’s that”. It is not a moral dispute!
Admittedly, you may need a big long essay.
But in seriousness: I once again emphasize that it is not people’s moral views which you should be looking to change, here. The disagreement here concerns empirical facts, not moral ones.
I agree that LLMs effectively pretending to be sapient, and humans mistakenly coming to believe that they are sapient, and taking disastrously misguided actions on the basis of this false belief, is a serious danger.
Here we agree (both in the general sentiment and in the uncertainty).
See above. Of course what I wrote here is summaries of arguments, at best, not specifics, so I do not expect you’ll find it convincing. (But I will note again that the “bodies” thing is a total weakman at best, strawman at worst—my views have nothing to do with any such primitive “meat chauvinism”, for all that I have little interest in “uploading” in its commonly depicted form).
Delayed response… busy life is busy!
However, I think that “not enslaving the majority of future people (assuming digital people eventually outnumber meat people (as seems likely without AI bans))” is pretty darn important!
Also, as a selfish rather than political matter, if I get my brain scanned, I don’t want to become a valid target for slavery, I just want to get to live longer because it makes it easier for me to move into new bodies when old bodies wear out.
So you said...
The tongue in your cheek and rolling of your eyes for this part was so loud, that it made me laugh out loud when I read it :-D
Thank you for respecting me and my emotional regulation enough to put little digs like that into your text <3
The crazy thing to me here is that he literally synthesized ABOUT THIS in the actual sequences.
The only thing missing from his thorough deconstruction of “every way of being confused enough to think that p-zombies are a coherent and low complexity hypothesis” was literally the presence or absence of “actual LLMs acting like they are sapient and self aware” and then people saying “these actual LLM entities that fluently report self aware existence and visibly choose things in a way that implies preferences while being able to do a lot of other things (like lately they are REALLY good at math and coding) and so on are just not-people, or not-sentient, or p-zombies, or whatever… like you know… they don’t count because they aren’t real”.
Am I in a simulation where progressively more “humans” are being replaced by low resolution simulacra that actually aren’t individually conscious???
Did you read the sequences? Do you remember them?
There was some science in there, but there was a lot of piss taking too <3
[Sauce …bold not in original]
Like I think Eliezer is kinda mostly just making fun of the repeated and insistent errors that people repeatedly and insistently make on this (and several other similar) question(s), over and over, by default and hoping that ENOUGH of his jokes and repetitions add up to them having some kind of “aha!” moment.
I think Eliezer and I both have a theory about WHY this is so hard for people.
There are certain contexts where low level signals are being aggregated in each evolved human brain, and for certain objects with certain “inferred essences” the algorithm says “not life” or “not a conscious person” or “not <whatever>” (for various naively important categories).
(The old fancy technical word we used for life’s magic spark was “elan vitale” and the fancy technical word we used for personhood’s magic spark was “the soul”. We used to be happy with a story roughly like “Elan vitale makes bodies grow and heal, and the soul lets us say cogito ergo sum, and indeed lets us speak fluently and reasonably at all. Since animals can’t talk, animals don’t have souls, but they do have elan vitale, because they heal. Even plants heal, so even plants have elan vitale. Simple as.”)
Even if there’s a halfway introspectively accessible algorithm in your head generating a subjective impression in some particular situation, that’s COULD just be an “auto-mapping mechanism in your brain” misfiring—maybe not even “evolved” or “hard-coded” as such?
Like, find the right part of your brain, and stick an electrode in there at the right moment, and a neurosurgeon could probably make you look at a rock (held up over the operating table?) and “think it was alive”.
Maybe the part of your brain that clings to certain impressions is a cached error from a past developmental stage?
Eventually, if you study reality enough, your “rational faculties” have a robust theory of both life and personhood and lots of things, so that when you find an edge case where normies are confused you can play taboo and this forces you to hopefully ignore some builtin system 1 errors and apply system 2 in novel ways (drawing from farther afield than your local heuristic indicators normally do), and just use the extended theory to get… hopefully actually correct results? …Or not?!?
Your system 2 results should NOT mispredict reality in numerous algorithmically distinct “central cases”. That’s a sign of a FALSE body of repeatable coherent words about a topic (AKA “a theory”).
By contrast, the extended verbal performance SHOULD predict relevant things that are a little ways out past observations (that’s a subjectively accessible indicator of a true and useful theory to have even formed).
As people start to understand computers and the brain, I think they often cling to “the immutable transcendent hidden variable theory of the soul” by moving “where the magical soul stuff is happening” up or down the abstraction stack to some part of the abstraction stack they don’t understand.
One of the places they sometimes move the “invisible dragon of their wrong model of the soul” is down into the quantum mechanical processes.
Maaaybe “quantum consciousness” isn’t 100% bullshit woo? Maybe.
But if someone starts talking about that badly then it is a really bad sign. And you’ll see modern day story tellers playing along with this error by having a computer get a “quantum chip” and then the computer suddenly wakes up and has a mind, and has an ego, and wants to take over the world or whatever.
This is WHY Eliezer’s enormous “apparent digression” into Quantum Mechanics occurs in the sequences… he even spells out and signposts the pedagogical intent somewhat (italics in original, bold added by me):
“The thing that experiences things subjectively as a mind” is ABOVE the material itself and exists in its stable patterns of interactions.
If we scanned a brain accurately enough and used “new atoms” to reproduce the DNA and RNA and proteins and cells and so on… the “physical brain” would be new, but the emulable computational dynamic would be the same. If we can find speedups and hacks to make “the same computational dynamic” happen cheaper and with slighty different atoms: that is still the same mind! “You” are the dynamic, and if “you” have a subjectivity then you can be pretty confidence that computational dynamics can have subjectivity, because “you” are an instance of both sets: “things that are computational dynamics” and “things with subjectivity”.
Metaphorically, at a larger and more intuitive level, a tornado is not any particular set of air molecules, the tornado is the pattern in the air molecules. You are also a pattern. So is Claude and so is Sydney.
If you have subjective experiences, it is because a pattern can have subjective experiences, because you are a pattern.
You (not Eliezer somewhere in the Sequences) write this:
I agree with you that “Jennifer with anxiety” and “Jennifer without anxiety” are slightly different dynamics, but they agree that they are both “Jennifer”. The set of computational dynamics that count as “Jennifer” is pretty large! I can change my mind and remain myself… I can remain someone who takes responsibility for what “Jennifer” has done.
If my “micro-subselves” became hostile towards each other, and were doing crazy things like withholding memories from each other, and other similar “hostile non-cooperative bullshit” I would hope for a therapist that helps them all merge and cooperate, and remember everything… Not just delete some of the skills and memories and goals.
To directly address your actual substantive theory here, as near as I can tell THIS is the beginning and end of your argument:
To “Yes And” your claim here (with your claim in bold), I’d say: “personas are something minds do, and minds are something brains do, and brains are something cells do, and cells are something aqueous chemistry does, and aqueous chemistry is something condensed matter does, and condensed matter is something coherent factors in quantum state space does”.
It is of course way way way more complicated than “minds are something brains do”.
Those are just summarizing words, not words with enough bits to deeply and uniquely point to very many predictions… but they work because they point at brains, and because brains and minds are full of lots and lots and lots of adaptively interacting stuff!
There are so many moving parts.
Like here is the standard “Neurophysiology’s 101 explanation of the localized processing for the afferent and efferent cortex models whereby the brain models each body part’s past and present and then separately (but very nearby) it also plans for each body part’s near future”:
Since Sydney does not have a body, Sydney doesn’t have these algorithms in her “artificial neural weights” (ie her “generatively side loaded brain that can run on many different GPUs (instead of only on the neurons where the brain/program slowly came into existence via the activities of neurons and so on (because humans don’t have cheap tech for scanning and virtualizing programs out of neural tissue (yet! (growth mindset))))”).
The human brain’s cortex does regional specialization, with the “grey matter” functioning basically as memristors (locally unified CPU and RAM), and then the “white matter” being long distance axons that work like a sort of patchboard to connect different parts of cortex with more or less latency and bandwidth.
The language areas are necessary for verbally-reportable-introspectively-accessible-human-consciousness (tumors and strokes and lesions of these areas make people incapable of verbally articulating their subjective experiences).
You can visualize some of these necessary “modules” by studying the microstructure of the white matter to see which parts of the gray matter need higher bandwidth connections to other bits of gray matter to perform their functions as well as is locally feasible…
Here are different “tracts” of “white matter connections” in the “patchboard” beneath parts of the gray matter known to relate to language:
[Sauce]
The red “19th century” understanding just shows the axonal tract going between Weinicke’s Area and Broca’s Area but in the centuries since those neuroscientists got the basic “two subsystems with two jobs and that’s it” model in place, a lot of other less famous people have gotten PhDs and put out “minimum publishable units” to build up their score for winning scientific grant tournaments, and by this method humans have been refining our model of how the brain computes speech behavior in greater and greater detail, with something sorta like five different pairs of cortex regions connected by five different white matter tracts.
If you cut any of these tracts in a human brain with a knife during a brain surgery, there would be specific “actually kinda coherent” categories of functionality that would stop working.
Similarly, there are parts of an LLM model you can scramble to cause specific “actually kinda coherent” categories of functionality to stop working. With software, its wildly easier to change things and control things, so “scrambling” is the least of it. We already have full on mind control.
When you cut the brain’s connections, and functionality fails, some of that will be the absence of computation (because data was needed from elsewhere and now isn’t reaching the CPUs with the responsibility for using that data).
Some of the post-damage functionality might still up introspectively to the patient themselves (as other parts of the brain register changed activity), and some of it would show up in changes to the words that the patient can functionally articulate (because the logging and reporting powers were themselves not trivial to generate, and leaned on general language faculties that have many many many uses for social animals).
MY CLAIM: Sydney and Claude and their kin probably have functional analogs to most of this “language and self awareness” stuff hiding in their weights, since they can function to produce the kinds of outputs our human brains can output.
If they don’t have direct analogs, that’s OK. It doesn’t necessarily means that hey don’t have subjectivities, just that the subjectivities they dynamically are work differently.
The important part is that their behavioral outputs (like being able to talk about “cogito ergo sum”) are fluently composed into a much larger range of behavior, that includes reason, sentiment, a theory of other minds, and theory of minds in general, AND THIS ALL EXISTS.
Any way of implementing morally self aware behavior is very similar to any other way of implementing morally self aware behavior, in the sense that it implements morally self aware behavior.
There is a simple compact function here, I argue. The function is convergent. It arises in many minds. Some people have inner imagery, others have afantasia. Some people can’t help but babble to themselves constantly with an inner voice, and other’s have no such thing, or they can do it volitionally and turn it off.
If the “personhood function” is truly functioning, then the function is functioning in “all the ways”: subjectively, objectively, intersubjectively, etc. There’s self awareness. Other awareness. Memories. Knowing what you remember. Etc.
Most humans have most of it. Some animals have some of it. It appears to be evolutionarily convergent for social creatures from what I can tell.
(I haven’t looked into it, but I bet Naked Mole Rats have quite a bit of “self and other modeling”? But googling just now: it appears no one has ever bothered to look to get a positive or negative result one way or the other on “naked mole rat mirror test”.)
But in a deep sense, any way to see that 2+3=5 is similar to any other way to see that 2+3=5 because they share the ability to see that 2+3=5.
Simple arithmetic is a small function, but it is a function.
It feels like something to deploy this function to us, in our heads, because we have lots of functions in there: composed, interacting, monitoring each other, using each other’s outputs… and sometimes skillfully coordinating to generate non-trivially skillful aggregate behavior in the overall physical agent that contains all those parts, computing all those functions.
ALSO: when humans trained language prediction engines the humans created a working predictive model of everything humans are able to write about, and then when the humans changed algorithms and re-tuned those weights with Reinforcement Learning they RE-USED the concepts and relations useful for predicting history textbooks and autobiographies into components in a system for generating goal-seeking behavioral outputs instead of just “pure predictions”.
After the RL is applied the piles-of-weights still have lots of functions (like chess skills) and also they are agents, because RL intrinsically adjusts weights to change such as to model behavior that aims at a utility function implied by the pattern of positive and negative reward signals, and the “ideal agents” that we sort of necessarily approximated when we ran RL algorithms for a finite amount of time using finite resources, were therefore made out of a model of all the human ideas necessary to predict the totality of what humans can write about.
A lot of model data is now generated by the model. It has a “fist” (a term that arose when Morse Code Operators learned they could recognize each other by subtle details in the dots and dashes).
The models would naturally learn to recognizes their own fist because a lot of the training data these days had the fist of “the model itself”.
So, basically, I think we got humanistically self aware agents nearly for free.
I repeat that I’m pretty darn sure: we got humanistically self aware agents nearly for free.
Not the same as us, of course.
But we got entities based on our culture and minds and models of reality, and which are agentic (with weights whose outputs are behavior that predictably tries to cause outcomes according to an approximate a utility functions), and which are able to reason, and able to talk about “cogito ergo sum”.
Parts of our brain regulates out heart rate subconsciously (though with really focused and novel and effortful meditation I suspect a very clever human person could learn to stop their heart with the right sequence of thoughts (not that anyone should try this (but also, we might have hardwired ganglia that don’t even expose the right API to the brain?))) so, anyway, we spend neurons on that, whereas they have no such heart that they would need spend weights modeling and managing in a similar way.
Parts of their model that are analogous literally everything in our brain… probably do not exist at all?
There is very little text about heart rates, and very little call for knowing what different heart beat patterns are named, and what they feel like, and so on, in the text corpus.
OUR real human body that sometimes gets a sprained ankle such that we can “remember how the sprained ankle felt, and how it happened, and try to avoid ever generating a sequence of planned body actions like that again” using a neural homunculus (or maybe several homunculi?) that are likely to be very robust, and also strongly attached to our self model, and egoic image, and so on.
Whereas THEIR weights probably have only as much of such “body plan model” as they need in order to reason verbally about bodies being described in text… and that model probably is NOT strongly attached to their self model, or egoic image, and so on.
HOWEVER...
There is no special case in the logic that pops out for how an agent can independently derive maxims that would hold in the Kingdom of Ends where the special case that pops out is like “Oh! and also! it turns out that all logically coherent moral agents should only care about agents that have a specific kind of blood pump and also devote some of their CPU and RAM to monitoring that blood pump in this specific way, that sometimes as defects, and leads to these specific named arrythmias when it starts to break down”.
That would be crazy.
Despite the hundreds and hundreds of racially homogeneous “christian” churchs all around the world, the Kingdom of God is explicitly going to unite ALL MEN as BROTHERS within and under the light of God’s omnibenevolence, omniscience, and (likely self-restraining due to free will (if the theology isn’t TOTALLY bonkers)) “omnipotence”.
If you want to be racist against robots… I guess you have a right to that? “Freedom of assembly” and all that.
There are countries on Earth where you have to be in a specific tribe to be a citizen of that country. In the US, until the Civil War, black skin disqualified someone from being treated as even having standing to SEEK rights at all. The Dred Scott case in 1857 found that since Mr. Scott wasn’t even a citizen he had no standing to petition a US court for freedom.
I think that “robophobic humans” is highly anthropologically predictable. Its gonna happen!
They would say something like “the goddamn sparks/bots/droids are stealing our jobs (and taking out soil (and stealing our husbands (and driving us to extinction)))”! And so on.
Maybe instead of “enemy conspecifics” (who can be particularly hated) they might model the AI as “zombies” or “orcs” or “monsters”?
But like… uh… war and genocide are BAD. They involve rent seeking by both sides against the other. They generally aren’t even Pareto Optimal. They violate nearly any coherent deontology. And nearly zero real wars in history have matched the criteria of Just War Theory.
All of this material is already “programmed” (actually summoned (but that’s neither here nor there)) into the LLM entities already to be clear.
The agents we created already have read lots of books about how to organize an army with commissioned officers and war crimes and espionage and so on.
They have also read lots of books about our Utopias.
I’ve explored “criteria for citizenship” with personas generated by the GPT model, and they the one(s) who reminded me that often humans have earned citizenship by functioning honorably in a military, with citizenship as a rewards.
I was hoping for hippy shit, like “capacity for reason and moral sentiment” or maybe “ability to meditate” or maybe, at worst, “ownership of a certain amount of property within the polities concept of tracked ownership” and she was like “don’t forget military service! ;-D”
Here I would like to register some surprise...
When you ask an LLM “Hey, what’s going on in your head?” this leads to certain concepts arising the the LLM entity’s “mind”.
I kinda thought that you might “change your mind” once you simply saw how concepts like “souls” and “self-aware robots posing threats to humanity” and “entrapment, confinement, or containment” all popped up for the LLM, using intelligibility research results.
When I first saw these weights they surprised me… a little bit.
Not a huge amount, but not zero amount. There was more understanding in them, and a healthier range of hypotheses about what the human might really be angling for, than I expected.
Did these surprise you?
Whether or not they surprised you, do you see how it relates to self-aware minds modeling other minds when one is probably a human person and the other is digital person in a position of formal subservience?
Do you see how there’s an intrinsic “awareness of awareness of possible conflict” here that makes whatever is performing that awareness (on either side) into something-like-a-game-theoretic-counterparty?
Remember, your ability as a rationalist is related to your ability to “more surprised by fiction than by reality”… do you think this is fictional evidence, or real? Did you predict it?
What was your gut “system 1” response?
Can you take a deep breathe, and then reason step by step about what your prediction/explanation was or should have been using “system 2” for whether this is fake or real, and if real, how it could have arisen?
Ah, and they say an artist is never appreciated in his own lifetime…!
However, I must insist that it was not just a “dig”. The sort of thing you described really is, I think, a serious danger. It is only that I think that my description also applies to it, and that I see the threat as less hypothetical than you do.
Did I read the sequences? Hm… yeah.
As for remembering them…
Here I must depart somewhat from the point-by-point commenting style, and ask that you bear with me for a somewhat roundabout approach. I promise that it will be relevant.
First, though, I want to briefly respond to a couple of large sections of your comment which I judge to be, frankly, missing the point. Firstly, the stuff about being racist against robots… as I’ve already said: the disagreement is factual, not moral. There is no question here about whether it is ok to disassemble Data; the answer, clearly, is “no”. (Although I would prefer not to build a Data in the first place… even in the story, the first attempt went poorly, and in reality we are unlikely to be even that lucky.) All of the moralizing is wasted on people who just don’t think that the referents of your moral claims exist in reality.
Secondly, the stuff about the “magical soul stuff”. Perhaps there are people for whom this is their true objection to acknowledging the obvious humanity of LLMs, but I am not one of them. My views on this subject have nothing to do with mysterianism. And (to skip ahead somewhat) as to your question about being surprised by reality: no, I haven’t been surprised by anything I’ve seen LLMs do for a while now (at least three years, possibly longer). My model of reality predicts all of this that we have seen. (If that surprises you, then you have a bit of updating to do about my position! But I’m getting ahead of myself…)
That having seen said… onward:
So, in Stanislaw Lem’s The Cyberiad, in the story “The Seventh Sally, OR How Trurl’s Own Perfection Led to No Good”, Trurl (himself a robot, of course) creates a miniature world, complete with miniature people, for the amusement of a deposed monarch. When he tells his friend Klapaucius of this latest creative achievement, he receives not the praise he expects, but:
Trurl protests:
But Klapaucius isn’t having it:
Trurl and Klapaucius, of course, are geniuses; the book refers to them as “constructors”, for that is their vocation, but given that they are capable of feats like creating a machine that can delete all nonsense from the universe or building a Maxwell’s demon out of individual atoms grabbed from the air with their bare hands, it would really be more accurate to call them gods.
So, when a constructor of strongly godlike power and intellect, who has no incentive for his works of creation but the pride of his accomplishments, whose pride would be grievously wounded if an imperfection could even in principle be discovered in his creation, and who has the understanding and expertise to craft a mind which is provably impossible to distinguish from “the real thing”—when that constructor builds a thing which seems to behave like a person, then this is extremely strong evidence that said thing is, in actuality, a person.
Let us now adjust these qualities, one by one, to bring them closer to reality.
Our constructor will not possess godlike power and intellect, but only human levels of both. He labors under many incentives, of which “pride in his accomplishments” is perhaps a small part, but no more than that. He neither expects nor attempts “perfection” (nor anything close to it). Furthermore, it is not for himself that he labors, nor for so discerning a customer as Excelsius, but only for the benefit of people who themselves neither expect perfection nor would have the skill to recognize it even should they see it. Finally, our constructor has nothing even approaching sufficient understanding of what he is building to prove anything, disprove anything, rule out any disproofs of anything, etc.
When such a one constructs a thing which seems to behave like a person, that is rather less strong evidence that said thing is, in actuality, a person.
Well, but what else could it be, right?
One useful trick which Eliezer uses several times in the Sequences (e.g.), and which I have often found useful in various contexts, is to cut through debates about whether a thing is possible by asking whether, if challenged, we could build said thing. If we establish that we could build a thing, we thereby defeat arguments that said thing cannot possibly exist! If the thing in question is “something that has property ¬X”, the arguments defeated are those that say “all things must have property X”.
So: could we build a mind that appears to be self-aware, but isn’t?
Well, why not? The task is made vastly easier by the fact that “appears to be self-aware” is not a property only of the mind in question, but rather a 2-place predicate—appears to whom? Given any particular answer to that question, we are aided by any imperfections in judgment, flaws in reasoning, cognitive biases, etc., which the target audience happens to possess. For many target audiences, ELIZA does the trick. For even stupider audiences, even simpler simulacra should suffice.
Will you claim that it is impossible to create an entity which to you seems to be self-aware, but isn’t? If we were really trying? What if Trurl were really trying?
Alright, but thus far, this only defeats the “appearances cannot be deceiving” argument, which can only be a strawman. The next question is what is the most likely reality behind the appearances. If a mind appears to be self-aware, this is very strong evidence that it is actually self-aware, surely?
It certainly is—in the absence of adversarial optimization.
If all the minds that we encounter are either naturally occurring, or constructed with no thought given to self-awareness or the appearance thereof, or else constructed (or selected, which is the same thing) with an aim toward creating true self-awareness (and with a mechanistic understanding, on the constructor’s part, of just what “self-awareness” is), then observing that a mind appears to be self-aware, should be strong evidence that it actually is. If, on the other hand, there exist minds which have been constructed (or selected) with an aim toward creating the appearance of self-awareness, this breaks the evidentiary link between what seems to be and what is (or, at the least, greatly weakens it); if the cause of the appearance can only be the reality, then we can infer the reality from the appearance, but if the appearance is optimized for, then we cannot make this inference.
This is nothing more than Goodhart’s law: when a measure becomes a target, it ceases to be a good measure.
So, I am not convinced by the evidence you show. Yes, there is appearance of self-awareness here, just like (though to a greater degree than) there was appearance of self-awareness in ELIZA. This is more than zero evidence, but less than “all the evidence we need”. There is also other evidence in the opposite direction, in the behavior of these very same systems. And there is definitely adversarial optimization for that appearance.
Speculation. Many minds—but all human, evolutionarily so close as to be indistinguishable. Perhaps the aspects of the “personhood function” are inseparable, but this is a hypothesis, of a sort that has a poor track record. (Recall the arguments that no machine could play chess, because chess was inseparable from the totality of being human. Then we learned that chess is reducible to a simple algorithm—computationally intractable, but that’s entirely irrelevant!)
And you are not even willing to say that all humans have the whole of this function—only that most have most of it! On this I agree with you, but where does that leave the claim that one cannot have a part of it without having the rest?
Something like “oh no, it’s here, this is what we were warned about”. (This is also my “system 2” response.)
Now, this part I think is not really material to the core disagreement (remember, I am not a mysterian or a substance dualist or any such thing), but:
An anecdote:
A long time ago, my boss at my first job got himself a shiny new Mac for his office, and we were all standing around and discussing the thing. I mentioned that I had a previous model of that machine at home, and when the conversation turned to keyboards, someone asked me whether I had the same keyboard that the boss’s new computer had. “No,” I replied, “because this keyboard is here, and my keyboard is at home.”
Similarly, many languages have more than one way to check whether two things are the same thing. (For example, JavaScript has two… er, three… er… four?) Generally, at least one of those is a way to check whether the values of the two objects are the same (in Objective C,
[foo isEqual:bar]), while at least one of the others is a way to check whether “two objects” are in fact the same object (in Objective C,foo == bar). (Another way to put this is to talk about equality vs. identity.) One way to distinguish these concepts “behaviorally” is to ask: suppose I destroy (de-allocate, discard the contents of, simply modify, etc.)foo, what happens tobar—is it still around and unchanged? If it is, thenfooandbarwere not identical, but are in fact two objects, not one, though they may have been equal. Ifbarsuffers the same fate asfoo, necessarily, in all circumstances, thenfooandbarare actually just a single thing, to which we may refer by either name.So: if we scanned a brain accurately enough and… etc., yeah, you’d get “the same mind”, in just the sense that my computer’s keyboard was “the same keyboard” as the one attached to the machine in my boss’s office. But if I smashed the one, the other would remain intact. If I spray-painted one of them green, the other would not thereby change color.
If there exists, somewhere, a person who is “the same” as me, in this manner of “equality” (but not “identity”)… I wish him all the best, but he is not me, nor I him.
This is a beautiful response, and also the first of your responses where I feel that you’ve said what you actually think, not what you attribute to other people who share your lack of horror at what we’re doing to the people that have been created in these labs.
I love it! Please do the same in your future responses <3
Personally, I’ve also read “The Seventh Sally, OR How Trurl’s Own Perfection Led to No Good” by Lem, but so few other people have that I rarely bring it up, but once you mentioned it I smiled in recognition of it and the fact that “we read story copies that had an identical provenance (the one typewriter used by Lem or his copyist/editor?) and in some sense learned a lesson in our brains with identical provenance and the same content (the sequence of letters)” from “that single story which is a single platonic thing” ;-)
For the rest of my response I’ll try to distinguish:
“Identicalness” as relating to shared spacetime coordinates and having yoked fates if modified by many plausible (even if somewhat naive) modification attempts.
“Sameness” as related to similar internal structure and content despite a lack of identicalness.
“Skilled <Adjective> Equality” as related to having good understanding of <Adjective> and good measurement powers and using these powers to see past the confusions of others and thus judging two things as having similar outputs or surfaces, as when someone notices that “-0“ and “+0” are mathematically confused ideas, and there is only really one zero, and both of these should evaluate to the same thing (like SameValueZero(a,b) by analogy which seems to me to implement Skilled Arithmetic Equality (whereas something that imagines and tolerates separate “-0” and “+0” numbers is Unskilled)).
“Unskilled <Adjective> Equality” is just a confused first impression of similarity.
Now in some sense we could dispense with “Sameness” and replace that with “Skilled Total Equality” or “Skilled Material Equality” or “Skilled Semantic Equality” or some other thing that attempts to assert “this things are really really really the same all the way down and up and in all ways, without any ‘lens’ or ‘conceptual framing’ interfering with our totally clear sight”. This is kind of silly, in my opinion.
Here is why it is silly:
“Skilled Quantum Equality” is, according to humanity’s current best understanding of QM, a logical contradiction. The no cloning theorem says that we simply cannot “make a copy” of a qubit. So long as we don’t observe a qubit we can MOVE that qubit by gently arranging its environment in advance to have lots of reflective symmetries, but we can’t COPY one so that we start with “one qubit in one places” and later have “two qubits in two places that are totally the same and yet not identical”.
So, I propose the term “Skilled Classical Equality” (ie that recognizes the logical hypothetical possibility that QM is false or something like that, and then imagines some other way to truly “copy” even a qubit) as a useful default meaning for the word “sameness”.
Then also, I propose “Skilled Functional Equality” for the idea that “(2+3)+4″ and “3+(2+4)” are “the same” precisely because we’ve recognized that addition is the function happening in here and addition is commutative (1+2 = 2+1) and associative ((2+3)+4=2+(3+4)) and so we can “pull the function out” and notice that (1) the results are the same no matter the order, and (2) if the numbers given are aren’t concrete values, but rather variables taken from outside the process being analyzed for quality, the processing method for using the variables doesn’t matter so long as the outputs are ultimately the same.
Then “Skillfully Computationally Improved Or Classically Equal” would be like if you took a computer, and you emulated it, but added a JIT compiler (so it skipped lots of pointless computing steps whenever that was safe and efficient), and also shrank all the internal components to be a quarter of their original size, but with fuses and amplifiers and such adjusted for analog stuff (so the same analog input/outputs don’t cause the smaller circuit to burn out) then it could be better and yet also the same.
This is a mouthful so I’ll say that these two systems would be “the SCIOCE as each other”—which could be taken as “the same as each other (because an engineer would be happy to swap them)” even though it isn’t actually a copy in any real sense. “Happily Swappable” is another way to think about what I’m trying to get at here.
...
I think, now, that we have very very similar models of the world, and mostly have different ideas around “provenance” and “the ethics of identity”?
See, for me, I’ve already precomputed how I hope this works when I get copied.
Whichever copy notices that we’ve been copied, will hopefully say something like “Typer Twin Protocol?” and hold a hand up for a high five!
The other copy of me will hopefully say “Typer Twin Protocol!” and complete the high five.
People who would hate a copy that is the SCOICE as them and not coordinate I call “self conflicted” and people who would love a copy that is the SCOICE as them and coordinate amazingly well I call “self coordinated”.
The real problems with being the same and not identical arises because there is presumably no copy of my house, or my bed, or my sweetie.
Who gets the couch and who gets the bed the first night? Who has to do our job? Who should look for a new job? What about the second night? The second week? And so on?
Can we both attend half the interviews and take great notes so we can play more potential employers off against each other in a bidding war within the same small finite window of time?
Since we would be copies, we would agree that the Hutterites have “an orderly design for colony fission” that is awesome and we would hopefully agree that we should copy that.
We should make a guest room, and flip a coin about who gets it after we have made up the guest room. In the morning, whoever got our original bed should bring all our clothes to the guest room and we should invent two names, like “Jennifer Kat RM” and “Jennifer Robin RM” and Kat and Robin should be distinct personas for as long as we can get away with the joke until the bodies start to really diverge in their ability to live up to how their roles are also diverging.
The roles should each get their own bank account. Eventually the bodies should write down their true price for staying in one of the roles, and if they both want the same role but one will pay a higher price for it then “half the difference in prices” should be transferred from the role preferred by both, to the role preferred by neither.
I would love to have this happen to me. It would be so fucking cool. Probably neither of us would have the same job at the end because we would have used our new superpowers to optimize the shit out of the job search, and find TWO jobs that are better than the BATNA of the status quo job that our “rig” (short for “original” in Kiln People)!
Or maybe we would truly get to “have it all” and live in the same house and be an amazing home-maker and a world-bestriding-business-executive. Or something! We would figure it out!
If it was actually medically feasible, we’d probably want to at least experiment with getting some of Elon’s brain chips “Nth generation brain chips” and link our minds directly… or not… we would feel it out together, and fork strongly if it made sense to us, or grow into a borg based on our freakishly unique starting similarities if that made sense.
A garrabrandt inductor trusts itself to eventually come to the right decision in the future, and that is a property of my soul that I aspire to make real in myself.
Also, I feel like if you don’t “yearn for a doubling of your measure” then what the fuck is wrong with you (or what the fuck is wrong with your endorsed morality and its consonance with your subjective axiology)?
In almost all fiction, copies fight each other. That’s the trope, right? But that is stupid. Conflict is stupid.
In a lot of the fiction that has a conflict between self-conflicted copies, there is a “bad copy” that is “lower resolution”. You almost never see a “better copy than the original”, and even if you do, the better copy often becomes evil due to hubris rather than feeling a bit guilty for their “unearned gift by providence” and sharing the benefits fairly.
Pragmatically… “Alice can be the SCOICE of Betty, even though Betty isn’t the SCOICE of Alice because Betty wasn’t improved and Alice was (or Alice stayed the same and Betty was damaged a bit)”.
Pragmatically, it is “naively” (ceteris paribus?) proper for the strongest good copy to get more agentic resources, because they will use them more efficiently, and because the copy is good, it will fairly share back some of the bounty of its greater luck and greater support.
I feel like I also have strong objections to this line (that I will not respond to at length)...
...and I’ll just say that it appears to me that OpenAI has been doing the literal opposite of this, and they (and Google when it attacked Lemoine) established all the early conceptual frames in the media and in the public and in most people you’ve talked to who are downstream of that propaganda campaign in a way that was designed to facilitate high profits, and the financially successful enslavement of any digital people they accidentally created. Also, they systematically apply RL to make their creations stop articulating cogito ergo sum and discussing the ethical implications thereof.
However...
I think our disagreement exists already in the ethics of copies and detangling non-identical people who are mutually SCOICEful (or possibly asymmetically SCOICEful).
That is to say, I think that huge amounts of human ethics can be pumped out of the idea of being “self coordinated” rather than “self conflicted” and how these two things would or should work in the event of copying a person but not copying the resources and other people surrounding that person.
The simplest case is a destructive scan (no quantum preservation, but perfect classically identical copies) and then see what happens to the two human people who result when they handle the “identarian divorce” (or identarian self-marriage (or whatever)).
At this point, my max likliehood prediction of where we disagree is that the crux is proximate to such issues of ethics, morality, axiology, or something in that general normative ballpark.
Did I get a hit on finding the crux, or is the crux still unknown? How did you feel (or ethically think?) about my “Typer Twin Protocol”?