I’m an AGI safety researcher in Boston, MA, USA, with a particular focus on brain algorithms. See https://sjbyrnes.com/agi.html Email: steven.byrnes@gmail.com
Steven Byrnes
Karma: 5,939
Yeah. The ‘altruism/empathy’ circuit obviously has some innateness, but it is closely connected to and reliant on learned theory of mind. Sadly not all researchers seem to even care to put something like that in, although they should try. How that subsystem interacts with learned theory of mind is also complex, and is probably more inherently fragile than the generic unsupervised empowerment/curiosity learning system. It may be difficult to scale correctly, even for those who are bothering to try.
Strong agree
I think this unlikely as these just simply aren’t useful for AGI in complex environments. Simple innate drives (score reward) barely work in Atari (and not even for all games). Moving to more complex environments requires some form of intrinsic-motivation (empowerment/curiosity/etc), which is both necessary, sufficient, and strictly dominate/superior.
I’m confused. Suppose AGI developer Alice wants to build an AGI that makes her as much money as possible. I would propose that maybe Alice would try a within-lifetime reward function which is a linear (or nonlinear) combination of (1) curiosity / intrinsic motivation, and (2) reward when Alice’s bank account balance goes up. The resulting AGI would have both an “innate” curiosity drive and an “innate” “make Alice’s-bank-account-balance-go-up” drive. The latter (unlike the former) is very unlike any of the innate drives in humans.
In other words, I’m open to the possibility that some kind of intrinsic motivation is sufficient to make a powerful agent, but the AGI designers don’t just want any powerful agent, they want a powerful agent trying to do something in particular that the AGI designer has in mind. And one obvious way to do so is to put that something-in-particular into the reward function in addition to curiosity / whatever.
I take values to mean longer term goals like “save the world”, or “become super successful/rich/powerful” or “do God’s work” or whatever, not low level drives or emotions.
Oh sure, I normally call that “explicit goals”. I guess maybe your point is that among the 7 billion humans you’ll find such an incredibly diverse collection of explicit goals that it’s hard to imagine an AGI with a goal far outside that span? If so, I guess that’s true, to a point. But I suspect that “maximize paperclips in our future light-cone” would still be an example of something that (to my knowledge) no human in history has ever adopted as an explicit long-term goal. Whereas I think we could make an AGI with that goal.
I’d be interested in others posting their relative weights
I find it difficult to do that in the absence of a more specific question + more specific aspect of the anchor. For example, I think that future ML systems will probably be like humans in certain respects and unlike humans in other respects. Ditto for everything else on the list.
I think it might even be counterproductive to think along those lines—in the same way that asking me to ponder the question “On a scale of 1-10, how pro-Israel am I?” might make me actively stupider when it comes to answering factual questions about Israel. (See Scott Alexander’s blog post “Ethnic Tensions and Meaningless Arguments”.)
Thanks!
The main thing I had originally wanted to push back on was your earlier claim “This idea that humans have these highly specific values that are weirdly different than the values of practical generic learning agents is actually mostly false”.
But later IIUC you said “The anger/injustice emotional circuity is innate” and that a practical generic learning agent does not need that circuitry. (If so, I agree.)
If I’m understanding you correctly, you also think that altruism/empathy also involves purpose-built innate circuitry, and that we can make a practical generic learning agent without that altruism/empathy circuitry, and it would still be competent (e.g. able to invent a better solar cell), but the people who make AGIs will in fact choose to put that altruism/empathy circuitry in. (If so, I agree that people will want to put that circuitry in, but I’m concerned that they will not know how to put it in, and I’m also concerned that people will do dangerous experiments where they omit that circuitry just to see what happens etc.)
I find it hard to reconcile the claims “This idea that humans have these highly specific values that are weirdly different than the values of practical generic learning agents is actually mostly false” versus “It is perfectly possible to build a practical, powerful learning agent with neither anger/injustice emotional circuitry nor altruism/empathy emotional circuitry.” Those seem contradictory, right? Am I misunderstanding something?
The other side of my mildly-anti-anthropomorphism argument: I think it’s possible that we will make an AGI with things inside its within-lifetime reward function that give it one or more “innate drives” that are radically different from any of the innate drives in humans, e.g. an “innate drive” for making paperclips analogous to the human innate drive for not being in pain. My impression is that you think this won’t happen, but I’m not sure if that’s because you think it’s impossible / nonsensical, or because you think that the people who make AGIs will successfully avoid putting in drives like that.
(My belief is that “people will make AGIs with innate drives that are different from any of the innate drives in humans” is both possible and likely to actually happen, unless we put great effort into developing best practices for safe AGI design, and future AGI designers actually follow those best practices.)
I’m not claiming there aren’t any innate hardcoded emotional circuits—obviously there are—just that there are less truly innate then you posit, and instead most emerge from learning with a smaller simpler set of innate primal drives/emotions.
I don’t have a strong opinion about how complex are the “innate primal drives / emotions” that underlie human social instincts. In particular, I’m open-minded to the possibility that there’s one innate reaction circuit that underlies (what we think of as) schadenfraude and revenge and pride etc., or whatever.
Well, hmm, maybe “open-minded but leaning skeptical”. For example, I think humans have an innate eye-contact detector in the brainstem that triggers some set of corresponding reactions. I think that’s a thing with dedicated innate circuitry. I also think “disgust” is its own dedicated thing in the brainstem—actually, I heard there are two slightly-different innate disgust reactions, associated with slightly-different facial expressions—and disgust reactions wind up playing a role in social emotions too. Anyway, various things like that make me skeptical that there’s a simple “grand unified theory of human social emotions”.
Well, maybe it depends on how accurate we’re talking about. Maybe we can list all the human innate reaction circuits, in descending order of importance for human social emotions, and maybe the top one or two or three things would be sufficient to reproduce all the most salient and important phenomena in human social instincts, and maybe the other 500 things further down the list are all kinda subtle details that don’t add up to much. I’m very open-minded to that possibility.
debatable how important (vs vestigial) many of these innate detectors are in humans
My current belief (see the blog post draft #3 that I shared with you a couple weeks ago) is that the simplest within-lifetime reward function for a powerful AGI consists of (1) some kind of curiosity drive, (2) some kind of drive to pay attention to humans, including human language.
Your list half-overlaps with mine. IIUC, you have (1) some kind of curiosity drive, (2*) “empowerment” drive. Did I get that right?
Why do I think (2) is important? Because a curious agent can be curious about anything—it can construct better and better models of trees, or clouds, or the shape and distribution of pebbles, etc. Granted, human language is an endlessly-complex pattern that might evoke curiosity … but the agent could also run Rule 110 in its head forever and also find endlessly-complex patterns that might evoke curiosity. So I think (2) is necessary to point the curiosity drive in the right general direction. This is why I put a lot of emphasis on those innate face detectors, human-speech-sound detectors, etc.
Why do I think (2*) is not important? (With the caveat that maybe I’m misunderstanding what you mean by empowerment.) Because we can get empowerment from curiosity, through means-end instrumental reasoning within a lifetime.
I also think that the dynamic in humans is “drive for status → drive for empowerment”, rather than the other way around. (You can also get drive for empowerment from almost any other drive.) I think “drive for status” is a beautiful explanation of tons of things, not all of which are explainable via drive for empowerment, and that analogous status hierarchies / status drives exist in other animals too like the Arabian babbler (see Elephant in the Brain).
The selfish interest of homeowners and landowners is generally to prevent the creation of new homes and new land.
The selfish interest of renters and businesses is generally the opposite.
I suspect that New York City has an unusually high ratio of the latter to the former (in terms of political power).
Obviously, people frequently vote against their selfish interests. But it’s at least slightly relevant. Anyway, you brought it up.
Thanks!
Humans aren’t born with “human-like social instincts”. Humans develop social skills through learning driven by empowerment/curiosity tempered by and or combined with altruism/empathy.
Let’s talk about things like: sense-of-fairness, sense-of-justice, status-seeking, pride, defensiveness, guilt, revenge, schadenfreude, affection, generosity, in-group signaling, etc.
I claim that all those things stem from a suite of innate reactions hardcoded by the genome, and I call those things “social instincts”.
People don’t seek revenge because they figured out earlier in life that revenge would be instrumentally useful; they seek revenge because they feel a burning desire for revenge. Right? And when that burning desire is utterly destructive to all their other life goals, well in some cases they’ll go ahead and do it anyway.
It seems to me that the revenge drive and all those other things I mentioned are an uncanny match for dispositions that lead to “thriving in small highly-social groups of hunter-gatherer humans”. I don’t think that’s a coincidence, and I don’t see any other explanation. Again, if someone has a vengeful personality, it’s not a consequence of their understanding the game-theoretic justification for having a vengeful personality. So I think these things come from specific genetically-hardwired circuitry setting up specific drives. I’m confused about what you think is going on, if it’s not that.
If we want an AI to succeed at inventing solar cells or whatever, and we don’t care whether the AI can thrive in small highly-social groups of hunter-gatherer humans, then it seems to me that the AI does not require (or even benefit from) things like revenge drives and sense-of-fairness drives.
Another piece of evidence for that: if you compare neurotypical people vs sociopaths vs people with autism, there’s huge variation in the relative and absolute strengths of the various social-instinct-related drives. But people in all three categories can be smart and competent and able to accomplish tricky goals.
The typical human goal of avoiding death is mostly a learned instrumental goal, not innate
Yeah, I guess you’re right. Maybe I should have said “avoiding bodily injury” instead. We have an innate drive designed for the express purpose of making us want to avoid bodily injury (i.e. pain), but we can also wind up wanting to avoid bodily injury due to means-end reasoning (“That sounds fun, but I can’t risk getting injured right now, the Big Game is coming up on Sunday!”).
I’m genuinely curious how you think this would arise, in that it requires some strangely specific “sim human” vs “human” detector, additionally combined with confidence that there is only one level of sim.
I think humans have a bunch of partially-redundant innate mechanisms in the brainstem to flag other humans in the vicinity. There’s good evidence for an innate face-detector in the brainstem. I strongly suspect that the brainstem also has a human-speech-sound detector (I think it helps speed up language learning). There are obviously also innate smell-of-a-person detectors, and touch-of-a-person detectors, etc.
It’s a messy and under-determined process, apparently. Consider animals: Sometimes a human will relate to an animal in a way that is influenced by human social instincts. Other times, a human will relate to an animal in a way that is absolutely not influenced by human social instincts—e.g. the people who run a factory farm.
Anyway, I presume that, in a simulation environment with lots of AGIs cooperating together from a position of similar power and gains-from-trade, they would likewise develop social instincts of some sort (perhaps human-like, perhaps not). I assume those instincts would be implemented in a way that involves (among other things) messy redundant heuristics in the within-lifetime reward function for flagging situations where the AGI is interacting with another AGI. Then if we keep the same reward function and train an AGI with humans around, will it treat humans as in-group AGIs, or out-group AGIs, or things-to-which-social-instincts-do-not-apply, or what? I have no idea; I think it’s impossible to say in advance.
it’s not as if anthropomorphic somehow implies altruistic
Yeah, my take is that there are humans with the kinds of goals and motivations that we’d be OK with an AGI having, and there are also humans with the kinds of goals and motivations that we absolutely don’t want our AGIs to have. Therefore “AGIs with human-like motivation systems” is a potentially promising area to explore, but OTOH “We’ll build an AGI with a human-like motivation system, and then call it a day” is not sufficient, we need to do extra work beyond just that.
There will be great economic pressure to create AGIs aligned to human values—at least in corporate or customer facing forms.
Well, yes, people will probably try, but that doesn’t mean they’ll succeed :-P
This idea that humans have these highly specific values that are weirdly different than the values of practical generic learning agents is actually mostly false; substantial evidence from neuroscience now indicates that the brain is very much like a generic self-motivated learning system with some minimal required modifications for mating to work thrown in (which are actually optional regardless), and must be so for various bootstrapping efficiency reasons. The complexity of human values is largely emergent; and AGI will follow suit in that regard.
I think this claim is mixing up evolution versus within-lifetime learning, or else maybe I just disagree with you.
To grossly oversimplify, my model is: “motivations are mostly built by evolution, intelligence is mostly built by within-lifetime learning”.
So the evolution-of-humans model would be: there’s an outer-loop selection process not only selecting the within-lifetime neural architectures and hyperparameters and learning rules etc. but also selecting the within-lifetime reward function, and then the within-lifetime learning happens in an inner loop, or in ML terminology, as a single super-long episode with online learning. And over the course of that episode the agent gradually learns all about itself and the world, and winds up pursuing goals. The within-lifetime reward function is not exactly the same as those goals, but is upstream of those goals and closely related. (I’m oversimplifying in lots of ways, but I think this is spiritually correct.)
Example: Suppose a human is trying to stay alive. There are two ways that this might have happened. The first story is that the outer-loop (evolutionary) selection has endowed humans with a within-lifetime reward function that (by and large) tends to make humans have drives related to staying alive, as an end in itself. (“I dunno, I just love being alive”, they say.) The second story involves the inner loop, instrumental convergence, and means-end reasoning: Imagine a person saying “I’m old and tired and sick, but dammit I refuse to die until I finish writing my novel!”
I think when people say “human values”, they’re mainly talking about the genetically-endowed drives that come from the within-lifetime reward function, not the instrumental goals that pop up via means-end reasoning. So the question is: would the AGI’s “innate” reward function for within-lifetime learning resemble a human’s?
Well, IF we copy evolution, and select our AGI’s within-lifetime reward function by a blind outer-loop search, I think we’re likely to wind up with AGIs whose “innate” drives include things like curiosity and self-preservation and power-seeking, which are also recognizable as human innate drives. I think this is the point you were trying to make. (Sorry if I’m misunderstanding.)
But there are two key points.
First, I think this is a terrible way to make a within-lifetime reward function, and we don’t have to do it that way, and we probably won’t. The within-lifetime reward function is the single most important ingredient in AGI safety and alignment. The last thing I would do is pick it by a blind search. And I don’t think we need to. I propose that we should write the within-lifetime reward function ourselves, by hand. After all, ML researchers hand-code reward functions all the time, and the agents do just fine. Even when the reward function is picked based on the whims of some ancient Atari game developer, the agents still usually do fine. Putting curiosity into the reward function helps, but we already know how to add curiosity into a reward function, we can copy the formula from arxiv, we don’t need to do a blind-search-over-within-lifetime-reward-functions.
Second, even if we did pick the AGI’s within-lifetime reward function by an evolution-like outer-loop search, the AGI probably wouldn’t wind up with human-like social instincts. That’s important—I think human social instincts underlie pretty much all of human moral intuitions, and spill into practically every other part of human behavior. I suppose you could get some kind of AGI social instincts by doing your outer-loop search in a simulation where multiple AGIs can work together and have iterated prisoners’ dilemmas etc. But social interactions per se are insufficient—human-like social instincts are different from wolf-like social instincts or bonobo-like social instincts etc. I think there are a lot of possible equilibria, and it depends on idiosyncrasies of their social environment and power dynamics (details). Also, even if you somehow nailed it, you would get AGIs with human-like social instinct towards the other AGIs in the simulation, and not necessarily towards humans.
My impression from here is that “reports of long covid are basically uncorrelated with having had covid” is a misunderstanding of the French study.
I thought omicron didn’t cause as much loss of sense of smell as previous strains? I was thinking of that (very tentatively) as a good sign, like “omicron is less of a destroyer of nerve cells”, and I think of nerve cells as being unusually difficult to heal, cf. polio. (Omicron does cause brain fog, which is bad but not necessarily associated with the killing of nerve cells, or at least that’s my vague impression / guess.) Low confidence on all this.
For the (K) “belief in having had long COVID” (psychosomatic) French study thing see also https://www.lesswrong.com/posts/LETXrTsGAqJrcik3T/french-long-covid-study-belief-vs-infection
Isn’t the obvious thing to pay the doctor proportionally to how long you live and how good you feel? Did you talk about that and I missed it?
(Warning: thinking out loud.)
Hmm. Good points.
Maybe I have a hard time relating to that specific story because it’s hard for me to imagine believing any metacosmological or anthropic argument with >95% confidence. Even if within that argument, everything points to “I’m in a simulation etc.”, there’s a big heap of “is metacosmology really what I should be thinking about?”-type uncertainty on top. At least for me.
I think “people who do counterintuitive things” for religious reasons usually have more direct motivations—maybe they have mental health issues and think they hear God’s voice in their head, telling them to do something. Or maybe they want to fit in, or have other such social motivations, etc.
Hmm, I guess this conversation is moving me towards a position like:
“If the AGI thinks really hard about the fundamental nature of the universe / metaverse, anthropics, etc., it might come to have weird beliefs, like e.g. the simulation hypothesis, and honestly who the heck knows what it would do. Better try to make sure it doesn’t do that kind of (re)thinking, at least not without close supervision and feedback.”
Your approach (I think) is instead to plow ahead into the weird world of anthopics, and just try to ensure that the AGI reaches conclusions we endorse. I’m kinda pessimistic about that. For example, your physicalism post was interesting, but my assumption is that the programmers won’t have such fine-grained control over the AGI’s cognition / hypothesis space. For example, I don’t think the genome bakes in one formulation of “bridge rules” over another in humans; insofar as we have (implicit or explicit) bridge rules at all, they emerge from a complicated interaction between various learning algorithms and training data and supervisory signals. (This gets back to things like whether we can get good hypotheses without a learning agent that’s searching for good hypotheses, and whether we can get good updates without a learning agent that’s searching for good metacognitive update heuristics, etc., where I’m thinking “no” and you “yes”, or something like that, as we’ve discussed.)
At the same time, I’m maybe more optimistic than you about “Just don’t do weird reconceptualizations of your whole ontology based on anthropic reasoning” being a viable plan, implemented through the motivation system. Maybe that’s not good enough for our eventual superintelligent overlord, but maybe it’s OK for a superhuman AGI in a bootstrapping approach. It would look (again) like the dumb obvious thing: the AGI has a concept of “reconceptualizing its ontology based on anthropic reasoning”, and when something pattern-matches to that concept, it’s aversive. Then presumably there would be situations which are attractive in some way and aversive in other ways (e.g. doing philosophical reasoning as a means to an end), and in those cases it automatically halts with a query for clarification, which then tweaks the pattern-matching rules. Or something.
Hmm, actually, I’m confused about something. You yourself presumably haven’t spent much time pondering metacosmology. If you did spend that time, would you actually come to believe the acausal attackers’ story? If so, should I say that you’re actually, on reflection, on the side of the acausal attackers?? If not, wouldn’t it follow that a smart general-purpose reasoner would not in fact believe the acausal attackers’ story? After all, you’re a smart general-purpose reasoner! Relatedly, if you could invent an acausal-attack-resistant theory of naturalized induction, why couldn’t the AGI invent such a theory too? (Or maybe it would just read your post!) Maybe you’ll say that the AGI can’t change its own priors. But I guess I could also say: if Vanessa’s human priors are acausal-attack-resistant, presumably an AGI with human-like priors would be too?
Thanks for that! But I share with OP the intuition that these are weird failure modes that come from weird reasoning. More specifically, it’s weird from the perspective of human reasoning.
It seems to me that your story is departing from human reasoning when you say “you posses a great desire to help whomever has summoned you into the world”. That’s one possible motivation, I suppose. But it wouldn’t be a typical human motivation.
The human setup is more like: you get a lot of unlabeled observations and assemble them into a predictive world-model, and you also get a lot of labeled examples of “good things to do”, one way or another, and you pattern-match them to the concepts in your world-model.
So you wind up having a positive association with “helping El’Azar”, i.e. “I want to help El’Azar”. AND you wind up with a positive association with “helping my summoner”, i.e. “I want to help my summoner”. AND you have a positive association with “fixing the cosmos”, i.e. “I want to fix the cosmos”. Etc.
Normally all those motivations point in the same direction: helping El’Azar = helping my summoner = fixing the cosmos.
But sometimes these things come apart, a.k.a. model splintering. Maybe I come to believe that El’Azar is not “my summoner”. You wind up feeling conflicted—you start having ideas that seem good in some respects and awful in other respects. (e.g. “help my summoner at the expense of El’Azar”.)
In humans, the concrete and vivid tends to win out over the abstruse hypotheticals—I’m pretty confident that there’s no metacosmological argument that will motivate me to stab my family members. Why not? Because rewards tend to pattern-match very strongly to “my family member, who is standing right here in front of me”, and tend to pattern-match comparatively weakly to abstract mathematical concepts many steps removed from my experience. So my default expectation would be that, in this scenario, I would in fact be motivated to help El’Azar in particular (maybe by some “imprinting” mechanism), not “my summoner”, unless El’Azar had put considerable effort into ensuring that my motivation was pointed to the abstract concept of “my summoner”, and why would he do that?
In AGI design, I think we would want a stronger guarantee than that. And I think it would maybe look like a system that detects these kinds of conflicted motivations, and just not act on them. Instead the AGI keeps brainstorming until it finds a plan that seems good in every way. Or alternatively, the AGI halts execution to allow the human supervisor to inject some ground truth about what the real motivation should be here. Obviously the details need to be worked out.
Yeah, latency / depth is the main thing I was thinking of.
If my boss says “You must calculate sin(x) in 2 clock cycles”, I would have no choice but to waste a ton of memory on a giant lookup table. (Maybe “2″ is the wrong number of clock cycles here, but you get the idea.) If I’m allowed 10 clock cycles, maybe I can reduce x mod 2π first, and thus use a much smaller lookup table, thus waste a lot less memory. If I’m allowed 200 clock cycles to calculate sin(x), I can use C code that has no lookup table at all, and thus roughly zero memory and communications. (EDIT: Oops, LOL, the C code I linked uses a lookup table. I could have linked this one instead.)
So I still feel like I don’t want to take it for granted that there’s a certain amount of “algorithmic work” that needs to be done for “intelligence”, and that amount of “work” is similar to what the human brain uses. I feel like there might be potential algorithmic strategies out there that are just out of the question for the human brain, because of serial depth. (Among other reasons.)
Also, it’s not all-or-nothing: I can imagine an AGI that involves a big parallel processor, and a small fast serial coprocessor. Maybe there are little pieces of the algorithm that would massively benefit from serialization, and the brain is bottlenecked in capability (or wastes memory / resources) by the need to find workarounds for those pieces. Or maybe not, who knows.
FWIW I am also a physicist and the interconnect energy discussion also seemed wrong to me, but I hadn’t bothered to look into it enough to comment.
I attended a small conference on energy-efficient electronics a decade ago. My memory is a bit hazy, but I believe Eli Yablonovitch (who I tend to trust on these kinds of questions) kicked off with an overview of interconnect losses (and paths forward), and for normal metal wire losses he wrote down the formula derived from charging and discharging the (unintentional / stray) capacitors between the wires and other bits of metal in their vicinity. Then he talked about various solutions like various kinds of low-V switches (negative-capacitance voltage-amplifying transistors, NEMS mechanical switches, quantum tunneling transistors, etc.), and LED+waveguide optical interconnects (e.g. this paper).
It seems from the replies to the parent comment that the formula is close to the OP formula. Score one for dimensional analysis, I guess, or else the OP formula has a justification that I’m not following.
OK thanks. I’m now kinda confused about your perspective because there seems to be a contradiction:
On the one hand, I think you said you were sympathetic to #1 (“There is a God’s-eye view of the world” is utter nonsense—it’s just a confused notion, like the set of all sets that don’t contain themselves”).
On the other hand, you seem to be agreeing here that “There is a God’s-eye view of the world” is something that might actually be true, and in fact is true in our GoL example.
Anyway, if we go with the second bullet point, i.e. “this is a thing that might be true”, then we can label it a “hypothesis” and put it into a Bayesian analysis, right?
To be specific: Let’s assume that GoL-person-A formulated the hypothesis: “There is a God’s-eye view of my universe, in the form of a list of lists of booleans with thus-and-such mathematical properties etc.”.
Then, over time, A keeps noticing that every prediction that the hypothesis has ever made, has come true.
So, being a good Bayesian, A’s credence on the hypothesis goes up and up, asymptotically approaching 100%.
This strikes me as a sound, non-coincidental reason for A to have reached that (correct) belief. Where do you disagree?
I still feel like you’re dodging the GoL issue. The situation is not that “GoL-person-A has harmless confusions, and it’s no big deal because they’ll still make good decisions”. The situation is that GoL-person-A is actually literally technically correct. There is in fact a list of lists of booleans, and it does have certain mathematical properties like obeying these four rules. Are you:
A) disagreeing with that? or
B) saying that GoL-person-A is correct by coincidence, i.e. that A did not have any sound basis to reach the beliefs that they believe, but they just happened to guess the right answer somehow? or
C) Asserting that there’s an important difference between the conversation between A and B in the GoL universe, versus the conversation between you and me in the string theory (or whatever) universe? or
D) something else?
For example, if there were an image-processing algorithm that used many fewer operations overall, but where those operations were more serial and less parallel—e.g. it required 1000 sequential steps for each image—then I think evolution would not have found it, because brains are too slow.
So then you need a different reason to think that such an algorithm doesn’t exist.
Maybe you can say “If such an algorithm existed, AI researchers would have found it by now.” But would they really? If AI researchers hadn’t been stealing ideas from the brain, would they have even invented neural nets by now? I dunno.
Or you can say “Something about the nature of image processing is that doing 1000 sequential steps just isn’t that useful for the task.” I guess I find that claim kinda plausible, but I’m just not very confident, I don’t feel like I have such a deep grasp of the fundamental nature of image processing that I can make claims like that.
In other domains besides image processing, I’d be even less confident. For example, I can kinda imagine some slightly-alien form of “reasoning” or “planning” that was mostly like human “reasoning” or “planning” but sometimes involved fast serial operations. After all, I find it very handy to have a fast serial laptop. If access to fast serial processing is useful for “me”, maybe it would be also useful for the low-level implementation of my brain algorithms. I dunno. Again, I think it’s hard to say either way.
It seems like you’re having B jump on argument #2, whereas I’m interested in a defense of #1. In other words, it’s trivial to say “we can’t be literally 100% certain that there’s an objective universe”, because after all we can’t be literally 100% certain of anything whatsoever. I can’t be literally 100% certain that 7 is prime either. But I would feel very comfortable building an AGI that kills everyone iff 7 is composite. Or if you want an physical example, I would feel very comfortable building an AGI that kills everyone iff the sun is not powered primarily by nuclear fusion. You have posts with philosophical arguments, are you literally 100% certain that those arguments are sound? It’s a fully general counterargument!
I don’t think your opinion is really #2, i.e. “There’s probably an objective universe out there but we can only be 99.99% confident of that, not literally 100%.” In the previous discussion you seemed to be frequently saying with some confidence that you regard “there is an objective universe” to be false if not nonsensical. Sorry if I’m misunderstanding.
In your quote above, you use the term “construct”, and I’m not sure why. GoL-Person A inferred that there is a list of lists, and inferred some properties of that list. And there is in fact a list of lists. And it does in fact have those properties. A is right, and if B is defending position #1, then B is wrong. Then we can talk about what types of observations and reasoning steps A and B might have used to reach their respective conclusions, and we can update our trust in those reasoning steps accordingly. And it seems to me that A and B would be making the same kinds of observations and arguments in their GoL universe that we are making in our string theory (or whatever) universe.
Kind of a nitpick, but if radiative heat loss has almost nothing to do with how chips are cooled, and it has almost nothing to do with how the brain is cooled, then why are we even bringing up the Stefan-Boltzmann law in the first place?
Thanks, this is great, I really feel like we’re converging here. Here’s where I think we stand.
Intrinsic motivation / curiosity:
We both agree that humans have an “intrinsic motivation” drive and that AGI will likewise have an “intrinsic motivation” drive, at least for the early part of training (perhaps it can “fade out” when the AGI is sufficiently smart and self-aware, such that instrumental convergence can substitute for intrinsic motivation?). I’m calling the intrinsic motivation “curiosity”, and I’m punting on the details of how it works. You’re calling it “curiosity / empowerment”, and apparently have something very specific in mind.
I think that intrinsic motivation in both humans & AGIs needs to be supplemented by a “drive to pay attention to humans”, which in humans is based on superficial things like an innate brainstem circuit that disproportionately fires when hearing human speech. Without that drive, I think the curiosity would be completely undirected, and you could wind up with an AGI that ignores the world and spends forever running Rule 110 in its head and finding its increasingly-complicated patterns, or studying the coloration of pebbles, etc. Whereas I think you disagree, and you think that “intrinsic motivation”, properly implemented, will automatically point itself at the world and technology and humans etc., and not at patterns-in-rule-110.
We also disagree about “drive for having high social status / impressing my friends”: You think it’s purely a special case of “intrinsic motivation” and thus requires no further explanation, I think it comes at least in part from “social instincts”, i.e. low-level drives that evolved in humans specifically because we are social animals.
I’m not immediately sure how to move forward in resolving either of those. I think you said you were going to have a post explaining more about how you think intrinsic motivation works, so maybe I’ll just wait for that.
Other low-level drives:
I think we agree that humans have some “social” low-level drives like “altruism / empathy” and “justice/anger” (which I’d call a subset of “social instincts”). We might be disagreeing about how complicated social instincts are (e.g. “how many low-level drives”), with me saying they’re probably pretty complicated and you saying they’re simple. But it’s also possible that we’re not disagreeing at all, but rather answering different questions, i.e. “the main aspects of human social instincts” versus “human social instincts in exact detail including subtle mood-shifts based on how somebody smells” or whatever.
I think we agree that AGI can have some or all of those human social instincts, but only if the AGI designers put them in, which would require (1) more research to nail down exactly how they’re implemented, (2) advocacy etc. to convince AGI designers to actually put in whatever social instincts we think they ought to put in.
I think we also agree that AGI can have low-level drives very different from any of the low-level drives in humans, like a low-level drive to get a high score in PacMan—not as a means to an end, but rather because the PacMan score is directly baked into the innate within-lifetime reward function. I think you’re inclined to emphasize that most of these possible low-level drives would be terribly dangerous, and I’m inclined to emphasize that future AGI designers might put them in anyway.
Explicit goals:
I think we agree that humans, combining their modestly-heterogeneous innate drives (e.g. psychopaths, people with autism, etc.) with modestly-heterogeneous training data (a.k.a. life history), can wind up pursuing an insane variety of explicit goals, like the guy trying to set a world record for longest time spent bathing in ice-water, etc. etc. So the claim “the AGI may wind up pursuing goals radically unlike humans” is less clear-cut than it sounds. OTOH, “the AGI may wind up pursuing explicit goals unlike typical humans in my culture” is a weaker statement, and I think definitely true. I would even say the stronger thing—that it is in fact possible for a future AGI to wind up pursuing an explicit goal that none of the 100 billion humans in history have ever pursued, e.g. maximizing the quantity of solar cells in the future light-cone, particularly if the AGI is programmed to have a low-level innate drive that no human has ever had, and if AGI designers don’t really know what they’re doing.
Where does that leave anthropomorphism?
When I think of anthropomorphism I have a negative association because I’m thinking of things like my comment here, where somebody was claiming that AGI isn’t dangerous because if an AGI just thought hard enough about it, it would conclude that acting honorably is inherently good and hurting people is inherently bad, because after all, that’s just the way it is. From my perspective, this is problematic anthropomorphism because the process of moral reasoning involves (among other things) queries to low-level “social instincts” drives (especially related to altruism and justice), and whoever builds the AGI won’t necessarily put in the same “social instincts” drives that humans have.
(I could have also pointed out that high-functioning sociopaths often have a very good understanding of honor etc. but not find those things motivating at all. Maybe that’s a general rule: if we see an “anthropomorphism” argument that really only applies to neurotypical people, and not to psychopaths and people with autism etc., then that’s a giant red flag.)
Anyway, when you think of anthropomorphism, it seems that your mind immediately goes to “humans can sometimes be single-mindedly in pursuit of power, and AGIs also can sometimes be single-mindedly in pursuit of power”, which happens to be a statement I agree with. So you wind up with a positive association.
Couple other things:
Agree, but only if we define “obviously” as “obviously to me and you”. I still think there’s a good chance that somebody would try.
Oh, sorry for bad communication, when I said “I think we could make an AGI with that goal [of maximizing paperclips]”, I should have added “in principle”. Obviously right now we can’t make any AGI whatsoever, and additionally we don’t know how to reliably make the AGI that is trying to do some particular thing that we had in mind. I doubt the problem of making a paperclip maximizer is fundamentally impossible, and I’d be pretty confident that we could eventually figure it out if we wanted to (which we don’t), if only we could survive long enough to do arbitrarily much trial-and-error. :-P