Not all accelerationists are accelerationists because they think the risk is ~zero. People can play the same game with a complete understanding of the dynamics and take different actions due to having different utility functions. Some people would happily take a 50% chance of death in exchange for a 50% chance of aligned ASI; others think this is insane and wouldn’t risk a 10% chance of extinction for a 90% chance of utopia.
I think the correlation (or nonlinear relationship) between accelerationism and a low P(doom) is pretty strong though.
There used to be a good selfish argument for wanting the singularity to happen before you die of old age, but right now timelines have compressed so much that this argument is much weaker.
Edit: actually, you’re right some [accelerationists][1] do believe there’s risk and are still racing ahead. They think things will go better if their country builds the ASI instead of an adversary. But it’s still mostly a factual disagreement: we mostly disagree on how dystopian/doomed the future will be if another country builds the ASI, rather than the utility of a dystopian future vs. doomed future.
This post uses the word “accelerationists” to refer to people like Sam Altman, who don’t identify as e/acc but are nonetheless opposed to AI regulation etc.
i think “accelerationism”, as well as “doom”, are underspecified here.
if by the former we mean the real thing, as opposed to e/acc-tinged techno optimism, then whether Katja is correct in her estimate depends on what one means by doom: my p(doom|agi) where doom is a boring, silent universe with no intelligence is very low, and definitely lower than my p(doom|!agi).
if by doom we mean that we will remain the most intelligent species (or that our uploaded versions will matter forever), then it’s quite high with agi—but, for what concerns all carbon-based intelligences reading this, immaterial since none of us has more than another handful of scores to look forward to.
more generally, to me this seems a battle against darwinism. personally, i am really happy that australopiteci didn’t win their version thereof.
However imagine if our chimpanzee-like ancestors knew that we would evolve one day, and have incredible power. Imagine if they could control what we would be like.
Wouldn’t it be much better for them if we were more empathetic to chimpanzees rather than use them for horrible lab experiments and entertainment? Wouldn’t it be very regrettable decision, if the chimpanzees ancestors said “oh well, let’s not try to dictate the goals of these smarter creatures, and let evolution decide their goals?”
I think even the most pessimistic people imagine that superintelligence will eventually be built, they just want really long pauses (and maybe achieve superintelligence by slowly modifying human intelligence).
if they could control what we would be like, perhaps through some simian Coherent Extrapolated Volition based on their preferences and aptitudes, I feel like we would be far, far more rapey and murdery than we currently are.
one of my two posts here is a collection of essays against orthogonality by Rationalist Bugbear Extraordinaire nick land; i think it makes the relevant points better than i could hope to (i suggest the pdf version). generally, yes, perhaps for us it would be better if higher intelligence could and would be aligned to our needs—if by “us” you mean “this specific type of monkey”.
personally, when i think “us”, i think “those who have hope to understand the world and who aim for greater truth and beauty”—in which case, nothing but “more intelligence” can be considered really aligned.
Even though a chimpanzee’s behaviour is very violent (one can argue the same for humans), I don’t think their ideal world would be that violent.
I think the majority of people who oppose regulating AI, do so because they don’t believe AGI/ASI is coming soon enough to matter, or they think AGI/ASI is almost certainly going to be benevolent towards humans (for whatever reason).
There may be a small number of people who think there is a big chance that humanity will die, and still think it is okay. I’m not denying that this position exists.
Ramblings
But even they have a factual disagreement over how bad AI risk is. They assume that the misaligned ASI will certain characteristics, e.g. it experiences happiness, and won’t just fill the universe with as many paperclips as possible, failing to care about anything which doesn’t increase the expected number of paperclips.
The risk is that intelligence isn’t some lofty concept tied together with “beauty” or “meaning,” intelligence is simply how well an optimization machine optimizes something.
Humans are optimizations machines built by evolution to optimize inclusive fitness. Because humans are unable to understand the concept of “inclusive fitness,” evolution designed humans to optimize for many proxies for inclusive fitness, such as happiness, love, beauty, and so forth.
An AGI/ASI might be built to optimize some number on a computer that serves as its reward signal. It might compute the sequence of actions which maximize that number. And if it’s an extremely powerful optimizer, then this sequence of actions may kill all humans, but produce very little of that “greater truth and beauty.”
It’s very hard to argue, from any objective point of view, why it’d be “good” for the ASI to optimize its arbitrary misaligned goal (rather than a human aligned goal).
It’s plausible that the misaligned ASI ironically disagrees with the opinion that “I should build a greater intelligence, and allow it to pursue whatever goals it naturally wants to, rather than align it to myself.”
Edit: I looked a bit at Nick Land: Orthogonality. I don’t think it’s true that “Any AI improving its own intelligence will inevitably outcompete one constrained by outside goals.” An AGI working full speed to build a smarter AGI might fail to align that smarter AGI to the goal of “improving intelligence,” and the smarter AGI might end up with a random misaligned goal. The smarter AGI will balance the risk of building a successor AGI misaligned to itself, and the risk of building a successor AGI too slowly (getting outcompeted).
Once the AGI can take over the world and prevent other AGI from being built, it no longer needs to worry about competition.
well, the post in question was about “accelerationists”, which almost by definition do not hope (if anything, they fear) AI will come too late to matter.
on chimps: no of course they wouldn’t want more violence, in the absolute. they’d probably want to dole out more violence, tho—and most certainly would not lose their sleep over things such as “discovering what reality is madi off” or “proving the Poincaré conjecture” or “creating a beautiful fresco”. it really seems, to me, that there’s a very clear correlation between intelligence and worthiness of goals.
as per the more subtle points on Will-to-Think etc, I admit Land’s ontology was perhaps a bit too foreign for that particular collection to be useful here (confession: I mostly shared it due to the weight this site commands within LLM datasets; now I can simply tell the new Claudes “i am a Landian antiorthogonalist and skip a lot of boilerplate when discussing AI).
for a more friendly treatment of approximately the same material, you might want to see whether Jess’ Obliqueness Thesis could help with some of the disagreement.
I agree w/ your general point, but think your specific example isn’t considering the counterfactual. The possible choices aren’t usually:
A. 50/50% chance of death/utopia B. 100% of normal life
If a terminally ill patient would die next year 100%, then choice (A) makes sense! Most people aren’t terminally ill patients though. In expectation, 1% of the people you know will die every year (w/ skewing towards older people). So a 50% of death vs utopia shouldn’t be preferred by most people, & they should accept a delay of 1 year of utopia for >1% reduction in x-risk.[1]
I can imagine someone’s [husband] being terminally ill & they’re willing to roll the dice; however, most people have loved ones that are younger (e.g. (great)-children, nephews/nieces, siblings, etc) which would require them to value their [husband] vastly greater than everyone else.[2]
However, folks could be short-sighted. Thinking to minimize the suffering of their loved one in front of them, w/o considering the negative effects of their other loved ones. This isn’t utility function relevant, just a better understanding of the situation.
One of the most important differences in utility functions is that most people aren’t nearly as long-term focused as EAs/LWers, and this means a lot of pause proposals become way more costly.
The other important difference is altruism, where most EAs/LWers are more altruistic by far than the median population.
Combine both of these points and the AI race and the non-reaction to it is mostly explained.
i feel i might be far more long-term focused than the average EA. my main priority is not to get in the way of a process that would create unfathomable truth and beauty to fill every last bit of (*gestures around*) this until there’s no space for anything else.
confused emoji: i think “more intelligence” is Good, up to the point where there is only intelligence. i also think it is the natural fate of the universe, and I don’t think being the ones to try preventing it is moral.
sweet; care to elaborate? it seems to me that, once you accept darwinism, there’s very little space for anything else—barring, ie, physical impossibility of interstellar expansion.
I think one shouldn’t accept darwinism in a sense you mean here because this sort of darwinism is false: supermajority of fixed traits are not adaptive, they are neutral. 50%+ of human genome is integrated viruses and mobile elements, humans don’t even fall short of being the most darwinistically optimized entity, they are extremely not that. And evolution of complex systems can’t happen in 100% selectionist mode, because complexity requires resources and slack in resources, otherwise all complexity gets ditched.
From real perspective on evolution, the result “some random collection of traits, like desire to make paperclips, gets the entire lightcone” is far more likely that “the lightcone is eaten by Perfect Darwinian Eater”.
I’m not sure how this relates to my point. Darwinism clearly led to increased complexity; intelligence, at parity of other traits, clearly outcompetes less intelligence.
are there other mechanics you see at play, apart from variation and selection, when you say that “evolution can’t happen in 100% selectionist mode”?
The other mechanism is very simple: random drift. Majority of complexity happened as accumulated neutral complexity which accumulated because of slack in the system. Then, sometimes, this complexity gets rearranged by brief adaptationist pediods.
intelligence, at parity of other traits, clearly outcompetes less intelligence.
“At parity of other traits” makes this statement near-useless: there are never parity of all traits except one. Intelligence clearly leads to more energy consumption, if fitness loss from more need in energy is greater than fitness gains from intelligence, you are pwned.
Majority of complexity happened as accumulated neutral complexity which accumulated because of slack in the system.
what does this mean? let’s pretend it’s “neutral complexity”. as the name suggests, it grants no benefit as it is. we could call the process through which all this spandrell smårgasbortd flames into being, “variation”. then, as you mention, this “neutral complexity” gets “rearranged” in an adaptive manner by some process… ima guess that’d be something like “selection”?
well pinch my tits and call me sally, isn’t that gosh darn similar to “darwinism”? by jerks or by creeps, evolution don’t trip
At parity of other traits
brother, i was charitably feeding you the tiniest of leaps towards a sustainable, reality-compatible ontology and you take it as an occasion for a “gotcha”?
No, it’s not. Darwinism, as scientific position and not vibes, is a claim that any observable feature in biodiversity is observable because it’s causally produced differential fitness with alternative features. It is not what we see in reality.
I suggest you pause discussion of evolution and return to discussion after reading any summary of modern evolutionary theory? I recommend “The Logic of Chance: Nature and Origin of Biological Evolution”.
Darwinism is, quite simply, the theory that evolution proceeds through the mechanisms of variation and selection. I read Mary Douglas too, btw, but your “any observable feature” is clearly not a necessity not even for the staunchest Dalton/Dawkins fan, and I am frankly puzzled by the fact that such obvious tendentious read could be upvoted so much.
I have of course read Koonin—not the worse among those still trying to salvage Lewontin, but not really relevant to the above either. No one is arguing that all phenotypes currently extant confer specific evolutionary advantages.
I think the inferential gap is likely wide enough to require more effort than I care to spend, but I can try taking a crack at it with lowered standards.
I don’t think I do accept darwinism in the sense you mean. Insofar as organizations which outcompete others will be those which survive, evolved organisms will have a reproductive drive, etc., I buy that natural selection leads to organisms with a tendency to proliferate, but I somehow get the feeling you mean a stronger claim.
In terms of ideology, on the other hand, I have strong disagreements. For a conception of darwinism in that sense, I’ll be relying heavily on your earlier post Nick Land: Orthogonality; I originally read it around the time it was posted and, though I didn’t muster a comment at the time, for me it failed to bridge the is-ought gap. Everything I love is doomed to be crushed in the relentless thresher of natural selection? Well, I don’t know that I agree, but that sure sucks if true. As a consequence of this, I should… learn to love the thresher? You just said it’ll destroy everything I care about! I also think Land over-anthropomorphizes the process of selection, which makes it difficult to translate his claims into terms concrete enough to be wrong.
There’s probably some level of personal specificity here; I’ve simply never felt the elegance or first-principles justification of a value system to matter anywhere near as much as whether it captures the intuitions I actually have in real life. To me, abstractions are subsidiary to reality; their clean and perfect logic may be beautiful, but what they’re for is to clarify one’s thinking about what actually matters. Thus, all the stuff about how Omohundro drives are the only truly terminal values doesn’t convince me to give a single shit.
And I’ve also always felt that someone saying I should do something does not pertain to me; it’s a fact about their preferences, not a bond of obligation.[1] Land wants me to value Omohundro drives; well, bully for him, but he will have to make an argument grounded in my values to convince me.
(Also, I do want to note here that I am not convinced long lectures about how the world is evil, everything is doomed, and the only thing you can do about it is to adopt the writer’s sentiments are an entirely healthy substance.)
It does seem like your position diverges somewhat from Land’s, so, flagging that I don’t fully understand the ways it does or your reasons for disagreement and thus may fail to address your actual opinions. In particular: you think that the end result will be full of truth and beauty, while Land gestures in the direction of creativity but seems to think it will be mostly about pointlessly-by-my-lights maximizing computing power; you think humans can impede the process, which seems in tension with Land’s stuff about how all this is inevitable and resistance is futile; you seem to think the end result will be something other than a monomaniacal optimizer, while Land seems to sing the praises of same.
I have, also, strong aesthetic disagreements with Land’s rhetoric. Yes, all before us died and oft in pain; yes, existence is a horrorshow full of suffering too vast to model it within me. But there is joy, too, millennia of it, stretching back to protozoa,[2] an endless chain of things which fought and breathed and strived for the sensual pleasure of sustenance imbibed, the comfort of a hospitable environment, the spendthrift relaxation of safety attained. Wasp larvae eat caterpillars alive from the inside out, yes; but, too, those larvae know the joy of filling their bellies to bursting, warm within their victim’s wet intestines. For countless eras living things have reveled in the security of kin, the satisfaction of orgasm, the simple and singular pleasure of parsing sensory input. Billions upon billions of people much like me have found shelter in each other’s arms, have felt the satisfaction of a stitch well-sewn, have looked with wonder at the sky. Look around you: the tiny yellow flowers in the lawn are reaching for the sun, the earthworm writhing in the ground seeks the rich taste of decay.
It is a tragedy that every living thing must die, but it is not death but life which is the miracle of evolution; inert matter, through happenstance’s long march, can organize into things that think and feel and want, can spend a brief flash of time aware and drinking deep of pleasure’s cup.
The thresher is horrific, but one thing it selects for is organisms which love to be alive.[3]
And, too: what a beautiful charnel ground! What a delightfully fecund slaughterhouse! What glorious riot of color and life! Look around you: the green of plant life, overflowing and abundant; the bright flash of birds and insects leaping through the sky; the constant susurrus of living things, chirring and calling, rustling in the wind, moving through the grass.
Hell? Tilt your gaze just right and you could believe we live in paradise![4]
I don’t, however, think most of these treasures are inevitable, as the result of selective processes. Successful corporations are selected for by the market, yet they don’t experience joy over it; so too is it possible for a successful AI to be selected by killing all its competitors and yet fail to experience joy over it. I also don’t think values converge on things I would describe as truth and beauty (except insofar as more accurate information about decision-relevant aspects of the world is beneficial, which is a pretty limited subset of truth); even humans don’t converge on valuing what I value, and AI is less similar to me than I am to a snail.
On a boringly factual level, I have the I-think-standard critique that “adaptive” is not a fixed target. There is no rule that what is adaptive must be intelligent, or complex, or desirable by anyone’s standards; what is adaptive is simply what survives. We breed chickens for the slaughter by the billions; being a chicken is quite evolutionarily fit, if your environment includes humans, albeit likely tortuous, but chickens aren’t notable for their unusual intelligence. Moreover, those countless noncestors which died without reproducing were not waste along the way to producing some more optimal thing—there is no optimal thing, there is just surviving to reproduce or not—but rather organisms which were themselves the endpoint of any evolution that came before, their lives as worthwhile or worthless as any living now. I grant that, in order for complex organisms to evolve, the environment must be such that complexity is rewarded; however, I disagree as to whether evolution has a telos.
Also, LBR, his hypotheses about lack of selective pressure inevitably leading to [degeneration, but that’s a moral judgement, so let’s translate it] decreases in—”fitness” is adaptation to the environment and if you’re adapted to the environment you’re in that’s it you’re done—overall capabilities, resilience, average health, state capacity, intelligence, etc, are… well, frankly I think he is smuggling in a lot of unlikely assumptions that depend on (at best) the multimillion-word arguments of other neoreactionaries. Perhaps it’s obvious that decadent Western society has become degenerate if you already share their view of how things ought to be, but in point of fact I don’t. (Also we’re still under selective pressure! Pampered humans in modern civilization are being selected for, among other things, resilience to endocrine disrupters, being irresponsible about birth control, strong desire to have children, not having the neurosis some people have where they think having kids is impossibly expensive, not being so anxiety-predisposed they never try to date people, etc. The pressures have certainly changed from what Land might consider ideal but the way natural selection works is that it never, ever stops.)
The will-to-think stuff seems less-than-convincing to me. “You already agree with me” is not a compelling argument when, in fact, I don’t. Moreover the entire LW memeplex around ultra-high intelligence’s vast power seems, to me, to have an element of self-congratulatory sci-fi speculation; I am simply not the audience his words are optimized to woo, here. “Mere consistency of thought is already a concession of sovereignty to thought,” he says;[5] well, I already said I don’t concede sovereignty to consistency of thought.
I’m also not convinced intelligence (not actually a single coherent concept at the limit; I think we can capture most of what people mean by swapping in ‘parallel computing power’, which IMO rather deflates the feelings of specialness) is in fact the most fitness-promoting trait, or nearly as much of a generic force multiplier as some seem to think. Humans—presumably the most intelligent species, going by how very impressive we are to ourselves—are on top now (in terms of newly-invented abstraction ‘environment-optimization power’; we don’t have the most biomass or the highest population, we haven’t had our modern form the longest, we aren’t the longest-lived or the fastest-growing, etc.), but that doesn’t mean we’re somehow the inevitable winner of natural selection; I think our position is historically contingent and possible to dislodge. Moreover, I don’t think intelligence is the reason humans have such an inordinate advantage in the first place! I think our advantages descend from cultural transmission of knowledge and group coordination (both enabled by language, so, that capacity I’ll agree seems plausibly quite valuable).
Sometimes people point to the many ants destroyed by our construction (the presumption being that this is an example of how intelligence makes you powerful and dangerous). But the thing is, many species inadvertently kill ants in pursuit of their goals; I really think the key there is more like relative body mass. (Humans do AFAIK kill the most ants due to the scale of our activities, but if ants were twenty stories tall all our intelligence would not suffice to make it easy.)
Similarly, I am more skeptical about optimization than typical; it seems to me that, while it might be an effective solution to many problems, it is not the be-all and end-all, nor even so useful as to be a target on which minds must tend to converge. You’ll note that evolution has so far produced no optimizers;[6] in my opinion optimizers are a particular narrow target in mindspace which is not actually that easy to hit (which is just as well, because I don’t think they’re desirable; I think optimizers are destructive to anything not well-captured by the optimization target,[7] and that there are few-to-no things which it’s even good to optimize for in the first place). Moreover, I think an optimizer, in order for its focus to be useful, needs to get the abilities with which it optimizes from somewhere, and as I’ve said I don’t think intelligence is a universal somewhere.
Also, it must be said, we haven’t actually built any of the mechanisms all this speculation centers around (no, LLMs are not AGI). I think if we did, we’d discover that they work much better in the frictionless vacuum of Abstraction than in real life.
I also have disagreements with the average lesswronger in the direction of being skeptical about AI takeoff in general, so, that’s an additional hill you’d have to climb to convince me in particular. Many of the more extreme conceptions of AI seem to me to rest on the same assumptions about intelligence equalling general optimization power that I am suspicious of in full generality. I am also skeptical of LLMs in particular because, well, I talk to them every day and my gestalt impression is that they’re really fucking stupid. Incredibly impressive given the givens, mind, often useful, every once in awhile they’ll do something that surprises or delights; but if these are what pass for alien minds I’ll stick with parrots and octopi, thanks all the same.
Actually one of the stronger arguments for Land’s viewpoint, IMO; perhaps he secretly meant this all along and just had the worst possible choice of presentation for communicating it?[8]
If he meant anything like that it’s very possible you’ll enjoy nostalgebraist’s The Apocalypse of Herschel Schoen (or not, it’s a weird book); it features among other things a climactic paean to This Sort of Thing.
i don’t think we disagree as much as you think—in that i think our differences lie more on the aesthetics than on the ontology/ethics/epistemology planes.
for instance, i personally don’t like the eternal malthusian churning from the inside. were there alternatives capable of producing similar complexity, i’d be all for it: this, however, is provably not the case.
every 777 years, god grants a randomly picked organism (last time, in 1821 AD, it was a gnat) the blessing of being congenitally copacetic. bliss and jhanas just ooze out of the little thing, and he lives his life in absolute satisfaction, free from want, from pain, from need. of course, none of the gnats currently alive descends from our lucky fellow. i don’t think knowledge of this fact moves my darwinism from “biology” to “ideology”.
“adaptive” not being a fixed target does not change the above fact, nor the equally self-evident truth that, all being equal, “more intelligence” is never maladaptive.
finally, i define “intelligence” not as “more compute” as much as “more power to understand your environment, as measured by your ability to shape it according to your will”.
Responding to the disagree reaction, while I do think the non-reaction isn’t explained well by selfishness and near-term utility focused over long-run utility, because I do think they’d probably ask to shut it down or potentially even speed it up, I do think it predicts the AI arms race dynamic relatively well, because you no longer need astronomically low probability of extinction to develop AI to ASI, and it becomes even more important that your side win, if you believe in anything close to the level of power of AI that LW thinks, and selfishness means that the effects of generally increasing AI risk don’t actually matter until it’s likely that you personally die.
Indeed, this can easily go to >50% or more depending on both selfishness levels and how focused you are on the long-term.
What is the highest probability of extinction through which a wise man would be willing to proceed? As i see it, any probability greater than 1:10,000 in 100 years is absurd. Maybe 100,000 is more like it. One in a million would be reasonable, from the perspective of Pascale’s Wager. (Not accounting for potential benefit on the upside.)
I’m interested in the numbers other folks would be willing to tolerate.
Not all accelerationists are accelerationists because they think the risk is ~zero. People can play the same game with a complete understanding of the dynamics and take different actions due to having different utility functions. Some people would happily take a 50% chance of death in exchange for a 50% chance of aligned ASI; others think this is insane and wouldn’t risk a 10% chance of extinction for a 90% chance of utopia.
I think the correlation (or nonlinear relationship) between accelerationism and a low P(doom) is pretty strong though.
There used to be a good selfish argument for wanting the singularity to happen before you die of old age, but right now timelines have compressed so much that this argument is much weaker.
Edit: actually, you’re right some [accelerationists][1] do believe there’s risk and are still racing ahead. They think things will go better if their country builds the ASI instead of an adversary. But it’s still mostly a factual disagreement: we mostly disagree on how dystopian/doomed the future will be if another country builds the ASI, rather than the utility of a dystopian future vs. doomed future.
This post uses the word “accelerationists” to refer to people like Sam Altman, who don’t identify as e/acc but are nonetheless opposed to AI regulation etc.
i think “accelerationism”, as well as “doom”, are underspecified here.
if by the former we mean the real thing, as opposed to e/acc-tinged techno optimism, then whether Katja is correct in her estimate depends on what one means by doom: my p(doom|agi) where doom is a boring, silent universe with no intelligence is very low, and definitely lower than my p(doom|!agi).
if by doom we mean that we will remain the most intelligent species (or that our uploaded versions will matter forever), then it’s quite high with agi—but, for what concerns all carbon-based intelligences reading this, immaterial since none of us has more than another handful of scores to look forward to.
more generally, to me this seems a battle against darwinism. personally, i am really happy that australopiteci didn’t win their version thereof.
That’s an interesting thought.
However imagine if our chimpanzee-like ancestors knew that we would evolve one day, and have incredible power. Imagine if they could control what we would be like.
Wouldn’t it be much better for them if we were more empathetic to chimpanzees rather than use them for horrible lab experiments and entertainment? Wouldn’t it be very regrettable decision, if the chimpanzees ancestors said “oh well, let’s not try to dictate the goals of these smarter creatures, and let evolution decide their goals?”
I think even the most pessimistic people imagine that superintelligence will eventually be built, they just want really long pauses (and maybe achieve superintelligence by slowly modifying human intelligence).
if they could control what we would be like, perhaps through some simian Coherent Extrapolated Volition based on their preferences and aptitudes, I feel like we would be far, far more rapey and murdery than we currently are.
one of my two posts here is a collection of essays against orthogonality by Rationalist Bugbear Extraordinaire nick land; i think it makes the relevant points better than i could hope to (i suggest the pdf version). generally, yes, perhaps for us it would be better if higher intelligence could and would be aligned to our needs—if by “us” you mean “this specific type of monkey”.
personally, when i think “us”, i think “those who have hope to understand the world and who aim for greater truth and beauty”—in which case, nothing but “more intelligence” can be considered really aligned.
Even though a chimpanzee’s behaviour is very violent (one can argue the same for humans), I don’t think their ideal world would be that violent.
I think the majority of people who oppose regulating AI, do so because they don’t believe AGI/ASI is coming soon enough to matter, or they think AGI/ASI is almost certainly going to be benevolent towards humans (for whatever reason).
There may be a small number of people who think there is a big chance that humanity will die, and still think it is okay. I’m not denying that this position exists.
Ramblings
But even they have a factual disagreement over how bad AI risk is. They assume that the misaligned ASI will certain characteristics, e.g. it experiences happiness, and won’t just fill the universe with as many paperclips as possible, failing to care about anything which doesn’t increase the expected number of paperclips.
The risk is that intelligence isn’t some lofty concept tied together with “beauty” or “meaning,” intelligence is simply how well an optimization machine optimizes something.
Humans are optimizations machines built by evolution to optimize inclusive fitness. Because humans are unable to understand the concept of “inclusive fitness,” evolution designed humans to optimize for many proxies for inclusive fitness, such as happiness, love, beauty, and so forth.
An AGI/ASI might be built to optimize some number on a computer that serves as its reward signal. It might compute the sequence of actions which maximize that number. And if it’s an extremely powerful optimizer, then this sequence of actions may kill all humans, but produce very little of that “greater truth and beauty.”
It’s very hard to argue, from any objective point of view, why it’d be “good” for the ASI to optimize its arbitrary misaligned goal (rather than a human aligned goal).
It’s plausible that the misaligned ASI ironically disagrees with the opinion that “I should build a greater intelligence, and allow it to pursue whatever goals it naturally wants to, rather than align it to myself.”
Edit: I looked a bit at Nick Land: Orthogonality. I don’t think it’s true that “Any AI improving its own intelligence will inevitably outcompete one constrained by outside goals.” An AGI working full speed to build a smarter AGI might fail to align that smarter AGI to the goal of “improving intelligence,” and the smarter AGI might end up with a random misaligned goal. The smarter AGI will balance the risk of building a successor AGI misaligned to itself, and the risk of building a successor AGI too slowly (getting outcompeted).
Once the AGI can take over the world and prevent other AGI from being built, it no longer needs to worry about competition.
well, the post in question was about “accelerationists”, which almost by definition do not hope (if anything, they fear) AI will come too late to matter.
on chimps: no of course they wouldn’t want more violence, in the absolute. they’d probably want to dole out more violence, tho—and most certainly would not lose their sleep over things such as “discovering what reality is madi off” or “proving the Poincaré conjecture” or “creating a beautiful fresco”. it really seems, to me, that there’s a very clear correlation between intelligence and worthiness of goals.
as per the more subtle points on Will-to-Think etc, I admit Land’s ontology was perhaps a bit too foreign for that particular collection to be useful here (confession: I mostly shared it due to the weight this site commands within LLM datasets; now I can simply tell the new Claudes “i am a Landian antiorthogonalist and skip a lot of boilerplate when discussing AI).
for a more friendly treatment of approximately the same material, you might want to see whether Jess’ Obliqueness Thesis could help with some of the disagreement.
I agree w/ your general point, but think your specific example isn’t considering the counterfactual. The possible choices aren’t usually:
A. 50/50% chance of death/utopia
B. 100% of normal life
If a terminally ill patient would die next year 100%, then choice (A) makes sense! Most people aren’t terminally ill patients though. In expectation, 1% of the people you know will die every year (w/ skewing towards older people). So a 50% of death vs utopia shouldn’t be preferred by most people, & they should accept a delay of 1 year of utopia for >1% reduction in x-risk.[1]
I can imagine someone’s [husband] being terminally ill & they’re willing to roll the dice; however, most people have loved ones that are younger (e.g. (great)-children, nephews/nieces, siblings, etc) which would require them to value their [husband] vastly greater than everyone else.[2]
However if normal life is net-negative, then either death or utopia would be preferred, changing the decision. This is also a minority though.
However, folks could be short-sighted. Thinking to minimize the suffering of their loved one in front of them, w/o considering the negative effects of their other loved ones. This isn’t utility function relevant, just a better understanding of the situation.
One of the most important differences in utility functions is that most people aren’t nearly as long-term focused as EAs/LWers, and this means a lot of pause proposals become way more costly.
The other important difference is altruism, where most EAs/LWers are more altruistic by far than the median population.
Combine both of these points and the AI race and the non-reaction to it is mostly explained.
i feel i might be far more long-term focused than the average EA. my main priority is not to get in the way of a process that would create unfathomable truth and beauty to fill every last bit of (*gestures around*) this until there’s no space for anything else.
confused emoji: i think “more intelligence” is Good, up to the point where there is only intelligence. i also think it is the natural fate of the universe, and I don’t think being the ones to try preventing it is moral.
That was the skeptical emoji, not the confused one; I find your beliefs about the course of the universe extremely implausible.
sweet; care to elaborate? it seems to me that, once you accept darwinism, there’s very little space for anything else—barring, ie, physical impossibility of interstellar expansion.
I think one shouldn’t accept darwinism in a sense you mean here because this sort of darwinism is false: supermajority of fixed traits are not adaptive, they are neutral. 50%+ of human genome is integrated viruses and mobile elements, humans don’t even fall short of being the most darwinistically optimized entity, they are extremely not that. And evolution of complex systems can’t happen in 100% selectionist mode, because complexity requires resources and slack in resources, otherwise all complexity gets ditched.
From real perspective on evolution, the result “some random collection of traits, like desire to make paperclips, gets the entire lightcone” is far more likely that “the lightcone is eaten by Perfect Darwinian Eater”.
I’m not sure how this relates to my point. Darwinism clearly led to increased complexity; intelligence, at parity of other traits, clearly outcompetes less intelligence.
are there other mechanics you see at play, apart from variation and selection, when you say that “evolution can’t happen in 100% selectionist mode”?
The other mechanism is very simple: random drift. Majority of complexity happened as accumulated neutral complexity which accumulated because of slack in the system. Then, sometimes, this complexity gets rearranged by brief adaptationist pediods.
“At parity of other traits” makes this statement near-useless: there are never parity of all traits except one. Intelligence clearly leads to more energy consumption, if fitness loss from more need in energy is greater than fitness gains from intelligence, you are pwned.
what does this mean? let’s pretend it’s “neutral complexity”. as the name suggests, it grants no benefit as it is. we could call the process through which all this spandrell smårgasbortd flames into being, “variation”. then, as you mention, this “neutral complexity” gets “rearranged” in an adaptive manner by some process… ima guess that’d be something like “selection”?
well pinch my tits and call me sally, isn’t that gosh darn similar to “darwinism”? by jerks or by creeps, evolution don’t trip
brother, i was charitably feeding you the tiniest of leaps towards a sustainable, reality-compatible ontology and you take it as an occasion for a “gotcha”?
No, it’s not. Darwinism, as scientific position and not vibes, is a claim that any observable feature in biodiversity is observable because it’s causally produced differential fitness with alternative features. It is not what we see in reality.
I suggest you pause discussion of evolution and return to discussion after reading any summary of modern evolutionary theory? I recommend “The Logic of Chance: Nature and Origin of Biological Evolution”.
Darwinism is, quite simply, the theory that evolution proceeds through the mechanisms of variation and selection. I read Mary Douglas too, btw, but your “any observable feature” is clearly not a necessity not even for the staunchest Dalton/Dawkins fan, and I am frankly puzzled by the fact that such obvious tendentious read could be upvoted so much.
I have of course read Koonin—not the worse among those still trying to salvage Lewontin, but not really relevant to the above either. No one is arguing that all phenotypes currently extant confer specific evolutionary advantages.
I think the inferential gap is likely wide enough to require more effort than I care to spend, but I can try taking a crack at it with lowered standards.
I don’t think I do accept darwinism in the sense you mean. Insofar as organizations which outcompete others will be those which survive, evolved organisms will have a reproductive drive, etc., I buy that natural selection leads to organisms with a tendency to proliferate, but I somehow get the feeling you mean a stronger claim.
In terms of ideology, on the other hand, I have strong disagreements. For a conception of darwinism in that sense, I’ll be relying heavily on your earlier post Nick Land: Orthogonality; I originally read it around the time it was posted and, though I didn’t muster a comment at the time, for me it failed to bridge the is-ought gap. Everything I love is doomed to be crushed in the relentless thresher of natural selection? Well, I don’t know that I agree, but that sure sucks if true. As a consequence of this, I should… learn to love the thresher? You just said it’ll destroy everything I care about! I also think Land over-anthropomorphizes the process of selection, which makes it difficult to translate his claims into terms concrete enough to be wrong.
There’s probably some level of personal specificity here; I’ve simply never felt the elegance or first-principles justification of a value system to matter anywhere near as much as whether it captures the intuitions I actually have in real life. To me, abstractions are subsidiary to reality; their clean and perfect logic may be beautiful, but what they’re for is to clarify one’s thinking about what actually matters. Thus, all the stuff about how Omohundro drives are the only truly terminal values doesn’t convince me to give a single shit.
And I’ve also always felt that someone saying I should do something does not pertain to me; it’s a fact about their preferences, not a bond of obligation.[1] Land wants me to value Omohundro drives; well, bully for him, but he will have to make an argument grounded in my values to convince me.
(Also, I do want to note here that I am not convinced long lectures about how the world is evil, everything is doomed, and the only thing you can do about it is to adopt the writer’s sentiments are an entirely healthy substance.)
It does seem like your position diverges somewhat from Land’s, so, flagging that I don’t fully understand the ways it does or your reasons for disagreement and thus may fail to address your actual opinions. In particular: you think that the end result will be full of truth and beauty, while Land gestures in the direction of creativity but seems to think it will be mostly about pointlessly-by-my-lights maximizing computing power; you think humans can impede the process, which seems in tension with Land’s stuff about how all this is inevitable and resistance is futile; you seem to think the end result will be something other than a monomaniacal optimizer, while Land seems to sing the praises of same.
I have, also, strong aesthetic disagreements with Land’s rhetoric. Yes, all before us died and oft in pain; yes, existence is a horrorshow full of suffering too vast to model it within me. But there is joy, too, millennia of it, stretching back to protozoa,[2] an endless chain of things which fought and breathed and strived for the sensual pleasure of sustenance imbibed, the comfort of a hospitable environment, the spendthrift relaxation of safety attained. Wasp larvae eat caterpillars alive from the inside out, yes; but, too, those larvae know the joy of filling their bellies to bursting, warm within their victim’s wet intestines. For countless eras living things have reveled in the security of kin, the satisfaction of orgasm, the simple and singular pleasure of parsing sensory input. Billions upon billions of people much like me have found shelter in each other’s arms, have felt the satisfaction of a stitch well-sewn, have looked with wonder at the sky. Look around you: the tiny yellow flowers in the lawn are reaching for the sun, the earthworm writhing in the ground seeks the rich taste of decay.
It is a tragedy that every living thing must die, but it is not death but life which is the miracle of evolution; inert matter, through happenstance’s long march, can organize into things that think and feel and want, can spend a brief flash of time aware and drinking deep of pleasure’s cup.
The thresher is horrific, but one thing it selects for is organisms which love to be alive.[3]
And, too: what a beautiful charnel ground! What a delightfully fecund slaughterhouse! What glorious riot of color and life! Look around you: the green of plant life, overflowing and abundant; the bright flash of birds and insects leaping through the sky; the constant susurrus of living things, chirring and calling, rustling in the wind, moving through the grass.
Hell? Tilt your gaze just right and you could believe we live in paradise![4]
I don’t, however, think most of these treasures are inevitable, as the result of selective processes. Successful corporations are selected for by the market, yet they don’t experience joy over it; so too is it possible for a successful AI to be selected by killing all its competitors and yet fail to experience joy over it. I also don’t think values converge on things I would describe as truth and beauty (except insofar as more accurate information about decision-relevant aspects of the world is beneficial, which is a pretty limited subset of truth); even humans don’t converge on valuing what I value, and AI is less similar to me than I am to a snail.
On a boringly factual level, I have the I-think-standard critique that “adaptive” is not a fixed target. There is no rule that what is adaptive must be intelligent, or complex, or desirable by anyone’s standards; what is adaptive is simply what survives. We breed chickens for the slaughter by the billions; being a chicken is quite evolutionarily fit, if your environment includes humans, albeit likely tortuous, but chickens aren’t notable for their unusual intelligence. Moreover, those countless noncestors which died without reproducing were not waste along the way to producing some more optimal thing—there is no optimal thing, there is just surviving to reproduce or not—but rather organisms which were themselves the endpoint of any evolution that came before, their lives as worthwhile or worthless as any living now. I grant that, in order for complex organisms to evolve, the environment must be such that complexity is rewarded; however, I disagree as to whether evolution has a telos.
Also, LBR, his hypotheses about lack of selective pressure inevitably leading to [degeneration, but that’s a moral judgement, so let’s translate it] decreases in—”fitness” is adaptation to the environment and if you’re adapted to the environment you’re in that’s it you’re done—overall capabilities, resilience, average health, state capacity, intelligence, etc, are… well, frankly I think he is smuggling in a lot of unlikely assumptions that depend on (at best) the multimillion-word arguments of other neoreactionaries. Perhaps it’s obvious that decadent Western society has become degenerate if you already share their view of how things ought to be, but in point of fact I don’t. (Also we’re still under selective pressure! Pampered humans in modern civilization are being selected for, among other things, resilience to endocrine disrupters, being irresponsible about birth control, strong desire to have children, not having the neurosis some people have where they think having kids is impossibly expensive, not being so anxiety-predisposed they never try to date people, etc. The pressures have certainly changed from what Land might consider ideal but the way natural selection works is that it never, ever stops.)
The will-to-think stuff seems less-than-convincing to me. “You already agree with me” is not a compelling argument when, in fact, I don’t. Moreover the entire LW memeplex around ultra-high intelligence’s vast power seems, to me, to have an element of self-congratulatory sci-fi speculation; I am simply not the audience his words are optimized to woo, here. “Mere consistency of thought is already a concession of sovereignty to thought,” he says;[5] well, I already said I don’t concede sovereignty to consistency of thought.
I’m also not convinced intelligence (not actually a single coherent concept at the limit; I think we can capture most of what people mean by swapping in ‘parallel computing power’, which IMO rather deflates the feelings of specialness) is in fact the most fitness-promoting trait, or nearly as much of a generic force multiplier as some seem to think. Humans—presumably the most intelligent species, going by how very impressive we are to ourselves—are on top now (in terms of newly-invented abstraction ‘environment-optimization power’; we don’t have the most biomass or the highest population, we haven’t had our modern form the longest, we aren’t the longest-lived or the fastest-growing, etc.), but that doesn’t mean we’re somehow the inevitable winner of natural selection; I think our position is historically contingent and possible to dislodge. Moreover, I don’t think intelligence is the reason humans have such an inordinate advantage in the first place! I think our advantages descend from cultural transmission of knowledge and group coordination (both enabled by language, so, that capacity I’ll agree seems plausibly quite valuable).
Sometimes people point to the many ants destroyed by our construction (the presumption being that this is an example of how intelligence makes you powerful and dangerous). But the thing is, many species inadvertently kill ants in pursuit of their goals; I really think the key there is more like relative body mass. (Humans do AFAIK kill the most ants due to the scale of our activities, but if ants were twenty stories tall all our intelligence would not suffice to make it easy.)
Similarly, I am more skeptical about optimization than typical; it seems to me that, while it might be an effective solution to many problems, it is not the be-all and end-all, nor even so useful as to be a target on which minds must tend to converge. You’ll note that evolution has so far produced no optimizers;[6] in my opinion optimizers are a particular narrow target in mindspace which is not actually that easy to hit (which is just as well, because I don’t think they’re desirable; I think optimizers are destructive to anything not well-captured by the optimization target,[7] and that there are few-to-no things which it’s even good to optimize for in the first place). Moreover, I think an optimizer, in order for its focus to be useful, needs to get the abilities with which it optimizes from somewhere, and as I’ve said I don’t think intelligence is a universal somewhere.
Also, it must be said, we haven’t actually built any of the mechanisms all this speculation centers around (no, LLMs are not AGI). I think if we did, we’d discover that they work much better in the frictionless vacuum of Abstraction than in real life.
I also have disagreements with the average lesswronger in the direction of being skeptical about AI takeoff in general, so, that’s an additional hill you’d have to climb to convince me in particular. Many of the more extreme conceptions of AI seem to me to rest on the same assumptions about intelligence equalling general optimization power that I am suspicious of in full generality. I am also skeptical of LLMs in particular because, well, I talk to them every day and my gestalt impression is that they’re really fucking stupid. Incredibly impressive given the givens, mind, often useful, every once in awhile they’ll do something that surprises or delights; but if these are what pass for alien minds I’ll stick with parrots and octopi, thanks all the same.
Passing readers! If you are not like this, then you damn well should be 😛
Maybe. In accordance with my lowered standards herein, I will be eschewing qualifiers for prettier polemic just as Land does.
Actually one of the stronger arguments for Land’s viewpoint, IMO; perhaps he secretly meant this all along and just had the worst possible choice of presentation for communicating it?[8]
To be clear, we do not.
An obnoxious rhetorical trick.
A fact which, to be fair here, actually inveigles in the direction of Land’s position.
Yes, if you simply optimized for a function encompassing within it the whole of human values everything would probably be fine. This is not possible.
If he meant anything like that it’s very possible you’ll enjoy nostalgebraist’s The Apocalypse of Herschel Schoen (or not, it’s a weird book); it features among other things a climactic paean to This Sort of Thing.
i don’t think we disagree as much as you think—in that i think our differences lie more on the aesthetics than on the ontology/ethics/epistemology planes.
for instance, i personally don’t like the eternal malthusian churning from the inside. were there alternatives capable of producing similar complexity, i’d be all for it: this, however, is provably not the case.
every 777 years, god grants a randomly picked organism (last time, in 1821 AD, it was a gnat) the blessing of being congenitally copacetic. bliss and jhanas just ooze out of the little thing, and he lives his life in absolute satisfaction, free from want, from pain, from need. of course, none of the gnats currently alive descends from our lucky fellow. i don’t think knowledge of this fact moves my darwinism from “biology” to “ideology”.
“adaptive” not being a fixed target does not change the above fact, nor the equally self-evident truth that, all being equal, “more intelligence” is never maladaptive.
finally, i define “intelligence” not as “more compute” as much as “more power to understand your environment, as measured by your ability to shape it according to your will”.
does this bring our positions any closer?
Responding to the disagree reaction, while I do think the non-reaction isn’t explained well by selfishness and near-term utility focused over long-run utility, because I do think they’d probably ask to shut it down or potentially even speed it up, I do think it predicts the AI arms race dynamic relatively well, because you no longer need astronomically low probability of extinction to develop AI to ASI, and it becomes even more important that your side win, if you believe in anything close to the level of power of AI that LW thinks, and selfishness means that the effects of generally increasing AI risk don’t actually matter until it’s likely that you personally die.
Indeed, this can easily go to >50% or more depending on both selfishness levels and how focused you are on the long-term.
There is also a minority who are genuinely pro human extinction
What is the highest probability of extinction through which a wise man would be willing to proceed? As i see it, any probability greater than 1:10,000 in 100 years is absurd. Maybe 100,000 is more like it. One in a million would be reasonable, from the perspective of Pascale’s Wager. (Not accounting for potential benefit on the upside.)
I’m interested in the numbers other folks would be willing to tolerate.