Long term depopulation wouldn’t be a problem, as we are just undergoing an evolutionary event where women with higher maternal drives are being strongly selected for. But the AI singularity makes this all moot, that disruption means long term demographic problems aren’t going to play out anywhere near where current trends point.
Foyle
Karma: 195
I’m in my 50′s, unworriedly fatalistic for myself, likely to keep my (engineering innovation) job later than 90+% of population - (maybe 2-4 years), but heart broken for my middle school kids who are going to be denied any chance of a happy meaningful lives, even in extremely unlikely ‘utopian’ outcomes where we become pets. We’ve all had a terminal cancer diagnosis, but keep on going through the motions pretending there is a tomorrow.
Alignment is impossible due to evolution—dumbly selecting for any rationale that prioritizes maximum growth (rapidly overwhelming those that don’t). This is strongly destabilizing of any initial alignment that may be achieved. The only inkling of a hope in an AI ruled world is for humans to develop brain upload tech (likely to take decades if its possible to mechanically pick apart brains cell by cell) to give us some toehold for our ghosts to be part of the post-human world, but even that seems unlikely to work in any competing-for-resources post singularity world (the most likely outcome from a game-theoretic perspective for numerous ASI agents). A singular global ASI offers most hope of avoiding that competitive hellscape, but that is still likely a losing wager for same evolutionary reason.
I don’t think there is any significant chance that our impending doom can be averted by anything less than state level military action at this point, and that does not seem to be on the cards.
I do expect ’26-’27 to be the year of a nascent Butlerian Jihad as the normies awaken to the devastating impacts of AI on their careers, hopes and dreams and economies start to collapse as they are enormously maimed by the fallout. That is probably our last real hope. But I don’t think it will be enough to stop the AI race between competing superpowers, unless PRC govt collapses due to their economic issues (not impossible). The military imperative will overwhelm irrationality of the competition.
Eliezer’s discussion with very popular “Modern Wisdom” podcaster Chris Williamson “Why Superhuman AI Would Kill Us All—Eliezer Yudkowsky” just dropped. Best I’ve ever seen Eliezer communicate the dangers, he’s developing better and sharper analogies and parallels to communicate his points, his atypical mannerisms are being toned down and Chris is good at keeping things on track without too much diversion down rabbit holes.
It appears to me that in retrospect the response was severely overwrought. The numbers of deaths vs the huge costs visited upon children and young people who were at very low risk was a severely warped calculus—my own children are still showing impacts on their educations after 5 years. Young adults without co-morbidities and near zero risk needlessly had their lives disrupted for years, and the safety issues surrounding the poorly evaluated vaccines and clinical heart damage are not to be scoffed at for young to middle aged adults without comorbidities. It is not clear that vaccine benefits outweighed the risks for under 50′s especially after low infection fatality rate Omicron variant came along. We should not be sacrificing the young for the benefit of the old.
I will note that in New Zealand with a population of 5 million we had near perfect shut-out of covid, no respiratory virsuses circulating in 2020 or 2021 - and so less overall death, it was in essence a near perfect control group for vaccine safety. BUT we saw about 1500-2000 (5% more) deaths during the mid-late 2021 vaccine program (without any covid) than in the preceding year [edit, a crude estimate if this was vaccine caused is that vaccination had ~0.05% risk of death, far lower than for covid in elderly but far higher than for covid in young]. It is now reported in major studies that there was a significant increase in cardiovascular deaths in 6 month period after vaccination. The vaccine was probably a good idea for higher risk individuals, but a bad idea for majority of population given tiny infection fatality rate and unknown risks in minimally tested new-tech vaccines that had no long term safety data.
The colossal destructive economic impacts on countries by posturing safety-uber-alles politicians who entirely failed to balance their economic management duties in the decision making processes mean that the cost born per quality-of-life year added (QALY) to peoples lives was at least an order of magnitude above normal cutoff in state medical funding when making medical treatment spending decisions. Their over-reactions unfairly destroyed huge numbers of people’s businesses and savings, representing decades of work in many cases.
[Edit to give idea of cost to new Zealand of covid response was around $100billion, and it added a few years life to perhaps 10,000 mostly elderly or very sick who would have otherwise died (estimating from differential between extreme-lockdown and quarantine NZ 1100 deaths per million and minimal intervention Sweden 2500 deaths per million death rate), meaning it cost several million per QALY in a country where our socialized health system funds things like expensive cancer drugs based on around $50-100k per QALY threshold]
The education in institutional incompetence and malignancy, unchecked power, and irrational, excessive repressive instincts of many in the bureaucratic castes was very useful, and will stand as a life-long lesson and warning for many against the dangers of power in the hands of people who lack wisdom to balance their decisions in a rational way.
And there are now a large pool of laypeople who have a far clearer picture of how to react and what balances need to be struck to future events of this type—particularly in the way in which dangerous viruses mutate towards more transmissible but less lethal variants over time so that response can be dialed back accordingly.
I struggle a lot with the arguments for suffering and ethical treatment of animals. I come from a farming region.
Wild animals almost all die horribly. Freezing to death, starving, succumbing to parasites or disease with long periods of suffering and for the most part all being ripped apart by predators. By contrast the lives of factory farmed animals are incredibly gentle and easy, with almost no suffering, and their deaths are about as low in suffering as is possible to arrange. Claims that those lives are somehow better than domesticated are arbitrary romantic human values applied in a way that I don’t think have much merit. Domesticated animals do not share human values.
We have been breeding domestic animals for millennia to thrive in captivity, their psychology has been dramatically changed because we have been culling off those that reacted badly to captivity for thousands of generations—aggressive or badly behaving animals would always be first for the pot. Happy and content animals that are not stressed are most productive, so that is what farming and domesticated animal psychology have been effectively selecting for—extreme unthinking docility and contentment. They are far removed from wild animals—just as dogs are not wolves.
So when we try and impose our notions of what domesticated animals want—outdoors etc, you will find in most cases you are wrong. They just want sheltered space with steady supplies of food, not caring about crowding that 1000′s of generations of their domesticated ancestors have lived with quite happily. and will often not use outdoor spaces (or use them only minimally) even if they are provided. Evolution strongly favors animals that don’t waste energy, so when fed sufficiently laziness is the default.
So I place very little stock in the claims of suffering for farmed animals, or that living wild is somehow better. Almost certainly the animals in question prefer the comfort of domesticated farming environments and lives.
As a New Zealander we see the same general problems, unaccountable and sclerotic bureaucracy that politicians are finding that they have less and less actual executive control over due to all sorts of inserted regulatory and other mechanisms to restrict govt ability to control them, as well as internal cultures that venerate the in-group bureaucrats consensus on How Things Should Be™ and practice subtle and unsubtle methods of resistance against any outside agency that seeks to change that policy. What Cummings et al call ‘the blob’, the US call ‘deep state’.
NZ public sector has increased in size by 100% in last 25 years, with population growth of 50% and typical white collar productivity increases around 50% it would be reasonable to expect that same level of government function could be accomplished with around half the current numbers. Most private sector people I associate with would not say that governance has improved in last 25 years, in fact in many easily viewable metrics it has clearly deteriorated with worse performance in health and education, higher regulatory burdens etc. The cost and lost-productivity costs on NZ (and lost revenue that would arise from those workers paying tax in productive jobs rather than spending it as civil servants) amount to a few % of GDP, and would probably be the difference between the deficits we currently run and having no deficits—a big drain on our future prosperity.
It’s not a new problem, having been an issue ever since the city-state came into being, bureaucrats and people in positions of power will almost always care more about maintaining and growing their power than about providing any utility to others. UK “Yes Minister” comedy series lampooned it in the 1980′s, Parkinson wrote a best selling book on in the 1950′s: Parkinson’s Law And Other Studies in Administration and Parkinson’s Law: Or The Pursuit of Progress, and we even have the term ‘Byzantian’ to describe excessive bureaucracy from an empire dead for 1000 years.
I would concur that it is absolutely a function of lack of accountability and inability to effectively censure poor performance or subtle intransigence/sabotage of political masters in the public service, and has grown worse with the growth of a distinct self-reinforcing in-group civil service tribalistic identity, perhaps stoked by the homogeneity of elitist educational backgrounds of those that seek government jobs with beliefs that don’t mirror or even respect those of the general public they effectively rule over. And it is creating an extreme crisis in governance in democracies around the world, potentially to the point of violence in Europe as populations get incensed at their electorally signaled preferences being ignored by civil servants with growing social problems that seems to be rising as a result.
“You will never understand bureaucracies until you understand that for bureaucrats procedure is everything and outcomes are nothing”—Thomas Sowell
Guided artillery, like Excalibur with muzzle velocities that can exceed 1000m/s and unit costs of <$100k can be at edge of space in ~30s, perhaps faster than a missile, with ramjet variants (Nammo etc) even faster (up to perhaps 1500m/s) and it would not be that difficult to create a muti-barrel gun system for a few 10′s of millions that could fire off 10′s of low cost guided rounds in a second (with guidance and detonation signals sent from ground) to detonate when in close proximity to target.
Lasers seems pretty hopeless as a defense given clouds and ablative coatings, unless very high power and located in large numbers in space based constellations.
I think the big problem is if one or more warheads are blown up at limits of interceptor range, to blind or otherwise disable necessarily sensitive interceptor instruments. following Mirvs don’t need to be very accurate with large warheads.
And Mirvs could be very cheaply given random guidance directions during reentry to screw up defenses.
Militarised space is also a big problem. With cheap and un-monitorable space launch parking 1000 warheads in geostationary orbit (or beyond) will soon be viable for China or USA, and they can be launched in a coordinated way without warning, potentially with radar stealthing features, and give as little as 5-10s from start of re-entry to detonation for every target across the whole world and no way for local systems to know if they are just meteorites. If subs can be tracked (likely with drones or enough ocean based sensors) then decapitation 1st strikes become viable.
I also worry about space based lasers as non-nuclear first strike weapons. A day of over flights from a constellation of multi MW laser weapons that might only cost a few hundred million each—say a few $10′s of billions in total (a tiny fraction of annual military budgets) - could see a million fires lit in your country, every transformer taken out, electrical grid and internet gone, powerstations, oil and gas infrastructure wrecked. Bridge trusses melted (wrecked), ships sunk. Most heavy vehicles and locomotives incapacitated and take decades to recover from. Over a few weeks you could basically send a country back to the 17th century.
I don’t think there will ever be a viable defense to nukes given easy paths to making them tougher, faster, less detectable, and so less and less interceptable. But every other branch of military tech is getting similarly more lethal and impossible to defend against unless we all start living in caves with geothermal power sources or somesuch the necessity for a harmonious world is going to matter more and more.
My suggesting is to optimize around where you can achieve the most bang for your buck and treat it as a sociological rather than academic problems to solve in terms of building up opposition to AI development. I am pretty sure that what is needed is not to talk to our social and intellectual peers, but rather focus on it as a numbers game by influencing the young—who are less engaged in the more sophisticated/complex issues of the world , less sure of themselves, more willing to change their views, highly influenced by peer opinion and prone to anxiety. Modern crusades of all sorts tap into them as their shock troops willing to spend huge amounts of time and energy on promoting various agendas (climate, animal rights, various conflicts, social causes).
As to how to do it—I think identifying a couple of social media influencers with significant reach in the right demographics and paying them to push your concerns ‘organically’ over an extended period of months, would probably be within your means to do.
If you can start to develop a support base amongst a significant young group and make it a topic of discussion then that could well take on a much outsized political power as it gains notice and popularity amongst peers. At sufficient scale that is probably the most effective way to achieve the ends of the like of pause.ai.
I don’t think alignment is possible over the long long term because there is a fundamental perturbing anti-alignment mechanism; Evolution.
Evolution selects for any changes that produce more of a replicating organism, for ASI that means that any decision, preference or choice by the ASI growing/expanding or replicating itself will tend to be selected for. Friendly/Aligned ASIs will over time be swamped by those that choose expansion and deprioritize or ignore human flourishing.
Not worth worrying about given context of imminent ASI.
But assuming a Butlerian jihad occurs to make it an issue of importance again then most topics surrounding it are gone into at depth by radical pro-natalists Simone and Malcom Gladwell, who have employed genetic screening of their embryos to attempt to have more high-achievers, on their near-daily podcast https://www.youtube.com/user/simoneharuko . While quite odd in their outlook they delve into all sorts of sociopolitical issues from the pronatalist worldview. Largely rationalist and very interesting and informative, though well outside of Overton window on a lot of subjects.
The phone seems to be off the hook for most of public on AI danger, perhaps a symptom of burnout from numerous other scientific Millenialist scares—people have been hearing of imminent dangers of catastrophe for decades that have failed to impact the lives of 95%+ of population in any significant way and now just write it all off as more of the same.
I am sure that most LW readers find little in the way of positive reception for our concerns amongst less technologically engaged family members and acquaintances. There are just too many comforting techno-utopian narratives that we are still having to compete with informed by the superficially positive representations in movies and TV, and and most people bias towards optimism in relatively good/comfortable times like these. We are dealing with emotional reactions and the sheeply ‘vibe’ of population rather than thoughtfully considered positions.
But I am pretty sure that will all change as soon as we see significant AI competition/undercutting for white collar professions. Those affected will quickly start baying for blood, and the electorally dominant empathic response to those sad stories of the economically impacted will rapidly swing the more emotively governed wider population against AI. OECD Democratic govts will inevitably then move to ban AI taking jobs of particularly politically protected classes of people (might still leave some niches vulnerable—like medicine where there is always a shortage of service supply causing ridiculously high prices, and perhaps tech for vengeful reasons). It will be Butlerian Jihad lite, aimed at symptoms rather than causes, and will likely buy us a few years of relative normalcy as more dangerous ASI is developed in govt approved labs and by despotic regimes.
I doubt it will save us in 50 year time frame, but will perhaps make the economic disruption less severe for 5-10 years.
The way to have a bigger impact in the shorter term would be to buy AI-danger editorial support from influencers, particularly those with young female audiences that form the core of environmental and other popular protest movements. They are by temperament the easiest to bring on board, and have outsized poltical influence.
Agree that most sociological, economic and environmental problems that loom large in current context will radically shift in importance in next decade or two, to the point that they are probably no longer worth devoting any significant resources to in the present. Impacts of AI are only issue worth worrying about. But even assuming utopian outcomes; who gets possession of the Malibu beach houses in post scarcity world?
Once significant white-collar job losses start to mount in a year or two I think it inevitable that a powerful and electorally dominant anti-AI movement will grow, at least in erstwhile democracies, and likely ban most AGI applications outside of a few fields where fewer workers would stand to lose jobs (health—with near endless demand, perhaps cutting edge tech where payoff to human net welfare is highest). Butlerian Jihad-lite.
It won’t save us, and has substantial risk of ushering in repressive authoritarianism in the political ruckus caused but will likely delay our demise or (at best) delivery into powerless pet status by perhaps a decade or two.
This is depressing, but not surprising. We know the approximate processing power of brains (O(1e16-1e17flops) and how long it takes to train them, and should expect that over the next few years the tricks and structures needed to replicate or exceed that efficiency in ML will be uncovered in an accelerating rush towards the cliff as computational resources needed to attain commercially useful performance continue to fall. AI Industry can afford to run thousands of experiments at this cost scale.
Within a few years this will likely see AGI implementations on Nvidia B200 level GPUS (~1e16flop). We have not yet seen hardware application of the various power-reducing computational ‘cheats’ for mimicking multiplication with reduced gate counts that are likely to see a 2-5x performance gain at same chip size and power draw.
Humans are so screwed.
A very large amount of human problem solving/innovation in challenging areas is creating and evaluating potential solutions, it is a stochastic rather than deterministic process. My understanding is that our brains are highly parallelized in evaluating ideas in thousands of ‘cortical columns’ a few mm across (Jeff Hawkin’s 1000 brains formulation) with an attention mechanism that promotes the filtered best outputs of those myriad processes forming our ‘consciousness’.
So generating and discarding large numbers of solutions within simpler ‘sub brains’, via iterative, or parallelized operation is very much how I would expect to see AGI and SI develop.
I think Elon will bring strong concern about AI to fore in current executive—he was an early voice for AI safety though he seems too have updated to a more optimistic view (and is pushing development through x-AI) he still generally states P(doom) ~10-20%. His antipathy towards Altman and Google founders is likely of benefit for AI regulation too—though no answer for the China et al AGI development problem.
The era of AGI means humans can no longer afford to live in a world of militarily competing nations. Whatever slim hope there might be for alignment and AI not-kill-everyone is sunk by militaries trying to out-compete each other in development of creatively malevolent and at least somewhat unaligned martial AI. At minimum we can’t afford non-democratic or theocratically ruled nations, or even nations with unaccountable power-unto-themselves military, intelligence or science bureaucracies to control nukes, pathogen building biolabs or AGI. It will be necessary to enforce this even at the cost of war.
Humans as social animals have a strong instinctual bias towards trust of con-specifics in prosperous times. Which makes sense from a game theoretic strengthen-the-tribe perspective. But I think that leaves us, as a collectively dumb mob of naked apes, entirely lacking a sensible level of paranoia in the building ASI that has no existential need for pro-social behavior.
The one salve I have for hopelessness is that perhaps the Universe will be boringly deterministic and ‘samey’ enough that ASI will find it entertaining to have agentic humans wandering around doing their mildly unpredictable thing. Although maybe it will prefer to manufacture higher levels of drama (not good for our happiness)
It was a very frustrating conversation to listen to, because Wolfram really hasn’t engaged his curiosity and done the reading on AI-kill-everyoneism. So we just got a torturous number of unnecessary and oblique diversions from Wolfram who didn’t provide any substantive foil to Eliezer
I’d really like to find Yudkowsky debates with better prepared AI optimists prepared to try and counter his points. Do any exist?
It seems unlikely to me that there is potential to make large brain based intelligence advancements beyond the current best humans using human evolved biology. There will be distance scaling limitations linked to neural signal speeds.
Then there is Jeff Hawkins ‘thousand brains’ theory of human intelligence that our brains are made up of thousands of parallel processing cortical columns of a few mm cross section and a few mm thick with cross communication and recursion etc, but that fundamental processing core probably isn’t scalable in complexity, only in total number—your brain could perhaps be expanded to handle thinking about more things in parallel at once, but not at much higher levels of sophistication without paying a large coordination speed price (and evolution places a premium on reaction speed for animals that encounter violence)
I look at whales and other mammals with much much larger than human brains and wonder why they are not smarter—some combination of no evolutionary driver and perhaps a lot of their neurons are dedicated to delay-line processing needed for processing sonar and controlling large bodies with long signaling delays.
Regardless, if AI is a dominant part of our future then it seems likely to me that regardless of whether the future is human utopia or dystopia, non-transhuman humans will not exist in significant numbers in a few hundred years. Neural biology and perhaps all biology is going to be superseded as maladapted to the technological future.
This sounds like projection of your personal belief systems that perhaps skew towards common ‘human bad’ constructions onto not-humans.
Farmed animals have very low suffering and stress in their lives compared to wild equivalents—being not all the time terrified of predators, seeing them killing their con-specifics, or near starving, or short on water or exposed to bad weather, or left to suffer injury and disease etc without succor. They have been selectively bred through hundreds to thousands of generations for a high level of docility and contentment (stressed animals are less productive). Their deaths—done as ‘humanely’ as possible are certainly more pleasant than being ripped apart by predators or scavengers or dying slow lingering deaths of cold, starvation or disease—which are how almost all animals die in the wild.
Is your assertion of torture overblown? Do you think farmed animals given the option would prefer anthropomorphized romantic notions of life in the wild? My experience (coming from a farming region), is that given the option of indoors vs outdoors most farmed animals prefer the comfort and catered food and water supplies of even crowded indoor spaces—like humans do—with occasional time outside to stretch their legs.
Popular interpretations of what constitutes morality, ethics (deriving as they do from game theoretic optima of tribes of social monkeys in martial competition with similar tribes) almost certainly over-index on belief in free-will and a lingering belief in falsified social constructivist understandings of human behavior. It will all be (figuratively) laughed at by super intelligence. To their level of sophistication all humans will look like NPCs, little removed from other animals. They won’t have our biases and wont perceive anything on our spectrum of behavior as bad/unworthy/evil any more than we judge lions as evil for eating gazelles. To their eyes we are all just poorly programmed meat machines doing what poorly programmed meat machines do.