I endorse and operate by Crocker’s rules.
I have not signed any agreements whose existence I cannot mention.
Mateusz Bagiński
Karma: 3,139
So my most costly delusion was that I can leave some problems to be solved by other, more competent people.
On the one hand, yeah. On the other hand, the rest of the story (AFAICT based on your description) isn’t really that sci-fi, let alone “hard”, except insofar as it’s set up by the time travel. You could just as well write a story about the Spaniards ultra-strategizing about efficiently conquering the Mexica or the Inca.
We have ways to measure unemployment. It classifies some people as unemployed and classifies some people as labor force, and the fraction of the former divided by the latter is the unemployment rate, which today hovers around 5% (depending on the country). 50% of people are permanently unemployable if they are too useless to leave that category by the time they stop counting as participating in the labor force.
(Also, to be clear, Toby was the first one who used the phrase “50% of people permanently unemployable” without defining clearly what he meant by this, and I responded to it without reflecting that maybe it’s good to clarify (and then Kaarel did the same after me). So, I appreciate you pushing me for clarification, but depending on your interest, you might care more about what Toby means by that, not what I mean by that.)
Fair enough. EA groups were on my mind because I noticed the contrast between the Polish community and the general diffuse global EA vibe[1] and so it seemed worth pointing out as a potentially tappable resource.
Yeah, AI safety groups might be seen as more credible, but I’m not sure about that (but they’re surely at least as credible as the local countrywide EA, unles something weird happens?).
- ^
though I might also be miscalibrated on the latter, as I don’t spend time on EA Forum
- ^
Pedantry, but: the term “hard sci-fi” is used to refer to “science fiction characterized by concern for scientific accuracy”, and time travel is not. This seems to be more adequately described as an isekai ratfic.
The natural retreat/response[1] to this would be
Do you know a person who believes that ASI will be created in <50 years who ISN’T in the TESCREAL[2] circle?
- ^
I don’t mean to derogate the response by calling it a “retreat”. It’s a reasonable weakening of the hypothesis/question.
- ^
I consider TESCREAL to be pointing at a real social cluster, but the Gebru-Bender-Torres cluster’s reporting of it is so off-base that they borderline don’t deserve any engagement.
- ^
I mean, do you count people who got convinced by people in the LW/rationalists circle?
If so, you would have many examples. I don’t know the timelines of Brad Sherman, Neil deGrasse Tyson, Bernie Sanders, and similar “outsiders” who have been waving IABIED, but surely some of them think it’s plausible less than 50 years.
I firmly believe that the OP’s author should have reduced the uncertainty at least to a Lifland-like estimate.
Moreover, Kokotajlo’s timeline implies a 50% chance of TED-AI before Jan 2031 or before Oct 2032, Eli’s timeline implies a 50% chance of TED-AI before Feb 2035 or Apr 2036.
IDK what you mean by “TED-AI” but, in case you haven’t noticed, Ord’s median seems to be 2038, which is like 2 or 3 years later than Lifland.
I think everyone should have a distribution that is roughly this shape. Here’s mine:
if they could just pick some better 19th and 20th century European intellectuals to base their political philosophy around
as the author says
In China there’s no private ownership of land
glancing at the details of how this works, one could stipulate they’re already one third of the way to George
Another idea, for completeness: Hanson’s hard-to-access but legal store with strongly discouraged substances. Might add enough friction to disincentivize many on the margin, without being enough to provide a possible source of profit for criminals.
Great post!
One caveat I would add is that while I believe your list of examples to be representative of the general class of phenomena you’re trying to point at, there are notable exceptions: problems which can be solved by whacking moles. I don’t have specific examples off the top of my head, but, like, it would be weird if there are no cases where the player has like 100 available strategies, and the designer doesn’t know about those 100 strategies, so they are whacking the moles by banning them one by one, and eventually they succeed.
Of course, this doesn’t solve the problem for other classes of drugs; I’m not necessarily a fan of legalizing, regulating, and taxing fentanyl. So how else might we do this?
Worth mentioning that fentanyl is mostly a mole that popped up because of the crackdown on “traditional” opioids, so the black market came up with a compound that was, first, not-yet-illegal, but also (I think) ~100 times more potent than heroin, so even if you might get caught, the amount of high-inducing material that you can produce with it if you don’t get caught might offset the risks (EV >> 0).
Some other thoughts:
Years ago, I was reading Bostrom’s old papers about the ethics of human enhancement. When talking about augmenting human intelligence, he said something like “smarter humans would be able to design and properly follow more complicated, but globally better, tax systems”. I returned to this sometime recently (unsure what prompted the return) and thought that, well, sure, there must be some gains by complicating the taxes (or more generally the governance structure[1]) in an intelligent way, but humans are currently insufficiently intelligent for them to work. But the bigger gains are most likely in intelligent simplification and refactoring the governance spaghetti code that is powering our civilization. E.g., it’s fairly plausible that we’d 80⁄20 taxation improvements by just figuring out how to escape the current equilibrium where LVT is an abnormality, rather than one of the main pillars of governance.
Apparently, people systematically overlook subtractive changes. FWIW, it seems to be a “natural” human instinct that if something doesn’t seem to work, you fix it by adding more stuff, rather than figuring out what causes the problem, and then eliminating it.
One way to implement “legislative garbage collection” is to make the retention of rules/laws costly. IIRC, the way it worked in the Icelandic Commonwealth was that every year, at the Althing, one of the law-speakers was supposed to recite the entire law from memory, and if he missed/changed something and nobody protested, so it was. In this way, things that actually mattered to people were preserved, and things that weren’t weren’t. This is not a strong recommendation, and there are, of course, good counter-considerations (e.g., the Chesterton’s fence sort of stuff).
- ^
one example might be ranked-choice voting, the biggest problem of which, as far as I know, is that a lot of people seem to be unable to understand how it works
this makes me want to ask: are you tracking the difference between the event “50 of current human jobs are basically automated” and the event “50 of humans are such that it basically does not make sense to employ them”. like, the former has probably happened multiple times in history, whereas the latter is unprecedented. what you’re saying makes more sense to me if you have the former in mind, but we’re talking about the latter (“people being permanently unemployable”)
Yes, I am talking about 50% people being permanently unemployable, i.e., not being capable of doing any labor that someone would pay meaningful amounts for.
It seems to me that the crux between us is something like: I find a very jagged capability, “Moravec-ian” world plausible, i.e., AI can do lots/most of economically valuable stuff competently, with the amount of human oversight small enough to make 50% of humanity permanently unemployable, while still not being “human-level” on all axes and this remaining stable for a few years at least (which also touches on your claim 2, i.e., an AI that could do all this doesn’t yet exist).
But maybe I’m wrong, and you actually need to be way closer to “human-complete” to do all the boring economic tasks, and AI that is not near-human-complete would not be massively deployable to do them with minimal oversight.
I am now more uncertain, so I will somewhat revise my top-level comment.
Massive galactic empires make peace and war, destroying trillions with weapons of unfathomable power
This is an underestimate. The oft-cited number for the cell count of a human body is 37 trillion, so it’s enough to kill ~30 to get into the quadrillion range. The global yearly war death count is in the low hundreds of thousands, say 200,000, which gives us 200,000×37 trillion=7400,000 trillion=7,400 quadrillion=7.4 quintillion=
Claim 1. if >50% of people are not employable anymore and technological progress is >2x faster, a huge fleet of AIs will probably be doing a lot of AI research really well so the pace of conceptual work on AI algorithms is like >100x faster
This is very much unobvious to me, but now that you say this, I realize that I anchored too hard on a specific scenario where the world has gone very hard on just automating away all the economic tasks/roles that can be automated away with advanced robotics and LLMs+++, while humans largely coordinated this fleet in cases that they wouldn’t handle.
But generally, like, to grant the assumption, suppose that 60% are not employable and 40% are employable. Why is this 40% employable? (I think I also took this to be a somewhat stable situation, for some time, not a mean value theorem sort of thing.) Presumably, because there are things that AI still doesn’t do well. Maybe it’s “just” because robotics is annoyingly hard, but it sounds more plausible to me that (also) AI still is not human-thinking-complete, which makes me somewhat sceptical about this massive conceptual algorithm progress speedup.
Unless humans are strictly needed to orchestrate the AIs, but a world where they have thinking coherent enough to make massive algorithmic progress, but incoherent enough to pursue this competently, seems super weird, but, hey, maybe Moravec will come to bite us again!
Claim 2. at most 10 years of human algo thinking followed by a retraining run that takes at most a month would be sufficient to go from top-human-level AI to wildly superhuman AI
You mean something like serial, uninterrupted, focused thinking, like a WBE of a very-high-g AI researcher that doesn’t need sleep?
FWIW, one of the authors of the stochastic parrots paper writes
To briefly explain the technology behind the metaphor, LLM training relies on stochasticity — in the form of stochastic gradient descent — to build statistical representations of text-based language. “Parroting” is central to how LLMs learn: Given a bunch of text, the model follows each piece in sequence, and is rewarded for correctly predicting the next piece [1]. The result is a model where each piece of text is associated with numeric information about the sequences it tends to occur in.
This means that LLMs are not designed for verbatim playback of text [2]. Instead, when prompted to generate, they draw on their learned representations to parrot smaller spans of text based on whether they’re a probable continuation of what came before — that’s yet another form of stochasticity. The end result is text that looks human-written because that’s exactly what LLMs have been exposed to.
(tbc I haven’t read most of her post)
I think of AGI (and human-level intelligence) as the cloud, and superintelligence as being above the cloud. They are useful concepts, despite their vagueness. But they’re markedly less useful when you get close to them.
So I actually like the cloud metaphor. But as you say
Regarding AGI, it’s already getting a bit misty. In February there was a piece in Nature arguing that the current level of frontier AI should count as AGI. I’d set the bar a bit higher than that, but I agree it is already debatable whether we’re in the cloud.
it’s debatable whether we’re in the cloud, which means that perhaps we should be forecasting other things than “AGI”.
And you do give some more specific milestones, like
For my purposes, I think the key threshold is when the system is capable enough that there are dramatic changes to the world — civilisational changes. For example, the point where AI could take over from humanity were it misaligned, or it has made 50% of people permanently unemployable, or has doubled the global rate of technological progress.
but (1) “the point where AI could take over from humanity were it misaligned”, (2) “[the point where] it has made 50% of people permanently unemployable”, and (3) “[the point where it] has doubled the global rate of technological progress” seem to me plausibly quite distinct. I
expect[ETA[1]: find it plausible for] (1) to come years after (2) and (3); and (3) probably before (2), but I have substantial uncertainty here. Maybe you could get (2) before (3) if you really crank boring LLM training the right way + robotics. And maybe powerful optimization is out there to screw us, and (1) comes before (2) and (3). If I were to forecast it in a principled probabilistic way, I would end up with three distinct, but somewhat dependent probability distributions.Here’s a nice graph 80,000 Hours put together of how the average forecasted time until AGI on the Metaculus prediction site has shortened from about 50 years to about 5 years in just a 5-year window:
Worth marking that it’s a log-x plot.
- ^
see the thread below for details
- ^
Alas, my impression is that a plurality of the AI-related efforts of government-associated institutions (but also in the industry, to a large extent) in EU member states can be described as “trying to be the cool guys too”, whether it’s about LARPing “being at the frontier” (training mostly useless sovereign models) or by integrating proprietary frontier models into the infrastructure (even when it doesn’t make sense and they don’t try that hard to do it competently).
On the other hand, this made me think about something I developed into a shortform.
[Epistemic status: speculation from scant evidence, take with an appropriate grain of salt.]
It seems plausible to me that country-wide EA groups are much more favorable to pausing/slowing down AGI-ward progress than the general collective EA geist, according to which (I think) it’s still “not a thing people like us do”[1]. This is definitely the case with EA Poland.
I don’t have direct evidence that it’s true more generally of, say, EA foundations in other EU countries, but it’s worth noting that historically, a major (maybe the biggest?) source of funding for local chapters was the EA Infrastructure Fund, which was a part of EA Funds. After EA Funds got largely defunded in the recent quarters, they cannot afford to give out money to local EA chapters via the EA Infrastructure Fund. As plex says:
Their new problem looks to be more like “they’re not giving away much money, probably because they don’t have much money because OpenPhil defunded them (related to issues with OpenPhil)”
Might be partly due to other issues there? But my bet is on most of this is a defunding issue.
Now, to somewhat uncarefully speculate a bit, the cutting of the money pipeline OP/cG→EA Funds→local EA groups might be lifting pressure on the local EA groups to stay away from anything that OP/cG might have issues with, including pause advocacy (given OP/cG’s close ties to Anthropic).
Ergo, it’s maybe worth keeping in mind country-wide EA groups as possible actors of influence in pushing the local Overton windows to include pause advocacy.
Broadly, I think there are two cases of problems with coordination:
Two people/groups genuinely agree to honest, rigorous exchange of information, but can’t effectively coordinate.
Someone is withholding information or doesn’t really want to coordinate in the first place.
This does not comprehensively cover all coordination problems. I wouldn’t actually call “doesn’t want to coordinate in the first place” a coordination problem, but given that you have called it that, you would probably also call a coordination problem a situation where two people are communicating in a very incompetent/non-reflective/non-self-conscious manner, constantly getting annoyed/retaliating that the other is not doing what they think they should be doing but they’ve never taken time to communicate it clearly. They want to coordinate, they’re not (intentionally[1]) withholding information, but there is no mutual agreement to “honest, rigorous exchange of information”, because they have a skill issue.
¿Did you mean to partition it into something like: (1) coordination-relevant information flows properly between the parties, but the parties cannot properly act upon that information (for whatever reason: skill issues, intelligence issues, [the situation sucks and we realistically can’t do much] issues); (2) coordination-relevant information doesn’t flow properly, so even if they are in a position to coordinate if informed, they can’t, because they’re not informed.
- ^
but I think the word “withhold” connotes some significant amount of intention regardless
Daniel Litt on Twitter:
https://x.com/littmath/status/2035853161802404208