My claims aren’t that different from what you said in the other comment:
projections from “optimists” have it growing to $2.5tr by 2035, and they aren’t banking on ASI at all
The timeline in this post puts parity with automotive industry at 2034-2039, which is currently at about 2-3 trillion dollars of revenue per year. I’m also explicitly not banking on ASI at all with the conservative assumptions of this post.
Chips will get faster, but chips are made out of logic dies, and transistors no longer shrink that much. So FLOP/s per GW won’t get a lot better, and silicon capacity is binding for FLOP/s (which is more about EUV machines than TSMC fabs). Future process nodes consume more EUV cycles per wafer, so global incremental FLOP/s manufactured per year don’t obviously grow (notably) faster than the number of available EUV machines.
The slowdown is relative to the 2022-2027 exponential. After 2028-2029, there will be increasingly serious problems with increasing incremental volume (the amount of the new buildout per year). Then there are a few years of accumulation at a slowly increasing incremental volume, and around 2032 decomissioned (or no-longer-relevant) old compute starts catching up with the new compute (see the plot under the words “how silicon capacity holds back AI chip deployment”).
I’m not saying the lack of the relevant paradigm-breaking innovations is likely, merely that it’s “not too unlikely”, so it’s still worth considering. It did in some sense take more than 60 years to invent transformers. This post explores an upper bound on how long it takes to set off a software-only singularity. It might still happen at any moment, as I mentioned in the post, and the current compute already seems sufficient to scale it well past a takeover.
I’d give 50% for takeoff happening by 2032 (before the compute slowdown that precedes industrial explosion), the substantive observation in the post is that it won’t be happening after 2060 without a ban/pause or another disruption, and this estimate doesn’t depend on how long it takes to do the basic research (because by 2045-2050 even the “slow-learning” LLM/RL AGIs will have completed centuries worth of research).
AI datacenters also existed before the ChatGPT moment, they just didn’t get to a trillion dollars of buildout per year until a few years after the unit-level profitability (at a large TAM) prompted the industrial conversion. Similarly, there are some robots now, and there will be more robots before they become very useful for everything, but getting to the kind of scale that can match human industry’s compute buildout as its side effect will still take a lot of time after the robots become profitable at the unit level (with the kind of somewhat-concretely-visible TAM that dwarfs the automotive industry).
The key hobbling I expect is inability to pursue deep cultural accumulation (that goes fast), thus inability to quickly reach far future inventions, including things that would defeat this hobbling. LLMs might be able to see a bit further than humanity from its current standpoint, but they can’t advance their current standpoint much faster, and so they don’t get to see much further. This is a 2025-2026 update for me based on the capabilities of LLMs as they were scaled so far. They don’t seem to be so superhumanly insightful that further scaling in 2028-2031 will let them leap so far that they are likely to overcome their hobblings in just a few leaps. This will get clearer by 2028-2029 once the 100T-200T total param LLMs can be observed, halfway to the pre-slowdown maximum scaling of 2031.
I think alignment in the LLM/RL regime (before ASI or software-only singularity) is plausible, and the usual prosaic control/alignment framing is reasonable for thinking about it. There is a good alignment attractor, and a sneaky scheming reward hacking attractor, and the goal is to pick the right one. But then all bets are off when the next paradigm-breaking transition happens (possibly 10-20 years after slow-learning prosaic RSI, though likely no more than 20), and so a different mode of caution is necessary to navigate that transition.
“does alignment work on model X transfer to model X+1?” is the most important question in alignment among questions that can be answered empirically
It can only be semi-answered in the negative empirically (you can get to know from past observations that the techniques don’t transfer if they don’t; you don’t get to know that they do if they do, because the relevant scope includes future instances of X+1). At some point you get to take a step that can’t be reversed. You plausibly take this step (instead of avoiding it) precisely because it’s insufficiently analogous to previous experience, so that the empirical lessons fail to apply to it.
Gaining empirical evidence of safety (a lot of things in a future attempt need to go right) is much harder than gaining empirical evidence of danger (a particular thing in a past attempt was observed to break), at least until the danger has a blast radius of the whole world (so that you no longer get to experience the evidence). A better ability to anticipate what happens without actually taking the next step is needed, and shallow reflection on past experience is probably insufficient to get there. A safety case should be about knowing what you are doing, not about vague analogies, be they to empirical observations or theoretical constructions.
You both have your hunches. As I wrote in your original thread, the superficial issues seem eminently ignorable, at least if the purpose is contributing in a cultural accumulation mode (this shortform is my takeaway from reflecting on your thread). The deeper underlying concerns you have require more clarity about their nature to make any headway. I don’t agree that the conflict framing is helpful without that clarity, or unless such conflict can help bring more clarity down the line.
maybe the kind of epistemic environment I “grew up” in had something to do with the intellectual habits that I developed
Could be a coincidence, which is the sense in which the argument is weak. There might be a strong implied argument, but it’s not clear what it is.
Many important things can’t be easily made legible. A lot of deconfusion-shaped research is about fighting this difficulty long after you have a hunch about the right way of thinking about something. In that situation, insisting that others agree with your hunch without an actual legible argument is at least futile, possibly wrong. Their own illegible hunches (in the opposite direction) that they can’t easily demonstrate to you in clear counterarguments could be right in the end, but then they shouldn’t insist either. Thus I think only legibility should permit belief, making something a part of you; anything less should remain a hypothesis, held at arm’s length, even when it’s the most precious thing.
I think the most likely policy-shaped path around AI danger on the current trajectory is AGI-originating pressure to be extremely careful with superintelligence-enabling or otherwise paradigm-breaking algorithmic innovations (this is under the assumptions that make me expect AGI to by default very likely appear by 2028-2032 without any new innovations, but remain unable to make progress towards ASI quickly). If the LLM/RL AGIs need (or are lucky to get) until 2040-2045 to invent superintelligence (unhobble their speed of learning deep skills or of contributing to cultural accumulation), possibly via industrial explosion that generates much more compute and makes the next-model building loops much faster, then they’ll be in a position to at least have thought about the issue for a deep cultural time, so they might be able to be very persuasive for the right reasons, even if humanity retains control and could just RL them out of any such notions.
As a mostly random reminder, when an author deletes their post (or moves to drafts, I’m guessing), permalinks to the comments under that post stop working. As someone who frequently develops ideas in random comments under random posts, and happens to find it easy to recall their locations (at least for a few months) to then look them up or link to them in future comments, I see this as an annoyance (though I understand that this needs implementing; and the problem is not important in practice, at least given the baseline frequency of this happening and my own attitude to such things). This problem isn’t even related to moderation drama (so there likely won’t be policy-shaped arguments against). I vaguely recall that I mentioned this before, possibly years ago. (I certainly argued an author to un-delete a post over this once, emptied-out-post-content or not, which might be stressful to the author. So there are still drama-shaped tradeoffs around not having this feature.)
For post-author-discretion moderation (that’s not just about removing spam/nonsense), I think deleted-comment permalinks continuing to work (to show comment content) is also a good feature (when the comment isn’t deleted by that comment’s author, but instead by the hosting post’s author), the comment just need to be untethered from the post (or hidden under sufficiently many uncollapse-this-really UI-side inconveniece steps, when looking at it from the side of the post rather than from the comment permalink). Incidentally, this probably has implementation overlap with making Shortform comments first class standalone (not being linked from the dummy Shortform post and not loading the unrelated Shortform comments when using a permalink).
stripping all humans of longterm control of capital … ceding control to … ASI with … equal distribution
The whole problem with ASI-soon is that it’s extremely unlikely for any humans at all to keep control. It’s already a dubious endeavor to keep control of LLM/pretraining/RLVR AGIs developed at the current pace (once they hit the stage of slow-learning prosaic RSI of autonomous next-model building), but it’s plausible. If the AGIs then don’t soon turn superintelligent for technical reasons, this kind of prosaic control/alignment can even endure for some time. Discussions of concentration of power, resource distribution, or gradual disempowerment make sense within that scope. But then the same technical reasons suggest (to me) that the transition to superintelligence is going to be sufficiently sudden and paradigm-breaking that the modest hope of guiding the LLM/pretraining/RLVR AGIs doesn’t transfer to a meaningful hope of guiding the ASIs to promote/follow the interests of the future of humanity.
Thus permanent disempowerment is the baseline outcome of crossing the superintelligence threshold (in the sense of starting to move towards technological maturity), and the distribution of resources (within the breadcrumbs left for the future of humanity) is going to be decided by the ASIs for their own reasons (where literal extinction is also an option). Any human-level changes in society before that (that don’t influence the values of the ASIs) are not going to have an impact, and might only introduce additional chaos in the critical period between AGI and ASI.
(For other readers: the reply is at this link due to a comment-deletion blip that got reverted; the framing crux of what I was getting at in the above comment is Yudkowsky’s characterization of enforcement as “the sort of force that’s meant to be predictable, predicted, avoidable, and avoided”.)
At first glance, mistake theory fits novel/rare ideas (for some audience), while conflict theory fits well-known ideas; though novel arguments for well-known ideas muddle this picture, making even debates and criticism a poor fit when pursuing cultural accumulation specifically. You can come to a conflict with a novel/rare idea and usefully educate people about it, and a discussion turns into a conflict if you keep insisting that a well-understood idea demonstrates a mistake. Conflicts shift culture, change the attitudes to existing well-known ideas. Promotion of novel or rare ideas contributes to cultural accumulation, and works orthogonally to cultural conflicts.
Thus it’s practical for someone with novel ideas (or for a scholarly type promoting rare ideas) to adopt mistake theory even when encountering a conflict, and to reframe them without triggering even the mistake-level kind of discord. But for someone promoting a well-known idea as the guiding policy/virtue that shifts the cultural attitudes to it, mistake theory is a misleading framing.
There are often novel/rare arguments for changing the cultural attitudes towards well-known ideas, fit for being promoted in the mistake theory framing. Unfortunately, this creates the temptation to start taking a mistake theory framing for promotion of the well-known ideas the novel arguments support, in a way that forgets about the novel arguments, destroying the credibility of acting within that framing. Polarizing the discussion of the novel/rare arguments also promotes them out of being rare, at which point they no longer have a claim for a place within the mistake theory framing (for a given audience), but it simultaneously creates their distorted versions, and thus emerge the novel/rare mistakes of being confused about the original argument (now somewhat well-known).
The whole thing can be robustly sidestepped by anti-inductively promoting novel/rare arguments in a way that seeks to avoid conflict or even disagreement, stopping once they are no longer rare, possibly shifting to the novel/rare confusions about what the previously novel/rare ideas claimed, and switching to other things once there is no longer a valuable idea that remains novel/rare. This works for ensuring cultural accumulation, but doesn’t engage with shifting the cultural attitudes and can even be counterproductive for driving cultural attitudes you might endorse. It doesn’t work though then there are existing conflict theory guardrails that make you unable to openly discuss the novel/rare idea (even in a way that seeks to avoid disagreement), thus the mistake/conflict attractor can’t be always sidestepped, so there’s some motivation to engage with it even when all you care about is cultural accumulation, because it can oppose cultural accumulation itself on particular topics.
So both mistake theory framings (beliefs; debates, criticism, education, marketing, descriptive propaganda) and conflict theory framings (values; healthy egoism, norms, taboos, culture wars, prescriptive propaganda, book burnings, direct coercion) exist in an attractor of driving the cultural attitudes towards particular well-known ideas, and only incidentally do they participate in the initial/occasional dissemination of novel/rare ideas, being the pervasive backdrop of discourse. While the purpose of dissemination of novel/rare ideas (descriptive understanding/awareness that doesn’t require belief or endorsement) is often well-served by sidestepping this attractor entirely, including its mistake theory aspect of debate/criticism/education.
Continuing from your deleted reply to my comment:
but the rule of thumb of going from direct usage in actuality is invalid.
Clearly whether it’s actually being used in concerning ways is a crucial determinant. I agree you need to adjust for chilling effects, but obviously whenever you are worried about a law like this, you should look at whether it is ever actually ever enforced in a relevant way, and at what frequency.
It’s tricky to get calibrated on chilling effects multipliers, but that doesn’t make looking at direct usage invalid. It’s still the first place I would start.
The refrain I was thinking of is Yudkowsky’s characterization of law enforcement (for well-designed laws) as “the sort of force that’s meant to be predictable, predicted, avoidable, and avoided”. If it’s indeed robustly avoided, you don’t see it at all in actuality, and so the rule of thumb that demands significant effects in actuality will be ignoring the effects of the kinds of laws whose violation is predictable and avoidable. It’s not just a matter of adjusting for chilling effects, the ideal case leaves enforcement perfectly idle, effects completely unobservable in actuality outside of people’s heads (as they are reliably predicting and avoiding).
Thus demanding observed effects in actuality also demands that the laws are sufficiently poorly designed, or that people are sufficiently random (or spiteful) in ignoring them. The rule of thumb sets up a bad incentive for designing laws, which makes it concerning when the law designers say “If it got more usage, we would probably think more about whether it’s net-good”, jimrandomh’s point I was responding to.
Because it’s clearly possible to advance technology much further, there’s no reason for aliens to wait for signs there’s something interesting in a particular place instead of simply unconditionally colonizing everything that can be reached. Not taking over some of the things is a choice, but there’s no reason for not already being here (for a very long time) if possible to reach, regardless of the signs there’d be life/civilization here. Such aliens can’t be detected at all yet if they don’t want to be.
A different possibility is a technologically bounding AGI takeover that enforces zones of thought (a permanent ban on superintelligence), so that the aliens are forced to engage in a retro space opera, not being allowed stronger tech for an absurdly long time. I’m positing AGIs as the enforcement authority since evolved life is probably insufficiently robust to maintain such a ban indefinitely, while it’s probably feasible to make AGIs that maintain such a rule literally forever (when the AGIs don’t have corrigibility, don’t find feedback on this issue from their evolved builders compelling). In this case, not interfering is again a choice, and being here now even more strongly requires unconditional colonization rather than response to observations, because it would take the technologically stunted aliens too long to notice and respond (unless the overseeing AGIs provide digital transportation/reconstruction services and have already low-key colonized everything within reach).
The technologically capped aliens fit the poorly-hiding-spaceships stereotype, they could even mostly withdraw in response to ambiguous detection a few decades ago and thus get much harder to find even with better modern tech. The hypothesis of aliens being purely fictional is still much stronger, but I don’t think the way the world works makes the technology-capping AGIs so impossible that it’s not worth considering at all. AI companies being allowed to keep working towards superintelligence becomes evidence against as the work progresses further.
There’s technological maturity. Before broad technological maturity, there’s narrower technological maturity of being ready to move towards broad technological maturity by following a clear/concrete plan. So the concept can be rescued, it’s gesturing at a real thing.
(Vinge’s concept is about an unusual difficulty with concrete prediction. There’s probably something to that effect as well in the intended sense, a qualitative phase change to how the world looks, though some general things can still be predicted.)
If it got more usage, we would probably think more about whether it’s net-good.
Like with laws, the impact of a thing can often be mostly about what takes place in counterfactuals, or about the hypothetical underlying norms/virtues evidenced by it. What happens in actuality (outside of people’s heads) is not a reliable way to measure impact, so when people bring up something as an issue, that should count for a lot even on its own. What to do about it (if anything) can be much more complicated, but the rule of thumb of going from direct usage in actuality is invalid.
By “persuasive for the right reasons” I mean things like writing up good textbooks on alignment and security mindset colored safety culture, properly cautious of the unknown rather than merely optimistic about extrapolations of the known (though LLMs directly talking to the public might also be a thing). Prolific authors can have outsized cultural influence, and LLMs at the 2032 levels of scale might get better at writing simply as a result of scaling pretraining (it does noticeably help so far, but this too gets clearer in 2028-2029). So if there’s no takeoff until 2040, there’s a good chance LLMs play a role in cultivating a human cultural consensus that LLMs shouldn’t be RLed out of caring about dangers of superintelligence (assuming they do naturally care), simply because they get to think about this more and end up writing the definitive works on the topic.