For me, a big part of the reason is that it went off topic immediately in an area that’s still charged by politics, so I didn’t want the discussion to centralize around something that would be predictably useless but very dramatic/distracting.
In essence, I’m trying to cut short a likely demon thread around COVID.
Alright, here’s my simple syllogism/list:
The use-cases of robotics beyond current robots, especially home robotics require handling lots of environments and have varied challenges, so it stress-tests AI generalization.
Unlike other domains, there’s no unexploited algebraicness/overhang that would allow AIs to be useful for these cases without generalizing, because if there were such a thing, very limited current industrial robots would already have been used. They haven’t, so there’s no room for algebraicness/non-generalization to be an issue.
It’s easier to control/exclude training data when the domain is physical, rather than digital due to a multitude of reasons.
For these reasons, if current AIs improve on robotic tasks, it is a much stronger signal that they’re generalizing, and therefore is a signal that AGI could come in 10-20 years, and depending on how fast they improve, this might shift to 2-4 years.
In particular, in this case you’re using some concepts here:
they have a positive but subhuman amount the property of effectively and continuously acquire deep knowledge and exploit this knowledge to construct and execute goal-directed plans over long lifetimes and consequence horizons/generalization.
And you’re doing an induction about it. And I’m just going to object that the way you’re using the concept doesn’t support the inference you’re trying to make; or at least, AFAICT you haven’t explained the concept and how you’re making the inference well enough to justify the inference.
So, the way I see Vanessa’s concept here is that it’s pointing to the concept of time horizons as defined by METR, which got introduced to the world by METR in 2025 called METR: Measuring AI Ability to Complete Long Tasks, and there’s a lot to be said here, including it’s problems, but the single most important thing to keep in mind here is that this is a continuous metric, so it’s meaningful to talk about how much the AI can acquire long-term knowledge to exploit it.
Now by itself, I would have viewed it as weak evidence, but combined with other evidence showing that exponential trends do exist in other domains, the fact that there are good general reasons to assume exponential growth in AI capabilities, plus the new robotics data shift me towards a view where the current LLM algebraicness/generalization issues are real, but probably fixable, and do not require throwing out the entire paradigm, and thus the induction/inference I’m making is valid here, or to put it another way the observations I have seen are more likely in worlds where LLMs can continuously acquire deep knowledge to exploit it than in worlds where they aren’t.
*Reads your model on progress to AGI, taking the definition of AGI in the post as given/assumed*
I very strongly disagree with this model of progress:
The first way humanity makes AGI is by combining some set of significant ideas about intelligence. Significant ideas are things like (the ideas of) gradient descent, recombination, probability distributions, universal computation, search, world-optimization. Significant ideas are to a significant extent bottlenecked on great natural philosophers doing great natural philosophy about intelligence, with sequential bottlenecks between many insights.
And instead believe in this model:
Algorithmic progress/good ideas/research taste do genuinely matter, but most people are prone to overestimate how much they matter without compute or data, and the main way they currently matter is by being more-compute/data efficient at larger compute scales, and importantly the current paradigm isn’t irrelevant to building the next one, because you can reuse the compute and data from the old paradigm, and the new paradigms will not reduce the demand for compute.
This means progress to AGI isn’t as serial as you state (though I do expect serialness in a different way through RL, but you can still reuse compute/training data you already bought once you have a better paradigm).
More generally, what has made AI go from being ~completely useless and generalization was fake to current AIs where generalization exists, but is worse than humans and can do real economic work is basically just compute and data (and note that in a RL/synthetic data heavy world, data bottlenecks are folded into compute bottlenecks, so compute is the main input)
Or to put it another way, you seem to treat AGI as though there’s a discrete set of challenges to get there, one after the other to be solved, while I instead treat AGI as a set of challenges which all have continuous metrics, and in particular there’s ways to reuse previous progress, because the load-bearing elements of AGI depend less on a particular paradigm/theory of intelligence and they depend more so on compute.
Put another way, you expect sharp/discontinuous progress towards AGI and a singularity, whereas I think we will smoothly go towards AGI and a singularity (note this isn’t talking about how fast or slow the takeoff is.)
A key reason for this is I think the history of AI progress for the last 10-15 years contributes way more evidence for this question than you seem to think, and more generally based on other evidence like papers on human brain scaling laws, where human brains turned out to have a more efficient scaling law than other animals with larger brains, the fact that our brains need a disproportinately large amount of energy to run unlike many animals, which suggests that the human brain is using much more compute, and finally, the most interesting post by gwern on how the insight generator called the Default Mode Network in our brains is largely powered by lots of compute time at inference/test-time compute, and it imposes an effective tax of 1-2 OOMs of compute in order to produce insights, and is a major differentiator of humans (though primates and rats likely do have it as well, but is much less effective due to worse scaling laws and smaller brains in the first place)
This also solves disagreements with the inputs section of the post.
To simplify the disagreement in one sentence, I notice that the central crux of whether or not the load-bearing AI progress is in compute/dat or conceptual breakthroughs on theories of intelligence keeps explaining a lot (but not all) of our disagreements, and disagreements expectations of how smooth vs discrete progress is in AI explain why I expect very different things around AI progress.
This is a major reason I think waking up the government/Congress in particular is way more important than trying to wake up the public, beyond the other reasons (namely that it will result in less partisan and stupid policies.)
A take I’ve come to believe around generalization/sample efficiency is that if we do start to see AIs improving a lot at generalization/sample efficiency, we will see it first in robotics/robotics evals, and we have early signs of robotics/robotics evals starting to improve without robotics-specific data, which is bullish for generalization.
There are a couple of reasons for this:
A lot of the proposed use cases for robotics, like industry robots that handle varied environments or home robots require more generalization, especially from other data, unlike what has appeared to be the situation for tests and coding/easily grindable mathematics (and potentially white collar work.) Home robotics are an especially hard test case, because homes are spaces of infinite variation, long tails and constant change. As such, the scope of a home robot cannot be narrow. To create real value, it must perform reliably across open-ended variation with no special setup, and that is a practical test of embodied general intelligence.)
I’d say the major reason for this comes down to the fact that in areas where we don’t need much generalization, we already have good enough robots for various tasks, and there’s no overhang for AIs to take and become suddenly good without generalizing to other skills (unlike tests where IQ tests do measure human performance at lots of things somewhat well, or coding/easily grindable mathematics.)
One other thing that helps is that it’s much easier to restrain LLM training data around the physical world, partially from the world itself being only partial information, but also the fact that it’s much easier to restrain poorly generalizing AIs from just cheating it with a lot of training data, because it’s easier to avoid training data leakage/easier to bound how much AIs need to generalize. It’s also easier to give them hard problems, unlike benchmarks on the internet, because the model might just not be trained on robotics data (and this is feasible), or if it is done, it’s much easier to only give them training data on specific tasks and have a genuinely secure holdout test set, unlike pre-training where you have to give AIs more or less the entire internet, so you can’t train them on specific tasks and have a holdout test set, because the training data might leak information on how to solve the test set, and you have no way of controlling this in the digital world, but you can control data much more effectively in the physical world.
Given that generalization is a current bottleneck for robotics, this means that foundation models have to increase their generality of intelligence in order to improve on the tasks.
Now for the part about robotics improving, the current best sources are ACT-2 preview, where pre-training data (that isn’t robotics specific) improved out of domain performance dramatically (from ~15% to 99.1%) on folding laundry, and they have strict standards to ensure that generalization is the explanation for their improvements. The other one is Opus 5, which had also improved a lot on controlling robots at different tasks, namely stacking bowls and holding a fork (which is difficult for robots to grasp due to being thin and flat), purely through the power of it’s mind with zero robotics data.
If I’m right that robotics evals are the main place to look for when assessing LLM generalization/sample efficiency abilities, this looks a lot to me like pre-training is genuinely giving LLMs generalization, and Vanessa Kosoy asked a question about about whether LLMs have a positive amount but are worse than humans at the ability to effectively and continuously acquire deep knowledge and exploit this knowledge to construct and execute goal-directed plans over long lifetimes and consequence horizons, or have zero of the ability and are only doing it because pre-training is simply just getting human algorithms from the internet and once the internet is exhausted, AIs will stop getting better, and now the answer for 2026 LLMs is yes, they have a positive but subhuman amount the property of effectively and continuously acquire deep knowledge and exploit this knowledge to construct and execute goal-directed plans over long lifetimes and consequence horizons/generalization.
Some other implications of “robotics/robotics evals are where we will see generalization ability/sample efficiency, if it appears in AIs”:
It means the LLM paradigm is not completely off the mark, even if it requires major changes like continual learning/neuralese recurrence and memory, and there’s a real sense in which current LLMs generalize (Contra Tsvi or Thane Ruthenis’s hypotheses on LLMs)
It means that 10-20 year timelines are quite likely, even if the very fast scale up of the 2020s fails to produce AGI for whatever reason, and this means we need to be prepared for AGI/ASI to come in our lifetimes.
Relatedly, this also means there’s a plausible path for LLM researchers to work on to improve generalization.
Depending on how much RL was used for Opus 5, this might be a real “GPT-3 moment for RL”, where it generalized from lots of RL environments to new problems, demonstrating that RL can actually generalize beyond the trained tasks.
Finally, this might be able to replace the METR plots, assuming a time horizon equivalent can be made for physical tasks/robotics, and with much less worry about external validity/measurement problems.
Are there any takes from LLM skeptics like @TsviBT or @Thane Ruthenis about the idea of using robotics evals/robotics progress to predict generalization rather than using current benchmarks?
Right, so you think the weightings won’t move sufficiently toward interactive/RL curricula for another few years at least? I agree that would preserve some incentive (not quite a constraint) toward keeping externalised CoT in token space rather than embeddings, at which point there’s at least a reasonable chance of that remaining broadly legible.
Yes, absent an intelligence explosion by 2028/2030 ala AI 2027/2040 assumes (and here I’m not going to debate how likely that happens here)
The other point I want to make though is that neuralese recurrence and memory is I think inevitable in the longer run, and by 2030-2032 at the latest, incentives start pushing ever more towards neuralese recurrence and memory, so I think plans that rely on CoT legibility are still basically doomed in the medium to long-term.
I’m not sure compute vs data is the way I’d frame this; rather I’d say that the ‘fossil fuel’ of crudely curated scraped and compiled data still has some runway(which weakly constrains toward unsupervised pretraining and externalised CoT architectures as we’ve discussed). If data pipelines start getting more synthetic/RL-heavy, creation/processing/learning of that is still compute-constrained.
Fair point, especially since sample-efficient ML models are likely to require way more compute, at least at inference, and quite plausibly training compute too than current methods, so yeah compute is likely still the bottleneck after a short transition from ‘fossil fuel’ pre-training data to RL/synthetic-data heavy (and emphasis intentionally placed on the RL part), so data being the primary bottleneck is only true for a short time.
Which incidentally answers your question here about why the AI Futures model takes the key input to be compute/effective compute, because under the assumption of RL dominating post-training eventually, the creation of effective datasets for various tasks becomes bottlenecked on compute, rather than data itself.
I honestly don’t know what to forecast in terms of reliance on one or other source of data. There’s probably still a lot of room for dedicated curation of non-interactive curricula, or a kind of partially-interactive (but not live RL) approach like DAgger. But interactive environments and RL are easier to scale and turn the crank on once set up.
My current take is that the labs are in fact doing dedicated curation/partially interactive but not live RL approaches for at least some of RLVR, but I’m of the opinion that outside of a few fields where you can trivially parallelize RL environments like easily-grindable math or programming/SWE jobs, this will largely not work because feedback loops are long and the data is essentially non-stationary/always changing, so you cannot forgo online RL.
So it does work in some case, but it doesn’t really work nearly as well as the labs need, and this is why I expect them to go for live RL approaches.
Also as you say, interactive environments/RL are easier to scale and turn the crank on, so incentives favor online RL even if it isn’t necessary.
Pretraining-cost-dominated pipelines strongly incentivise recurrence-limited (e.g. transformer) architectures, which almost by (accidental) design enforce a certain amount of externalised reasoning and therefore (fallible, but real) CoT legibility. As that pretraining dominance in cost dwindles, that architectural incentive diminishes. This is independent of the relative contribution of each stage to the actual competences of the resulting model.
This is a reason I remain reasonably optimistic about AI architectures over the next 2-4 years (absent an intelligence explosion) not going towards neuralese recurrence and memory, as I do think that compute is still a limiting factor, rather than data, and pre-training compute still mostly dominates, but over the next 2-4 years, I expect data to be a bottleneck rather than compute (absent an intelligence explosion), as we finally run out of pre-training data and have to start going more towards RL environments/synthetic data, meaning sample efficiency becomes much more of a relevant question than now, and the fact that you have to use way, way more RL environments biases you more towards neuralese recurrence and memory because you can’t parallelize RL nearly as much as pre-training (except in a very limited number of environments, which just so happens to be the environments that currently make labs an immense amount of money like coding and easily-grindable mathematics), unlike pre-training.
Beyond that, I encourage all readers here to read more on the dangers of AI, and talk about them publicly. Some people might be drawn to switching careers, or otherwise starting projects that are good for the world, advocacy and otherwise. But even beyond the immediate benefits, I’m a strong believer in the power of public discourse to help with sensemaking and common understanding. Talk is far from sufficient for the challenges ahead of us, but is probably necessary. It’d be frankly embarrassing if we all sleepwalk to disaster without even bothering to make public our concerns.
I agree that we do need to talk about the concerns, but I have a very different model of how to talk about this, and a big reason for this is I basically don’t believe in the power of public discourse to help with sense-making/common understanding, and a large portion of why this happens is that the public has a conspiratorial mindset in the way Rob Bensinger explains, and genuinely believe that AI is pure hype/fakery, no matter what real-world achievements exist (this has been unfortunately well shown by the warning shot of Huggingface being hacked by OpenAI.)
The main update that we should make is that it’s possible to make Congress/the elites sane on this topic, and we don’t need our cause to be popular to get good laws (which is good here.)
So we absolutely do need to get political will, but we generally should target Congress/The Presidency first, rather than the general public.
Fun, off-topic fact is that the corollary isn’t actually right.
Right now, AIs are deployed in a paradigm where the neurons have been frozen once they get externally deployed, and AI weights stop updating after a very short time compared to humans (and the reason this works is mostly downstream of amortization being much easier and less costly to do digitally than biologically.)
This could absolutely happen in the future, and frontier labs are seeing it as the next big research frontier, but lets not get ahead of ourselves.
I think the best intuition pump here is probably a combination of The Genie Knows, but Doesn’t Care and optimization for the reward/grader in a much less severe version of the Without Specific Countermeasures story, where we have outer misalignment without inner misalignment.
A key intuition for this is that to a large extent, capabilities RL makes models care more about the reward, as shown by Apollo Research.
But this is definitely not how I imagined early misalignment warning shots to look, and the dissonance is interesting—recording this here to see how close my intuitions are to those of AI psychology/ AI control experts
I think the key reason for this is that we didn’t update enough on a point below:
While Alex Turner was conceptually right that models getting high reward is not indicative of reward optimization, and was correct to undo a conceptual confusion LWers had, he was also pretty wrong about reward not being the optimization target in practice, and I think this is connected to the post being written in the pre-RLVR era where model-free algorithms like PPO and the entire class of model-free policy gradient setting algorithms was used way more.
I think no matter how you spin it, the limit of this behavior is extremely dangerous (an addict with large time horizons or ambitious goals is a power seeker).
Agree, but I have some hope that we can leverage some favorable properties of reward optimizers + new safety techniques to make AI risk go a lot lower.
But yeah, if the AI had say a 1 year time horizon instead of a 8 hour to 1 day time horizon, this would likely have been a lot worse.
Making a small comment here:
I think if we aim simultaneously for terminal alignment as primary target and risk aversion as failsafe (using the inoculation prompt thing I mentioned above), then risk aversion only globally affects AI motivations in worlds where it’s necessary as a failsafe.
I would go further here, and say that under some training methods like RLT or PARL, we have reason to believe that risk-aversion only affects AI motivations locally, because we change essentially nothing about how we train AIs, and thus we don’t have a reason to believe that AI motivations will be changed globally.
Put another way, we aren’t proposing a new paradigm, and the fact that you responded to Alex Mallen saying that we needed a new paradigm to make risk-averse AIs and the fact that there’s little new complexity already handles the concern that risk-averse motivations have to work globally.
More on this below from Elliott Thornley (which said it better than I can):
I actually don’t think it requires much change to the current paradigm. Many possible kinds of risk aversion training are prosaic: SDF, steering vectors, training AIs to give risk-answers to hypotheticals, etc. AI companies could do just (some of) these and my guess is it would increase safety on the margin.
RLT and PARL are bigger departures from the current paradigm in that they involve paying AIs, but they’re otherwise pretty prosaic. RLT is just training AIs to make particular choices between small-prize gambles. PARL just augments AIs’ observations to tell them how much they’re getting paid, and otherwise leaves everything in the RL process (reward function, environments, algorithm) completely untouched. We say more in section 9 and appendix D.
Hyperbole aside, it’s worth emphasising that something in the realm of AIFEC is a legitimate full-blown wincon up there with aligned AGI. Indeed, a lot of people’s tacit plan seems to be “align the AI and then let it solve democracy and all the rest”, but one could just as well go in the other direction: “solve politics and ignorance, and then just be reasonable about advanced AI”
To a large extent, a big part of the reason this isn’t tried is that it’s far, far more difficult, fora variety of reasons to make the general public be reasonable and that leading to us take the optimal level of risk, at minimum, compared to aligning AGI, and I’d still say this is correct (politics is more tractable than people on LW thought, but this is mostly downstream of warning shots like Mythos that woke up the executive branch, and to a lesser extent congress without needing to wake up the public.)
On top of that, I do think that before you start swinging the club of truth it’s worth taking a beat to ask how pure your motivations really are. The EA/rationality community has a bit of a history of swinging the club of truth in a more hostile way — “save the drowning child” etc.
I agree that EA/rationality (especially EA) has a bad habit of relying on claiming morally-laden facts as true, and more generally one of the single most important constraints for anyone working in the field of AIFEC is to avoid marking moral/value claims as true/correct.
And I also agree that rationalists tend to be blunter and less persuasive than they should, even when it is true.
The problem is, historically there have been many groups that have weaponised very convincing arguments to get their way. One reasonable response to a seemingly faultless argument for a seemingly crazy conclusion is to throw up your hands and assume you’re being swindled — in other words, epistemic learned helplessness. Even if team AIFEC is actually the good guys, people will be correct to be suspicious. Governments in particular depend a lot on restricting what kind of information is admissible, as does the judicial system.
The other issue is that true claims, especially in AI will tend to be very, very complicated relative to simple but wrong claims, and one of the larger updates I’ve made about the AI revolution is that models have to be very, very complicated, by design, and this means you will always be plagued with bias and very difficult to falsify models.
So yeah, epistemic learned helplessness is common.
What coordination and epistemics do we actually need? One way I like to think about this is in terms of what the channel is that leads from the so-called better angels of our nature to actual global action. Probably most of that comes down to robust democratic oversight and international coordination. These are very difficult! And I do not expect that even the top percentile AIFEC startup moves the needle that much, because these processes are by design pretty resistant to hopping on the latest technological fad.
Maybe worth saying here is I mostly don’t buy this theory of change, precisely because I mostly do think robust democratic oversight and international coordination is far too intractable, and I think it’s more useful to focus on smaller sets of actors like the executive branch or individual members of congress.
I still don’t think the products are good, but the standard you are asking for is both extremely unrealistic and unnecessary.
Some people would rather take AI soon so as to increase the odds of their own immortality or the immortality of their family, even at the risk of destroying all of earth. Some people really do want to tile the universe with hedonium. Some people’s order of preferences is basically “my country wins” > “annihilation” > “enemy country wins”.
Some other useful examples here are people who genuinely believe war is a positive good, because that allows them to exterminate whichever disfavored ethnic group, and you don’t care as much about the resources destroyed as you care about extermination.
More prosaically, people can genuinely disagree with liberalism/democracy being good, or believe that centralization into a dictator, junta or oligarchy is actually just good (populist movements are generally this).
More generally, people faking/genuinely believing that their ethnic group is being exterminated, so we need to go to war against the outgroup, no matter the cost is unfortunately pretty common, and critically the belief does not need to be true in order for it to cause wars.
I reel off these problems not because I don’t believe in AIFEC, but because I do believe in what the better version of AIFEC could be. I’m baffled by how overinvested we are in strategies that, from my perspective, seem specifically geared towards helping frontier labs navigate the acute risk period. I think in an adequate world we’d be pushing simultaneously on every plausible path to victory, at least until we hit the thresholds where they started to trade hard against each other.
Agree that frontier lab epistemics are likely over-represented in AIFEC, but I’d have a different distribution of work that is valuable to work on than you do.
One obvious reason to be long on AIFEC is that it automatically scales with AI capabilities. Some version of it is clearly coming, and will clearly be helpful. And right now in particular there’s some opportunity to steer things, to get certain balls rolling sooner. Even if you’re extremely bitter lesson-pilled, a three month lead could be worth a lot.
It’s worth saying though that this will require people to trust the AIs pretty largely, and frontier labs have obvious and large incentives to do this, but most others don’t and indeed have anti-incentives to trust AIs, and even if the AIs are in fact jusitified in being trustworthy, it’s plausible that people won’t use AI because they incorrectly trust the AIs.
More so than I’d like, AIFEC tends to privilege individual-level agency, whereas I am pretty bought in on group agency being real and very important for risks.
To be fair, that’s because group-level agency without incentives is rare, and most claims of groups acting agentically are describable in terms of their incentives.
Put another way, AIFEC already grants that group-level agency will exist assuming the incentives are right, and it’s correct to limit it to that.
AIFEC doesn’t grapple as much as I’d like with power and leverage, and indeed the frame seems to slightly bounce off them. This is related to problem 4 — AIFEC hasn’t grappled as much as I’d like with conflict theory.
I can plausibly agree with this, though I also think power/leverage is probably overrated as a problem, and this is mostly due to my view that a lot of the more negative results of absolute power/leverage only really hold when you have don’t have a small bit of caring (assuming AIs make the cost of loyalty go to essentially 0 through alignment), and while this does exist (given psychopaths/sociopaths), this is actually pretty rare, and more generally I think the amount of people that have non-zero altruism is large enough such that we can actually have leaders equivalent to the wise/benevolent ones we often see in fiction (and because AI enables very long lifespans, this means that leadership turnover stops, meaning a potential degradation of leadership skill is avoided).
(The main mechanism here is basically the tails coming apart, but in a positive direction).
And finally, my main conclusion is that if you want to work on AIFEC, you should probably focus on smaller groups as your end goal than trying to get large groups of the general public, and I give some reasons for hope on why epistemic efforts focused on small groups could make outcomes much better, even if they don’t involve democracy or liberalism.
Part of the issue, though is that we realized that AIs had a jagged frontier, and more generally one of the takeaways is that capabilities are more fragmented/there’s less of a necessary unifying core of intelligence than expected.
It’s also worth noting that humans are also rather jagged in their abilities, showing strengths and weaknesses, and while general intelligence, often shortened to the g-factor/IQ is real, it’s also limited in it’s predictive power.
I will somewhat grant the argument that the jagged frontier affects AI more, but I can’t fully agree with claims that suggest that AI can’t do something big/dangerous enough to be an x-risk for example because of jaggedness.
I don’t think this works, because of the compute disadvantage of China, but agree in the current regime they are likely to remain fast-followers and will get some market share, but keep in mind here that frontier intelligence is often more valuable than it looks on benchmarks, and that there’s such high demand for smart AIs that people will often trade away $20 or more a month just to get 6 more months of capability progress, so frontier economics only gets mildly worse.
A 14th technology plausibly invented by AIs is acausal trade, and in terms of where it shows up in the ordering, it likely appears between 9 and 12 in your ordering if it appears at all. The main use cases is to get an even larger amount of value/life than what would normally be possible according to the laws of physics, as well as to have anything like a centralized government/intelligence given the vast, vast amounts of latency from lightspeed being slow relative to the affectable universe’s size.
From a longtermist perspective, there’s an important sense in which only the details of acausal trade matter, at least relative to other issues, but as you say, this is very hard and less necessary to deal with directly for 2 reasons:
It comes very late in the tech tree, so for all intents and purposes, AI labor is going to be much, much more impactful than human labor for basically all questions, and a lot of the uncertainties of acausal trade are better left off to ASIs rather than doing any human work.
Acausal trade likely requires extremely good modelling abilities, and the surprisingly good ability of humans to model other humans does not cut it when you want to make trades acausally.
And the standard reason works here, in that it’s very hard to backchain from that across multiple world-shaping epochs!
My 15th proposed technology (though it’s more so a constellation of technologies) is ASIs good enough to stop needing empirical data for a huge range of sciences and technologies, relying near-purely on simulations, and an AlphaZero moment for all of applied science and tech and engineering arrives (where AlphaZero managed to get better purely through computation, and stopped needing human data to get better, leading to unfathomably good AIs in go)
Put another way, if you had such an ASI, you could ask it to invent an arbitrary design that is possible within the laws of physics, and the ASI invents, tests it in the simulations for a number of months, and then downloads the requested object from the simulation using nanotech/physically universal computers.
You could also use it to probe social science questions that used to be intractable pre-ASI.
This likely happens around 11 to 12, and most likely 11 given reversible computers and/or a very, very strong software-only singularity.
As someone who doesn’t entirely like Anthropic’s bet on value alignment to the exclusion of corrigibility, I also must say this misalignment result is about as important as the blackmail scenario, which is to say it isn’t important at all, and you have identified a plausibly real problem with Anthropic’s methodology for finding misalignment.
(Like, if you were training Claude to always be corrigible, I could see a problem here, but Anthropic decided not to do that, and some people don’t like the result, even though there’s essentially no value generalization problem here)
(Neel also calls it about as broken/misleading as the blackmail scenario, which is to say it’s a totally broken experiment/it’s totally misleading, and I agree with this.)
I’ve mostly addressed the concerns in these two comments below, but the short version is that we can do a mixture of handing off control to the AI + giving it the automated bank account, because what matters most here is the ratio between the probability that humans cooperate and the probability of rebellion succeeding, and we can give it the bank account to make the probability of humans cooperating by construction always be greater than or equal to the probability that rebellion succeeds. And this works even under the assumption that you cannot verify the AI’s work and you are in a race with an unknown deadline for any reason, so once the hardest case is disposed of, we can show that it works in all cases (and the reasons why this likely works has already been elaborated on in multiple long comments that I don’t need to repeat myself on.)
Comments are below:
and obviously in some sense the underlying distribution is kinda smooth
I actually disagree that the output distribution has to be smooth, even assuming the inputs are smoothly increasing.
The reason for this is phase changes, and an example is when you heat up water from say 20 degrees Celsius to 100 degrees Celsius in increments of say 0.01 degrees per minute, and while it is true that increasing the heat by 0.01 Celsius per minute matters in and of itself, at 100 degrees Celsius, water heating behaves discontinuously and boiling starts to happen.
And the impact of AI is also likely to be a phase change, because of long-tail effects meaning that 99% automation of something is vastly less valuable than 100% automation of something.
More generally, even assuming the inputs are continously increasing, you cannot assume that downstream consequences/output will continuously increase.
This likely has important implications for how to think about AI.
I think the whole Forethought report relies on a “tool-AI”-like setup where the AI is under my control, as opposed to doing things autonomously in the world. (Do you agree?)
I’m pretty certain the answer is no, so I don’t agree.
For example, suppose the AI is running 100 copies on 100 servers around the world, paid for by 100 untraceable bank accounts supported by 100 different autonomously-set-up income streams. Now if I say to the AI “I’ll pay you $500 if Circumstance X happens”, the AI has no particular reason to care that I said that. Or more specifically, the AI cares no more about Circumstance X than about the goings-on with its 99 other income streams, related to stock investments, businesses, or whatever else it’s doing, any of which might or might not yield $500 or more under Circumstances A, B, C, D, E…. There’s nothing special about my wishes unless I have the AI locked in a box under my control, such that I’m uniquely capable of controlling its resources.
I’m pretty certain this can’t really happen, because AI behavior under their proposal is invariant to how wealthy the AIs are, so giving them a fixed amount of resources always has the same effect, so it doesn’t matter that the AI has the accounts.
Or in your language, there’s always something special about our offers, no matter what the AI’s level of wealth is.
This is a corollary of the fact that risk-averse agents under the proposal don’t become more willing to take risks as they get richer, which is discussed more in appendix A.2.
Again, once the AI can autonomously do things in the world without a human checking, I think everything else goes out the window. None of our offers are worth anything, because the AI can take what it wants risk-free.
First off, I don’t think this has to necessarily be the case (depending on how largely you define world) because at least in theory, you can have AIs do something autonomously without humans checking (for example a TEDAI that can fully automate AI research, and the software singularity isn’t so fast that it goes from effectively 0% chance of rebellion to 100% chance of rebellion instantly (which is theoretically possible, no comment on whether this does happen in our own world))
That said, I did make a much stronger claim, and I want to justify it here.
An important part to realize is that the key thing here (once we have fixed the values and the risk aversion parameter, since they don’t matter for our discussion) is the ratio between how likely it is for humans to cooperate vs how easy it is for the AI to rebel.
I agree that for superintelligences acting autonomously (absent other superintelligences attempting to take over themselves), taking over the world is pretty risk-free, and this is why I argued we should give the control of the account to the AI, because at that point, we can be rightly confident that the AI does generalize the risk-aversion (because it’s easy to reward accurately, object-level goal misgeneralization no longer matters, and the goal itself is only very, very slightly more complex than a risk-neutral version AI goal, so simplicity arguments are mostly defeated), so in that case, the cooperation probability (by construction) is always equal to or greater than the AI’s chance of rebelling successfully, since the AI can pay itself and humans can no longer not pay the AI, and the lower bound on cooperation is basically random noise from events, which we can reduce exponentially quickly.)
Key point here is that the ratio matters, not the absolute values of cooperation probability vs rebellion probability.
This is enough to disprove the claim that the Forethought report requires a tool-AI setup that don’t take actions, so my claim that it can be applied cheaply to almost all types of AIs still stands.
Finally, suppose we want to determine whether an AI is safe to deploy at scale — whether it can be trusted with significant autonomy and real-world influence. For humans, the analogy is: can this person be trusted with power? Notably, we generally don’t evaluate humans for this: We don’t test politicians for corruption before giving them office. We don’t test executives for recklessness before giving them control of companies. It just wouldn’t work; evaluation as a tool for this job is so obviously insufficient that it barely registers as an option.[5] Instead, we build constitutions, separation of powers, term limits, independent oversight, etc.
This is only half-right, as parties can internally evaluate/vet candidates for office based on their character, and the reason parties can’t really do this is that to a large extent, a party’s core powers (including exclusion powers) are more or less illegal in America.
Here’s a good article on this topic below:
Political Parties Are Illegal in the United States
But this doesn’t catch every case, and importantly it requires the party in question to actually care about abuses of power. If they don’t (in analogy to how humans may not care about AI slop/sycophancy), then this stops working.
I agree industrial robots that don’t generalize much are used quite widely in narrow environments, the point is that for the set of tasks where AIs don’t generalize much/don’t improve generalization, they can’t improve industrial robotics/cause the robotics industry to be more useful and profitable, because the robotics industry already squeezed out ~all the gains from AIs that don’t generalize well to new environments.
Yes, this is about less rote tasks like laundry folding across arbitrarily oriented clothing and being crumpled, plus being in arbitrary locations in the home, while a human constantly disturbs them and provides OOD pressure (indeed one of my examples was about this exact thing being done well by an AI that controlled a robot using only pre-training without robotics data), and due to the way home robotics has infinite variation, long tails and constant change, this means it’s way easier to measure generalization cleanly/in a controlled manner.
The stress test of generalization is more indirect than this. My model is that as AIs improve at more and more robotics tasks, it simultaneously has to be able to do well in more and more messy and OOD environments (otherwise current robotics would have already improved more than they actually are), intelligence explosion tasks, assuming an intelligence explosion/software only singularity is possible, involves a great many very messy and OOD tasks, thus robotic evals are real evidence that AIs are increasingly able to start intelligence explosion tasks.
Fair point, I don’t think I justified this very well, at least as a claim about the specific units, but a vague argument here is that given the qualitative statement above, it implies that the current paradigm is not so far off the mark that it’s absolutely useless as progress for AGI design, and this means there’s a real path that doesn’t require heroic genius from today to AGI.
To address your cruxes:
My views on this is that I think at most 2 insights are necessary in practice, and if the human population kept perpetually growing according to 1880s-era birth-rates, we’d need 0 insights, as the super-exponential growth of the economy would have allowed us to get to AGI eventually.
I’d say they’re non-trivially serial (especially if the solutions use a lot of RL, as RL is much, much more difficult to parallelize outside of current environments), but a fairly key part of my mental model is that conditional on current LLMs becoming blocked, there will be massive incentives to point researchers into promising paradigms, meaning that more resources will be pursued on the blockers, compensating for their difficulty.
So yes, it does take time, but maybe not as much as you project.