I’ve been active in the meatspace rationality community for years, and have recently started posting regularly on LW. Most of my posts and comments are about AI and alignment.
Posts I’m most proud of, and / or which provide a good introduction to my worldview:
Without a trajectory change, the development of AGI is likely to go badly
Steering systems, and a follow up on corrigibility.
I also wrote a longer self-introduction here.
PMs and private feedback are always welcome.
I appreciate the offer to bet! I’m probably going to decline though—I don’t really want or need more skin-in-the-game on this question (many of my personal and professional plans assume short timelines.)
You might be interested in this post (and the bet it is about), for some commentary and issues with operationalizing bets like this.
Also, you might be able to find someone else to bet with you—I think my view is actually closer to the median among EAs / rationalists / alignment researchers than yours. For example, the Open Phil panelists judging this contest say:
Panelist credences on the probability of AGI by 2043 range from ~10% to ~45%.
Sometimes this doesn’t even matter. Sports betting is very popular even though it’s usually negative sum.
Yes. PredictIt used to attract a lot of “dumb money”—people who just wanted to bet on their favorite candidate (or against disfavored candidates). They also used to run weekly markets on polling averages and things like the number of times Trump would tweet that tended to attract people who just wanted to do some skill-based gambling, whether they actually had the skill or not.
PredictIt charges high transaction fees with no outside subsidies, so all of the markets were extremely negative-sum. Despite this, gamblers and [Candidate X] True Believers managed to provide ample subsidy to attract some more knowledgable traders. (Based on the comments section of some of the popular markets, there were many people who lost thousands or tens of thousands. Probably some of the biggest losers were gambling addicts who destroyed their finances in the process. A pretty big negative externality of negative-sum markets where amateur participation is allowed.)
I think the biggest problem with these estimates is that they rely on irrelevant comparisons to the human brain.
What we care about is how much compute is needed to implement the high-level cognitive algorithms that run in the brain; not the amount of compute needed to simulate the low-level operations the brain carries out to perform that cognition. This is a much harder to quantity to estimate, but it’s also the only thing that actually matters.
See Biology-Inspired AGI Timelines: The Trick That Never Works and other extensive prior discussion on this.
I think with enough algorithmic improvement, there’s enough hardware lying around already to get to TAI, and once you factor this in, a bunch of other conditional events are actually unnecessary or much more likely. My own estimates:
Event | Forecastby 2043 or TAGI, |
| We invent algorithms for transformative AGI | 90% |
| 100% | |
| 100% | |
| 100% | |
| 100% | |
| We avoid derailment by human regulation | 80% |
| We avoid derailment by AI-caused delay | 95% |
| We avoid derailment from wars (e.g., China invades Taiwan) | 95% |
| We avoid derailment from pandemics | 95% |
| We avoid derailment from severe depressions | 95% |
| Joint odds | 58.6% |
Explanations:
Inventing algorithms: this is mostly just a wild guess / gut sense, but it is upper-bounded in difficulty by the fact that evolution managed to discover algorithms for human level cognition through billions of years of pretty dumb and low bandwidth trial-and-error.
“We invent a way for AGIs to learn faster than humans.” This seems like it is mostly implied by the first conditional already and / or already satisfied: current AI systems can already be trained in less wall-clock time than it takes a human to develop during a lifetime, and much less wall clock time than it took evolution to discover the design for human brains. Even the largest AI systems are currently trained on much less data than the amount of raw sensory data that is available to a human over a lifetime. I see no reason to expect these trends to change as AI gets more advanced.
“AGI inference costs drop below $25/hr (per human equivalent)”—Inference costs of running e.g. GPT-4 are already way, way less than the cost of humans on a per-token basis, it’s just that the tokens are way, way lower quality than the best humans can produce. Conditioned on inventing the algorithms to get higher quality output, it seems like there’s plausibly very little left to do to here.
Inventing robots—doesn’t seem necessary for TAI; we already have decent, if expensive, robotics and TAI can help develop more and make them cheaper / more scalable.
Scaling production of chips and power—probably not necessary, see the point in the first part of this comment on relevant comparisons to the brain.
Avoiding derailment—if existing hardware is sufficient to get to TAI, it’s much easier to avoid derailment for any reason. You just need a few top capabilities research labs to continue to be able to push the frontier by experimenting with existing clusters.
Surprised to see the downvotes on this.
I think there’s a strong case to be made that moral considerations are already having a noticeable effect on slowing AI progress. There are probably many bright EAs and rationalists who would be working in AI capabilities, if they didn’t believe that was likely to lead to the extinction of humanity, and that the extinction of humanity is morally wrong.
As capabilities advance further, more “normies” are probably going to make the connection between the development of AGI to the threat of extinction, and then make the relatively smaller leap to the belief that extinction is morally wrong. And normies seem more likely to react to this realization with backlash, loud outrage, and mood affiliation with narrow AI, than EAs and rationalists tend to.
Personally, I strongly predict that any outrage or backlash will not actually result in sufficiently drastic, correctly targeted, and globally coordinated interventions needed to slow AI progress by enough to make a real difference. (In part, because, as Geoffrey points out, the triggers for this backlash will likely be problems caused by misuse of narrow AI, which is not where most of the danger actually lies.) But it seems worth tracking this as an avenue where things could change, perhaps rapidly.
Meta: I really like ideas and concrete steps for how to practice the skill of thinking about something. I think there are at least three methods for learning how to think productively about a particular problem:
Reading material written by others (not necessarily passively; this might include checking your understanding of the material you read)
Doing exercises, practice problems, toy projects, etc. that are focused directly on the object-level problem.
Doing exercises to practice the “cognitive motions” needed to think productively about a problem. I think the exercise(s) described in this post are a good example of this.
And I think the last option is often neglected (in all fields, not just AGI safety) because there’s not a lot of written material on how to actually do it. Note that it is a different thing than the more general skill of learning to learn and meta-cognition—different kinds of technical problems can require learning different, domain-specific kinds of cognitive motions.
If you’ve absorbed enough of the Sequences and other rationality material through osmosis, you might be able to figure out the kind of cognitive motions you need, and how to practice and develop them on your own (or maybe you’ve done some of the exercises in the CFAR handbook and learned to generalize the lessons they try to teach).
But having someone more experienced write down the kind of cognitive motions you need, along with exercises for how to learn and practice them, can probably get more people up to speed much more quickly. I think posts like this are a great step in that direction.
Object-level tip for the breaker phase: thinking about how a literal human might break your alignment proposal can be a useful way for building intuitions and security mindset. A lot of real alignment schemes involve doing something with human-ish level intelligence, and thinking about how an actual human would break or escape from something is often more natural and less prone to veering into vague or magical thinking than positing capabilities that a hypothetical super-intelligent AI system might have.
If you can’t figure out how an actual human can break things, you can relax the constraint a bit by thinking about what a human with the ability to make 10 copies of themselves, think 10x as fast, write code with superhuman accuracy and speed, etc. could do instead.
Threat modelling is the term for this kind of thinking in the field of computer security.
Navigate the initial alignment problem:2 getting to the first point of having very powerful (human-level-ish), yet safe, AI systems. For human-level-ish AIs, I think it’s plausible that the alignment problem is easy, trivial or nonexistent. It’s also plausible that it’s fiendishly hard.
Can you clarify what you mean by human-level-ish and safe? These terms seem almost contradictory to me—human-ish cognition is extremely unsafe in literal humans, and not just because it could be misused or directed towards dangerous or destructive ends by other humans.
”Transformative and safe” (the phrase you use in Nearcast-based “deployment problem” analysis) seems less contradictory. I can imagine AI systems and other technologies that are transformative (e.g. biotech, nanotech, AI that is far below human-level at general reasoning but superhuman in specific domains), and still safe or mostly safe when not deliberately misused by bad actors.
Dangerousness of human-level cognition has nothing to do with how hard or easy alignment of artificial systems is: a literal human, trapped in an AI lab, can probably escape or convince the lab to give them “parole” (meaning anything more than fully-controlled and fully-monitored-in-realtime access to the internet). Literal mind-reading might be sufficient to contain most or all humans, but I don’t think interpretability tools currently provide anything close to “mind-reading” for AI systems, and other security precautions that AI labs currently take also seem insufficient for containing literal humans under even mildly adversarial conditions.
Maybe alignment turns out to be easy, and / or it turns out to be trivial to make the AI not want to escape or get parole. But alignment on that level is definitely not a solved problem in humans, so the aim for human-level AI has always seemed kind of strange to me.
A possible explanation for this phenomenon that feels somewhat natural with hindsight: there’s a relatively large minimum amount of compute required to get certain kinds of capabilities working at all. But once you’re above that minimum, you have lots of options: you can continue to scale things up, if you know how to scale them and you have the compute resources to do so, or you can look for algorithmic improvements which enable you to do more and get better results with less compute. Once you’re at this point, perhaps the main determiner of the relative rate of progress between algorithms and compute is which option researchers at the capabilities frontier choose to work on.
I inline-reacted to the first sentence of this comment. The comment takes up too much vertical space for the green highlighting to be visible when I hover over the react icon at the bottom though, so I have no way of seeing exactly what I reacted to while it is highlighted. Maybe hovering over underlined text should show the reaction?
I like this post as a vivid depiction of the possible convergence of strategicness. For literal wildfires, it doesn’t really matter where or how the fire starts—left to burn, the end result is that the whole forest burns down. Once the fire is put out, firefighters might be able to determine whether the fire started from a lightning strike in the east, or a matchstick in the west. But the differences in the end result are probably unnoticeable to casual observers, and unimportant to anyone that used to live in the forest.
I think, pretty often, people accept the basic premise that many kinds of capabilities (e.g. strategicness) are instrumentally convergent, without thinking about what the process of convergence actually looks like in graphic detail. Metaphors may or may not be convincing and correct as arguments, but they certainly help to make a point vivid and concrete.
I agree the analogy breaks down in the case of very adversarial agents and / or big gaps in intelligence or power. My point is just that these problems probably aren’t unsolvable in principle, for humans or AI systems who have something to gain from cooperating or trading, and who are at roughly equal but not necessarily identical levels of intelligence and power. See my response to the sibling comment here for more.
yes but part of what makes interpretability hard is it may be incomputable.
Aside: I think “incomputable” is a vague term, and results from computational complexity theory often don’t have as much relevance to these kind of problems as people intuitively expect. See the second part of this comment for more.
(What do you mean by “waveform” reader”?)
An oscilloscope. Note that it isn’t particularly realistic to hook up a scope to the kind of hardware that current AI systems are typically trained and run on.
But what I was trying to gesture at with this comment, is this is the kind of problem that the AI might be able to help you with. If the AI isn’t willing or able to fold itself up into something that can be run entirely on a single, human-inspectable CPU in an airgapped box, running code that is amenable to easily proving things about its behavior, you can just not cooperate with it, or not do whatever else you were planning to do by proving something about it, and just shut it off instead.
(If the AI is already adversarial to the point where it won’t let you shut it off, and is running on a distributed system, you’ve already lost. Willingness to fold itself up and be shut off means that the AI is already pretty aligned; it wouldn’t surprise me if this problem is alignment-complete.)
As for your practical difficulties, I agree these are all problems. I am not saying the problem you pose isn’t hard, just that there doesn’t seem to be anything that makes if fundamentally impossible to solve in principle.
There is lots of academic research on hardware security and secure hardware, verifiable computing (e.g. using zk-SNARKs), formally verified programming, chain-of-trust, etc. that attempt to make progress on small pieces of this problem (not necessarily with a specific focus on AI). Stitching all of these things together into an actually end-to-end secure system for interacting with a smarter-than-human AI system is probably possible, but will require solving many unsolved problems, and designing and building AI systems in different ways than we’re currently doing. IMO, it’s probably better to just build an AI that provably shares human values from the start.
Could this exist in the case of a super intelligent agent?
Probably, if the agents make themselves amenable to interpretability and inspection? If I, as a human, am going through a security checkpoint (say, at an airport), there are two ways I can get through successfully:
be clever enough to hide any contraband or otherwise circumvent the security measures in place.
just don’t bring any contraband and go through the checkpoint honestly.
As I get more clever and more adversarial, the security checkpoint might need to be more onerous and more carefully designed, to ensure that the first option remains infeasible to me. But the designers of the checkpoint don’t necessarily have to be smarter than me, they just have to make the first option difficult enough, relative to any benefit I would gain from it, such that I am more likely to choose the second option.
The point is that in practice it’s not necessary to model any humans,
Right, but my point is that it’s still necessary for something to model something. The bot arena setup in the paper has been carefully arranged so that the modelling is in the bots, the legibility is in the setup, and the decision theory comprehension is in the author’s brains.
I claim that all three of these components are necessary for robust cooperation, along with some clever system design work to make each component separable and realizable (e.g. it would be much harder to have the modelling happen in the researcher brains and the decision theory comprehension happen in the bots).
Two humans, locked in a room together, facing a true PD, without access to computers or an arena or an adjudicator, cannot necessarily robustly cooperate with each other for decision theoretic reasons, even if they both understand decision theory.
PrudentBot is modelling its counterparty, and the setup in which it runs is what makes the modelling and legibility possible. To make PrudentBot work, the comprehension of decision theory, counterparty modelling, and legibility are all required. It’s just that these elements are spread out, in various ways, between (a) the minds of the researchers who created the bots (b) the source code of the bots themselves (c) the setup / testbed that makes it possible for the bots to faithfully exchange source code with each other.
Also, arenas where you can submit a simple program are kind of toy examples—if you’re facing a real, high-stakes prisoner’s dilemma and you can set things up such that you can just have some programs make the decisions for you, you’re probably already capable of coordinating and cooperating with your counterparty sufficiently well that you could just avoid the prisoner’s dilemma entirely, if it were happening in real life and not a simulated game.
If someone succeeds in getting, say a ~13B parameter model to be equal in performance (at high-level tasks) to a previous-gen model 10x that size, using a 10x smaller FLOPs budget during training, isn’t that a pretty big win for Eliezer? That seems to be kind of what is happening: this list mostly has larger models at the top, but not uniformly so.
I’d say, it was more like, there was a large minimum amount of compute needed to make things work at all, but most of the innovation in LLMs comes from algorithmic improvements needed to make them work at all.
Hobbyists and startups can train their own models from scratch without massive capital investment, though not the absolute largest ones, and not completely for free. This capability does require massive capital expenditures by hardware manufacturers to improve the underlying compute technology sufficiently, but massive capital investments in silicon manufacturing technology are nothing new, even if they have been accelerated and directed a bit by AI in the last 15 years.
And I don’t think it would have been surprising to Eliezer (or anyone else in 2008) that if you dump more compute at some problems, you get gradually increasing performance. For example, in 2008, you could have made massive capital investments to build the largest supercomputer in the world, and gotten the best chess engine by enabling the SoTA algorithms to search 1 or 2 levels deeper in the Chess game tree. Or you could have used that money to pay for researchers to continue looking for algorithmic improvements and optimizations.
Also, I didn’t mean for this distinction to be particularly interesting—I am still slightly concerned that it is so pedantic / boring / obvious that I’m the only one who finds it worth distinguishing at all.
I’m literally just saying, a description of a function / mind / algorithm is a different kind of thing than the (possibly repeated) execution of that function / mind / algorithm on some substrate. If that sounds like a really deep or interesting point, I’m probably still being misunderstood.
What source code and what machine code is actually being executed on some particular substrate is an empirical fact about the world, so in general, an AI (or a human) might learn it the way we learn any other fact—by making inferences from observations of the world.
For example, maybe you hook up a debugger or a waveform reader to the AI’s CPU to get a memory dump, reverse engineer the running code from the memory dump, and then prove some properties you care about follow inevitably from running the code you reverse engineered.
In general though, this is a pretty hard, unsolved problem—you probably run into a bunch of issues related to embedded agency pretty quickly.
To get an intuition for why these problems might be possible to solve, consider the tools that humans who want to cooperate or verify each other’s decision processes might use in more restricted circumstances.
For example, maybe two governments which distrust each other conduct a hostage exchange by meeting in a neutral country, and bring lots of guns and other hostages they intend not to exchange that day. Each government publicly declares that if the hostage exchange doesn’t go well, they’ll shoot a bunch of the extra hostages.
Or, suppose two humans are in a True Prisoner’s Dilemma with each other, in a world where fMRI technology has advanced to the point where we can read off semantically meaningful inner thoughts from a scan in real-time. Both players agree to undergo such a scan and make the scan results available to their opponent, while making their decision. This might not allow them to prove that their opponent will cooperate iff they cooperate, but it will still probably make it a lot easier for them to robustly cooperate, for valid decision theoretic reasons, rather than for reasons of honor or a spirit of friendliness.
but it’s not clear that an agent needs to understand the formal theory in order to actually avoid D,D.
They definitely don’t, in many cases—humans in PDs cooperate all the time, without actually understanding decision theory.
The hierarchy is meant to express that robustly avoiding (D,D) for decision theory-based reasons, requires either that the agent itself, or its programmers, understand and implement the theory.
Each level is intended to be a pre-requisite for the preceding levels, modulo the point that, in the case of programmed bots in a toy environment, the comprehension can be either in the bot itself, or in the programmers that built the bot.
I don’t see how level 2 depends on anything in level 3 - being at level 2 just means you understand the concept of a Nash equilibrium and why it is an attractor state. You have a desire to avoid it (in fact, you have that desire even at level 1), but you don’t know how to do so, robustly and formally.
I feel like this is common enough—”are they helping me out here just because they’re really nice, or because they want to get in my good graces or have me owe them a favor?”—that authors often have fictional characters wonder if it’s one or the other. And real people certainly express similar concerns about, say, whether someone donates to charity for signaling purposes or for “altruism”.
That’s a good example, though I was originally thinking of an agent which behaves actually kindly, not because it expects any favor or reciprocation, nor because it is trying to manipulate the agent it is being kind to (or any other agent(s)) as part of some larger goal.
An agent might be capable of behaving in such a manner, as well as understanding the true and precise meaning of kindness, as humans understand it, but without having any of the innate drives or motivations which cause humans to behave kindly.
Such an agent might actually behave kindly despite lacking such drives though, for various reasons: perhaps an inclination to the kindness behavior pattern has somehow been hardcoded into the agent’s mind, or, if we’re in the world of HPMOR, the agent has taken some kind of Unbreakable Vow to behave kindly.
Many people around the world believe that aliens live among us or have visited Earth at some point. By base rates, it’s not surprising that some of these people are high-ranking officials in the DoD or other governments.
Some of these officials have access to classified / non-public info, and are probably sincerely convinced that they have found strong evidence for aliens. But I haven’t seen anything to suggest that the publicly available information is actually strong evidence, nor that the believers with access to private information are particularly strong rationalists with well-calibrated priors and strong Bayesian updating skills. So the fact that they allegedly believe something isn’t much evidence that such a thing is true.