Ajeya Cotra

Karma: 2,150

Ajeya Cotra 26 Sep 2023 2:56 UTC
LW: 48 AF: 25
0
AF
on: There should be more AI safety orgs
(Cross-posted to EA Forum.)
I’m a Senior Program Officer at Open Phil, focused on technical AI safety funding. I’m hearing a lot of discussion suggesting funding is very tight right now for AI safety, so I wanted to give my take on the situation.
At a high level: AI safety is a top priority for Open Phil, and we are aiming to grow how much we spend in that area. There are many potential projects we’d be excited to fund, including some potential new AI safety orgs as well as renewals to existing grantees, academic research projects, upskilling grants, and more.
At the same time, it is also not the case that someone who reads this post and tries to start an AI safety org would necessarily have an easy time raising funding from us. This is because:
- All of our teams whose work touches on AI (Luke Muehlhauser’s team on AI governance, Claire Zabel’s team on capacity building, and me on technical AI safety) are quite understaffed at the moment. We’ve hired several people recently, but across the board we still don’t have the capacity to evaluate all the plausible AI-related grants, and hiring remains a top priority for us.
  - And we are extra-understaffed for evaluating technical AI safety proposals in particular. I am the only person who is primarily focused on funding technical research projects (sometimes Claire’s team funds AI safety related grants, primarily upskilling, but a large technical AI safety grant like a new research org would fall to me). I currently have no team members; I expect to have one person joining in October and am aiming to launch a wider hiring round soon, but I think it’ll take me several months to build my team’s capacity up substantially.
  - I began making grants in November 2022, and spent the first few months full-time evaluating applicants affected by FTX (largely academic PIs as opposed to independent organizations started by members of the EA community). Since then, a large chunk of my time has gone into maintaining and renewing existing grant commitments and evaluating grant opportunities referred to us by existing advisors. I am aiming to reserve remaining bandwidth for thinking through strategic priorities, articulating what research directions seem highest-priority and encouraging researchers to work on them (through conversations and hopefully soon through more public communication), and hiring for my team or otherwise helping Open Phil build evaluation capacity in AI safety (including separately from my team).
  - As a result, I have deliberately held off on launching open calls for grant applications similar to the ones run by Claire’s team (e.g. this one); before onboarding more people (and developing or strengthening internal processes), I would not have the bandwidth to keep up with the applications.
- On top of this, in our experience, providing seed funding to new organizations (particularly organizations started by younger and less experienced founders) often leads to complications that aren’t present in funding academic research or career transition grants. We prefer to think carefully about seeding new organizations, and have a different and higher bar for funding someone to start an org than for funding that same person for other purposes (e.g. career development and transition funding, or PhD and postdoc funding).
  - I’m very uncertain about how to think about seeding new research organizations and many related program strategy questions. I could certainly imagine developing a different picture upon further reflection — but having low capacity combines poorly with the fact that this is a complex type of grant we are uncertain about on a lot of dimensions. We haven’t had the senior staff bandwidth to develop a clear stance on the strategic or process level about this genre of grant, and that means that we are more hesitant to take on such grant investigations — and if / when we do, it takes up more scarce capacity to think through the considerations in a bespoke way rather than having a clear policy to fall back on.

Ajeya Cotra 26 Jan 2023 2:06 UTC
LW: 31 AF: 15
9
AF
in reply to: habryka’s comment on: Thoughts on the impact of RLHF research

my guess is most of that success is attributable to the work on RLHF, since that was really the only substantial difference between Chat-GPT and GPT-3

I don’t think this is right—the main hype effect of chatGPT over previous models feels like it’s just because it was in a convenient chat interface that was easy to use and free. My guess is that if you did a head-to-head comparison of RLHF and kludgey random hacks involving imitation and prompt engineering, they’d seem similarly cool to a random journalist / VC, and generate similar excitement.

Ajeya Cotra 14 Dec 2022 21:16 UTC
LW: 4 AF: 3
AF
in reply to: Richard_Ngo’s comment on: ricraz’s Shortform

I strongly disagree with the “best case” thing. Like, policies could just learn human values! It’s not that implausible.

Yes, sorry, “best case” was oversimplified. What I meant is that generalizing to want reward is in some sense the model generalizing “correctly;” we could get lucky and have it generalize “incorrectly” in an important sense in a way that happens to be beneficial to us. I discuss this a bit more here.

But if Alex did initially develop a benevolent goal like “empower humans,” the straightforward and “naive” way of acting on that goal would have been disincentivized early in training. As I argued above, if Alex had behaved in a straightforwardly benevolent way at all times, it would not have been able to maximize reward effectively.

That means even if Alex had developed a benevolent goal, it would have needed to play the training game as well as possible—including lying and manipulating humans in a way that naively seems in conflict with that goal. If its benevolent goal had caused it to play the training game less ruthlessly, it would’ve had a constant incentive to move away from having that goal or at least from acting on it.[35] If Alex actually retained the benevolent goal through the end of training, then it probably strategically chose to act exactly as if it were maximizing reward.

This means we could have replaced this hypothetical benevolent goal with a wide variety of other goals without changing Alex’s behavior or reward in the lab setting at all—“help humans” is just one possible goal among many that Alex could have developed which would have all resulted in exactly the same behavior in the lab setting.

If I had to try point to the crux here, it might be “how much selection pressure is needed to make policies learn goals that are abstractly related to their training data, as opposed to goals that are fairly concretely related to their training data?”...As usual, there’s the human analogy: our goals are very strongly biased towards things we have direct observational access to!)

I don’t understand why reward isn’t something the model has direct access to—it seems like it basically does? If I had to say which of us were focusing on abstract vs concrete goals, I’d have said I was thinking about concrete goals and you were thinking about abstract ones, so I think we have some disagreement of intuition here.

Even setting aside this disagreement, though, I don’t like the argumentative structure because the generalization of “reward” to large scales is much less intuitive than the generalization of other concepts (like “make money”) to large scales—in part because directly having a goal of reward is a kinda counterintuitive self-referential thing.

Yeah, I don’t really agree with this; I think I could pretty easily imagine being an AI system asking the question “How much reward would this episode get if it were sampled for training?” It seems like the intuition this is weird and unnatural is doing a lot of work in your argument, and I don’t really share it.

Ajeya Cotra 14 Dec 2022 19:14 UTC
LW: 3 AF: 2
AF
in reply to: Lauro Langosco’s comment on: ricraz’s Shortform
Yeah, I agree this is a good argument structure—in my mind, maximizing reward is both a plausible case (which Richard might disagree with) and the best case (conditional on it being strategic at all and not a bag of heuristics), so it’s quite useful to establish that it’s doomed; that’s the kind of structure I was going for in the post.

Ajeya Cotra 14 Dec 2022 19:03 UTC
LW: 7 AF: 6
AF
in reply to: Richard_Ngo’s comment on: ricraz’s Shortform
Note that the “without countermeasures” post consistently discusses both possibilities (the model cares about reward or the model cares about something else that’s consistent with it getting very high reward on the training dataset). E.g. see this paragraph from the above-the-fold intro:

Once this progresses far enough, the best way for Alex to accomplish most possible “goals” no longer looks like “essentially give humans what they want but take opportunities to manipulate them here and there.” It looks more like “seize the power to permanently direct how it uses its time and what rewards it receives—and defend against humans trying to reassert control over it, including by eliminating them.” This seems like Alex’s best strategy whether it’s trying to get large amounts of reward or has other motives. If it’s trying to maximize reward, this strategy would allow it to force its incoming rewards to be high indefinitely.[6] If it has other motives, this strategy would give it long-term freedom, security, and resources to pursue those motives.

As well as the section Even if Alex isn’t “motivated” to maximize reward.… I do place a ton of emphasis on the fact that Alex enacts a policy which has the empirical effect of maximizing reward, but that’s distinct from being confident in the motivations that give rise to that policy. I believe Alex would try very hard to maximize reward in most cases, but this could be for either terminal or instrumental reasons.

With that said, for roughly the reasons Paul says above, I think I probably do have a disagreement with Richard—I think that caring about some version of reward is pretty plausible (~50% or so). It seems pretty natural and easy to grasp to me, and because I think there will likely be continuous online training the argument that there’s no notion of reward on the deployment distribution doesn’t feel compelling to me.

Ajeya Cotra 4 Aug 2022 16:49 UTC
LW: 7 AF: 3
0
AF
in reply to: Rob Bensinger’s comment on: Two-year update on my personal AI timelines
Yeah I agree more of the value of this kind of exercise (at least within the community) is in revealing more granular disagreements about various things. But I do think there’s value in establishing to more external people something high level like “It really could be soon and it’s not crazy or sci fi to think so.”

Ajeya Cotra 4 Aug 2022 16:47 UTC
LW: 3 AF: 2
0
AF
in reply to: Shiroe’s comment on: Two-year update on my personal AI timelines

Can you say more about what particular applications you had in mind?

Stuff like personal assistants who write emails / do simple shopping, coding assistants that people are more excited about than they seem to be about Codex, etc.

(Like I said in the main post, I’m not totally sure what PONR refers to, but don’t think I agree that the first lucrative application marks a PONR—seems like there are a bunch of things you can do after that point, including but not limited to alignment research.)

Ajeya Cotra 4 Aug 2022 16:46 UTC
LW: 9 AF: 5
6
AF
in reply to: Adam Scholl’s comment on: Two-year update on my personal AI timelines
I don’t see it that way, no. Today’s coding models can help automate some parts of the ML researcher workflow a little bit, and I think tomorrow’s coding models will automate more and more complex parts, and so on. I think this expansion could be pretty rapid, but I don’t think it’ll look like “not much going on until something snaps into place.”

Ajeya Cotra 3 Aug 2022 5:09 UTC
LW: 7 AF: 3
1
AF
in reply to: Zach Stein-Perlman’s comment on: Two-year update on my personal AI timelines
(Coherence aside, when I now look at that number it does seem a bit too high, and I feel tempted to move it to 2027-2028, but I dunno, that kind of intuition is likely to change quickly from day to day.)

Ajeya Cotra 3 Aug 2022 0:05 UTC
LW: 16 AF: 4
3
AF
in reply to: Zach Stein-Perlman’s comment on: Two-year update on my personal AI timelines
Hm, yeah, I bet if I reflected more things would shift around, but I’m not sure the fact that there’s a shortish period where the per-year probability is very elevated followed by a longer period with lower per-year probability is actually a bad sign.

Roughly speaking, right now we’re in an AI boom where spending on compute for training big models is going up rapidly, and it’s fairly easy to actually increase spending quickly because the current levels are low. There’s some chance of transformative AI in the middle of this spending boom—and because resource inputs are going up a ton each year, the probability of TAI by date X would also be increasing pretty rapidly.

But the current spending boom is pretty unsustainable if it doesn’t lead to TAI. At some point in the 2040s or 50s, if we haven’t gotten transformative AI by then, we’ll have been spending 10s of billions training models, and it won’t be that easy to keep ramping up quickly from there. And then because the input growth will have slowed, the increase in probability from one year to the next will also slow. (That said, not sure how this works out exactly.)

Ajeya Cotra 25 Jul 2022 1:45 UTC
LW: 6 AF: 2
2
AF
in reply to: TurnTrout’s comment on: Without specific countermeasures, the easiest path to transformative AI likely leads to AI takeover

Where does the selection come from? Will the designers toss a really impressive AI for not getting reward on that one timestep? I think not.

I was talking about gradient descent here, not designers.

Ajeya Cotra 24 Jul 2022 2:43 UTC
LW: 9 AF: 7
0
AF
in reply to: Ben Pace’s comment on: Without specific countermeasures, the easiest path to transformative AI likely leads to AI takeover
It doesn’t seem like it would have to prevent us from building computers if it has access to far more compute than we could access on Earth. It would just be powerful enough to easily defeat the kind of AIs we could train with the relatively meager computing resources we could extract from Earth. In general the AI is a superpower and humans are dramatically technologically behind, so it seems like it has many degrees of freedom and doesn’t have to be particularly watching for this.

Ajeya Cotra 23 Jul 2022 18:52 UTC
LW: 4 AF: 3
0
AF
in reply to: Ben Pace’s comment on: Without specific countermeasures, the easiest path to transformative AI likely leads to AI takeover
Neutralizing computational capabilities doesn’t seem to involve total destruction of physical matter or human extinction though, especially for a very powerful being. Seems like it’d be basically just as easy to ensure we + future AIs we might train are no threat as it is to vaporize the Earth.

Ajeya Cotra 22 Jul 2022 22:53 UTC
LW: 11 AF: 3
6
AF
in reply to: greg’s comment on: Without specific countermeasures, the easiest path to transformative AI likely leads to AI takeover
My answer is a little more prosaic than Raemon. I don’t feel at all confident that an AI that already had God-like abilities would choose to literally kill all humans to use their bodies’ atoms for its own ends; it seems totally plausible to me that—whether because of exotic things like “multiverse-wide super-rationality” or “acausal trade” or just “being nice”—the AI will leave Earth alone, since (as you say) it would be very cheap for it to do so.

The thing I’m referring to as “takeover” is the measures that an AI would take to make sure that humans can’t take back control—while it’s not fully secure and doesn’t have God-like abilities. Once a group of AIs have decided to try to get out of human control, they’re functionally at war with humanity. Humans could do things like physically destroy the datacenters they’re running on, and they would probably want to make sure they can’t do that.

Securing AI control and defending from human counter-moves seems likely to involve some violence—but it could be a scale of violence that’s “merely” in line with historical instances where a technologically more advanced group of humans colonized or took control of a less-advanced group of humans; most historical takeovers don’t involve literally killing every single member of the other group.

The key point is that it seems likely that AIs will secure the power to get to decide what happens with the future; I’m pretty unsure exactly how they use it, and especially if it involves physically destroying Earth / killing all humans for resources. These resources seem pretty meager compared to the rest of the universe.

Ajeya Cotra 22 Jul 2022 15:37 UTC
3 points
0
in reply to: Mathieu Putz’s comment on: Without specific countermeasures, the easiest path to transformative AI likely leads to AI takeover
I mean things like tricks to improve the sample efficiency of human feedback, doing more projects that are un-enhanced RLHF to learn things about how un-enhanced RLHF works, etc.

Ajeya Cotra 20 Jul 2022 23:43 UTC
LW: 22 AF: 10
14
AF
in reply to: habryka’s comment on: Without specific countermeasures, the easiest path to transformative AI likely leads to AI takeover
I’m pretty confused about how to think about the value of various ML alignment papers. But I think even if some piece of empirical ML work on alignment is really valuable for reducing x-risk, I wouldn’t expect its value to take the form of providing insight to readers like you or me. So you as a reader not getting much out of it is compatible with the work being super valuable, and we probably need to assess it on different terms.

The main channel of value that I see for doing work like “learning to summarize” and the critiques project and various interpretability projects is something like “identifying a tech tree that it seems helpful to get as far as possible along by the Singularity, and beginning to climb that tech tree.”

In the case of critiques—ultimately, it seems like having AIs red team each other and pointing out ways that another AI’s output could be dangerous seems like it will make a quantitative difference. If we had a really well-oiled debate setup, then we would catch issues we wouldn’t have caught with vanilla human feedback, meaning our models could get smarter before they pose an existential threat—and these smarter models can more effectively work on problems like alignment for us.[1]

It seems good to have that functionality developed as far as it can be developed in as many frontier labs as possible. The first steps of that look kind of boring, and don’t substantially change our view of the problem. But first steps are the foundation for later steps, and the baseline against which you compare later steps. (Also every step can seem boring in the sense of bringing no game-changing insights, while nonetheless helping a lot.)

When the main point of some piece of work is to get good at something that seems valuable to be really good at later, and to build tacit knowledge and various kinds of infrastructure for doing that thing, a paper about it is not going to feel that enlightening to someone who wants high-level insights that change their picture of the overall problem. (Kind of like someone writing a blog post about how they developed effective management and performance evaluation processes at their company isn’t going to provide much insight into the abstract theory of principal-agent problems. The value of that activity was in the company running better, not people learning things from the blog post about it.)

I’m still not sure how valuable I think this work is, because I don’t know how well it’s doing at efficiently climbing tech trees or at picking the right tech trees, but I think that’s how I’d think about evaluating it.

[1] Or do a “pivotal act,” though I think I probably don’t agree with some of the connotations of that term.

Ajeya Cotra 20 Jul 2022 21:59 UTC
LW: 12 AF: 8
6
AF
in reply to: habryka’s comment on: Without specific countermeasures, the easiest path to transformative AI likely leads to AI takeover
I was mainly talking about the current margin when I talked about how excited I am about the theoretical vs empirical work I see “going on” right now and how excited I tend to be about currently-active researchers who are doing theory vs empirical research. And I was talking about the future when I said that I expect empirical work to end up with the lion’s share of credit for AI risk reduction.

Eliezer, Bostrom, and co certainly made a big impact in raising the problem to people’s awareness and articulating some of its contours. It’s kind of a matter of semantics whether you want to call that “theoretical research” or “problem advocacy” / “cause prioritization” / “community building” / whatever, and no matter which bucket you put it in I agree it’ll probably end up with an outsized impact for x-risk-reduction, by bringing the problem to attention sooner than it would have otherwise been brought to attention and therefore probably allowing more work to happen on it before TAI is developed.

But just like how founding CEOs tend to end up with ~10% equity once their companies have grown large, I don’t think this historical problem-advocacy-slash-theoretical-research work alone will end up with a very large amount of total credit.

On the main thrust of my point, I’m significantly less excited about MIRI-sphere work that is much less like “articulating a problem and advocating for its importance” and much more like “attempting to solve a problem.” E.g. stuff like logical inductors, embedded agency, etc seem a lot less valuable to me than stuff like the orthogonality thesis and so on.

Ajeya Cotra 20 Jul 2022 17:14 UTC
6 points
2
in reply to: PeterMcCluskey’s comment on: Without specific countermeasures, the easiest path to transformative AI likely leads to AI takeover
I think the retroactive editing of rewards (not just to punish explicitly bad action but to slightly improve evaluation of everything) is actually pretty default, though I understand if people disagree. It seems like an extremely natural thing to do that would make your AI more capable and make it more likely to pass most behavioral safety interventions.

In other words, even if the average episode length is short (e.g. 1 hour), I think the default outcome is to have the rewards for that episode be computed as far after the fact as possible, because that helps Alex improve at long-range planning (a skill Magma would try hard to get it to have). This can be done in a way that doesn’t compromise speed of training—you simply reward Alex immediately with your best guess reward, then keep editing it later as more information comes in. At all points in time you have a “good enough” reward ready to go, while also capturing the benefits of pushing your model to think in as long-term a way as possible.

Ajeya Cotra 20 Jul 2022 15:05 UTC
3 points
0
in reply to: Shiroe’s comment on: Without specific countermeasures, the easiest path to transformative AI likely leads to AI takeover
Thanks, but I’m not working on that project! That project is led by Beth Barnes.

Ajeya Cotra 19 Jul 2022 23:44 UTC
LW: 14 AF: 9
1
AF
in reply to: TurnTrout’s comment on: Without specific countermeasures, the easiest path to transformative AI likely leads to AI takeover
Hm, not sure I understand but I wasn’t trying to make super specific mechanistic claims here—I agree that what I said doesn’t reduce confusion about the specific internal mechanisms of how lying gets to be hard for most humans, but I wasn’t intending to claim that it was. I also should have said something like “evolutionary, cultural, and individual history” instead (I was using “evolution” as a shorthand to indicate it seems common among various cultures but of course that doesn’t mean don’t-lie genes are directly bred into us! Most human universals aren’t; we probably don’t have honor-the-dead and different-words-for-male-and-female genes).

I was just making the pretty basic point “AIs in general, and Alex in particular, are produced through a very different process from humans, so it seems like ‘humans find lying hard’ is pretty weak evidence that ‘AI will by default find lying hard.’”

I agree that asking “What specific neurological phenomena make it so most people find it hard to lie?” could serve as inspiration to do AI honesty research, and wasn’t intending to claim otherwise in that paragraph (though separately, I am somewhat pessimistic about this research direction).