Akash

• Akash 31 May 2023 18:16 UTC

  LW: 6 AF: 3

  2

  AF

  on: An­nounc­ing Apollo Research

  Congratulations on launching!

  On the governance side, one question I’d be excited to see Apollo (and ARC evals & any other similar groups) think/​write about is: what happens after a dangerous capability eval goes off?

  Of course, the actual answer will be shaped by the particular climate/​culture/​zeitgeist/​policy window/​lab factors that are impossible to fully predict in advance.

  But my impression is that this question is relatively neglected, and I wouldn’t be surprised if sharp newcomers were able to meaningfully improve the community’s thinking on this.

AI Safety Newslet­ter #8: Rogue AIs, how to screen for AI risks, and grants for re­search on demo­cratic gov­er­nance of AI

• Akash 26 May 2023 19:39 UTC

  7 points

  3

  on: Seek­ing (Paid) Case Stud­ies on Standards

  Excited to see this! I’d be most excited about case studies of standards in fields where people didn’t already have clear ideas about how to verify safety.

  In some areas, it’s pretty clear what you’re supposed to do to verify safety. Everyone (more-or-less) agrees on what counts as safe.

  One of the biggest challenges with AI safety standards will be the fact that no one really knows how to verify that a (sufficiently-powerful) system is safe. And a lot of experts disagree on the type of evidence that would be sufficient.

  Are there examples of standards in other industries where people were quite confused about what “safety” would require? Are there examples of standards that are specific enough to be useful but flexible enough to deal with unexpected failure modes or threats? Are there examples where the standards-setters acknowledged that they wouldn’t be able to make a simple checklist, so they requested that companies provide proactive evidence of safety?

• Akash 25 May 2023 13:37 UTC

  9 points

  0

  on: The Office of Science and Tech­nol­ogy Policy put out a re­quest for in­for­ma­tion on A.I.

  I’ve been working on a response to the NTIA request for comments on AI Accountability over the last few months. It’s likely that I’ll also submit something to the OSTP request.

  I’ve learned a few useful things from talking to AI governance and policy folks. Some of it is fairly intuitive but still worth highlighting (e.g., try to avoid jargon, remember that the reader doesn’t share many assumptions that people in AI safety take for granted, remember that people have many different priorities). Some of it is less intuitive (e.g., what actually happens with the responses? How long should your response be? How important is it to say something novel? What kinds of things are policymakers actually looking for?)

  If anyone is looking for advice, feel free to DM me.

AI Safety Newslet­ter #7: Dis­in­for­ma­tion, Gover­nance Recom­men­da­tions for AI labs, and Se­nate Hear­ings on AI

Eisen­hower’s Atoms for Peace Speech

AI Safety Newslet­ter #6: Ex­am­ples of AI safety progress, Yoshua Ben­gio pro­poses a ban on AI agents, and les­sons from nu­clear arms control

AI Safety Newslet­ter #5: Ge­offrey Hin­ton speaks out on AI risk, the White House meets with AI labs, and Tro­jan at­tacks on lan­guage models

• Akash 4 May 2023 1:31 UTC

  6 points

  3

  on: How MATS ad­dresses “mass move­ment build­ing” concerns

  Glad to see this write-up & excited for more posts.

  I think these are three areas that MATS feels like it has handled fairly well. I’d be especially excited to hear more about areas where MATS thinks it’s struggling, MATS is uncertain, or where MATS feels like it has a lot of room to grow. Potential candidates include:

  • How is MATS going about talent selection and advertising for the next cohort, especially given the recent wave of interest in AI/​AI safety?

  • How does MATS intend to foster (or recruit) the kinds of qualities that strong researchers often possess?

  • How does MATS define “good” alignment research?

  Other things I’m be curious about:

  • Which work from previous MATS scholars is the MATS team most excited about? What are MATS’s biggest wins? Which individuals or research outputs is MATS most proud of?

  • Most peoples’ timelines have shortened a lot since MATS was established. Does this substantially reduce the value of MATS (relative to worlds with longer timelines)?

  • Does MATS plan to try to attract senior researchers who are becoming interested in AI Safety (e.g., professors, people with 10+ years of experience in industry)? Or will MATS continue to recruit primarily from the (largely younger and less experienced) EA/​LW communities?

AI Safety Newslet­ter #4: AI and Cy­ber­se­cu­rity, Per­sua­sive AIs, Weaponiza­tion, and Ge­offrey Hin­ton talks AI risks

• Akash 2 May 2023 17:22 UTC

  5 points

  −3

  on: AGI safety ca­reer advice

  (Pasting this exchange from a comment thread on the EA Forum; bolding added)

  Peter Park:

  Thank you so much for your insightful and detailed list of ideas for AGI safety careers, Richard! I really appreciate your excellent post.

  I would propose explicitly grouping some of your ideas and additional ones under a third category: “identifying and raising public awareness of AGI’s dangers.” In fact, I think this category may plausibly contain some of the most impactful ideas for reducing catastrophic and existential risks, given that alignment seems potentially difficult to achieve in a reasonable period of time (if ever) and the implementation of governance ideas is bottlenecked by public support.

  For a similar argument that I found particularly compelling, please check out Greg Colbourn’s recent post: https://​​forum.effectivealtruism.org/​​posts/​​8YXFaM9yHbhiJTPqp/​​agi-rising-why-we-are-in-a-new-era-of-acute-risk-and

  Richard:

  I don’t actually think the implementation of governance ideas is mainly bottlenecked by public support; I think it’s bottlenecked by good concrete proposals. And to the extent that it is bottlenecked by public support, that will change by default as more powerful AI systems are released.

  Akash:

  I appreciate Richard stating this explicitly. I think this is (and has been) a pretty big crux in the AI governance space right now.

  Some folks (like Richard) believe that we’re mainly bottlenecked by good concrete proposals. Other folks believe that we have concrete proposals, but we need to raise awareness and political support in order to implement them.

  I’d like to see more work going into both of these areas. On the margin, though, I’m currently more excited about efforts to raise awareness [well], acquire political support, and channel that support into achieving useful policies.

  I think this is largely due to (a) my perception that this work is largely neglected, (b) the fact that a few AI governance professionals I trust have also stated that they see this as the higher priority thing at the moment, and (c) worldview beliefs around what kind of regulation is warranted (e.g., being more sympathetic to proposals that require a lot of political will).

Dis­cus­sion about AI Safety fund­ing (FB tran­script)

• Akash 27 Apr 2023 20:11 UTC

  2 points

  0

  in reply to: jacquesthibs’s comment on: Refram­ing the bur­den of proof: Com­pa­nies should prove that mod­els are safe (rather than ex­pect­ing au­di­tors to prove that mod­els are dan­ger­ous)

  Nice—very relevant. I agree with Evan that arguments about the training procedure will be relevant (I’m more uncertain about whether checking for deception behaviorally will be harder than avoiding it, but it certainly seems plausible).

  Ideally, I think the regulators would be flexible in the kind of evidence they accept. If a developer has evidence that the model is not deceptive that relies on details about the training procedure, rather than behavioral testing, that could be sufficient.

  (In fact, I think arguments that meet some sort of “beyond-a-reasonable-doubt” threshold would likely involve providing arguments for why the training procedure avoids deceptive alignment.)

• Akash 27 Apr 2023 19:11 UTC

  4 points

  2

  in reply to: aphyer’s comment on: Refram­ing the bur­den of proof: Com­pa­nies should prove that mod­els are safe (rather than ex­pect­ing au­di­tors to prove that mod­els are dan­ger­ous)

  Can you say more about what part of this relates to a ban on AI development?

  I think the claim “AI development should be regulated in a way such that the burden of proof is on developers to show beyond-a-reasonable-doubt that models are safe” seems quite different from the claim “AI development should be banned”, but it’s possible that I’m missing something here or communicating imprecisely.

• Akash 25 Apr 2023 19:16 UTC

  3 points

  1

  in reply to: Marius Hobbhahn’s comment on: Refram­ing the bur­den of proof: Com­pa­nies should prove that mod­els are safe (rather than ex­pect­ing au­di­tors to prove that mod­els are dan­ger­ous)

  This makes sense. Can you say more about how aviation regulation differs from the FDA?

  In other words, are there meaningful differences in how the regulatory processes are set up? Or does it just happen to be the case that the FDA has historically been worse at responding to evidence compared to the Federal Aviation Administration?

  (I think it’s plausible that we would want a structure similar to the FDA even if the particular individuals at the FDA were bad at cost-benefit analysis, unless there are arguments that the structure of the FDA caused the bad cost-benefit analyses).

Refram­ing the bur­den of proof: Com­pa­nies should prove that mod­els are safe (rather than ex­pect­ing au­di­tors to prove that mod­els are dan­ger­ous)

Deep­Mind and Google Brain are merg­ing [Linkpost]

AI Safety Newslet­ter #2: ChaosGPT, Nat­u­ral Selec­tion, and AI Safety in the Media

Re­quest to AGI or­ga­ni­za­tions: Share your views on paus­ing AI progress

AI Safety Newslet­ter #1 [CAIS Linkpost]