AI Risk

TagLast edit: 2 Nov 2022 20:27 UTC by brook

AI Risk is analysis of the risks associated with building powerful AI systems.

Superintelligence FAQ

Scott Alexander20 Sep 2016 19:00 UTC

118 points

32 comments27 min readLW link

What failure looks like

paulfchristiano17 Mar 2019 20:18 UTC

391 points

53 comments8 min readLW link 2 reviews

Specification gaming examples in AI

Vika3 Apr 2018 12:30 UTC

43 points

9 comments1 min readLW link 2 reviews

An artificially structured argument for expecting AGI ruin

Rob Bensinger7 May 2023 21:52 UTC

91 points

26 comments19 min readLW link

MIRI announces new “Death With Dignity” strategy

Eliezer Yudkowsky2 Apr 2022 0:43 UTC

333 points

534 comments18 min readLW link

Discussion with Eliezer Yudkowsky on AGI interventions

Rob Bensinger and Eliezer Yudkowsky

11 Nov 2021 3:01 UTC

328 points

251 comments34 min readLW link 1 review

PreDCA: vanessa kosoy’s alignment protocol

Tamsin Leake20 Aug 2022 10:03 UTC

50 points

8 comments7 min readLW link

(carado.moe)

“Corrigibility at some small length” by dath ilan

Christopher King5 Apr 2023 1:47 UTC

32 points

3 comments9 min readLW link

(www.glowfic.com)

Intuitions about goal-directed behavior

Rohin Shah1 Dec 2018 4:25 UTC

54 points

15 comments6 min readLW link

Epistemological Framing for AI Alignment Research

adamShimi8 Mar 2021 22:05 UTC

55 points

7 comments9 min readLW link

Conjecture internal survey: AGI timelines and probability of human extinction from advanced AI

Maris Sala22 May 2023 14:31 UTC

153 points

5 comments3 min readLW link

(www.conjecture.dev)

AGI Ruin: A List of Lethalities

Eliezer Yudkowsky5 Jun 2022 22:05 UTC

863 points

683 comments30 min readLW link

AGI in sight: our look at the game board

Andrea_Miotti and Gabriel Alfour

18 Feb 2023 22:17 UTC

226 points

135 comments6 min readLW link

(andreamiotti.substack.com)

Where I agree and disagree with Eliezer

paulfchristiano19 Jun 2022 19:15 UTC

862 points

217 comments18 min readLW link

Open Problems in AI X-Risk [PAIS #5]

Dan H and ThomasW

10 Jun 2022 2:08 UTC

57 points

6 comments36 min readLW link

What can the principal-agent literature tell us about AI risk?

apc8 Feb 2020 21:28 UTC

104 points

29 comments16 min readLW link

On how various plans miss the hard bits of the alignment challenge

So8res12 Jul 2022 2:49 UTC

292 points

83 comments29 min readLW link

Another (outer) alignment failure story

paulfchristiano7 Apr 2021 20:12 UTC

236 points

38 comments12 min readLW link 1 review

[Question] Will OpenAI’s work unintentionally increase existential risks related to AI?

adamShimi11 Aug 2020 18:16 UTC

53 points

56 comments1 min readLW link

Developmental Stages of GPTs

orthonormal26 Jul 2020 22:03 UTC

140 points

71 comments7 min readLW link 1 review

A Gym Gridworld Environment for the Treacherous Turn

Michaël Trazzi28 Jul 2018 21:27 UTC

74 points

9 comments3 min readLW link

(github.com)

Bing chat is the AI fire alarm

Ratios17 Feb 2023 6:51 UTC

112 points

62 comments3 min readLW link

Stampy’s AI Safety Info—New Distillations #1 [March 2023] (Expansive interactive FAQ)

markov7 Apr 2023 11:06 UTC

42 points

0 comments2 min readLW link

(aisafety.info)

Robin Hanson’s latest AI risk position statement

Liron3 Mar 2023 14:25 UTC

55 points

17 comments1 min readLW link

(www.overcomingbias.com)

Meta AI announces Cicero: Human-Level Diplomacy play (with dialogue)

Jacy Reese Anthis22 Nov 2022 16:50 UTC

93 points

64 comments1 min readLW link

(www.science.org)

A transcript of the TED talk by Eliezer Yudkowsky

Mikhail Samin12 Jul 2023 12:12 UTC

102 points

12 comments4 min readLW link

AI will change the world, but won’t take it over by playing “3-dimensional chess”.

boazbarak and benedelman

22 Nov 2022 18:57 UTC

128 points

95 comments24 min readLW link

Don’t accelerate problems you’re trying to solve

Andrea_Miotti and remember

15 Feb 2023 18:11 UTC

100 points

26 comments4 min readLW link

Are minimal circuits deceptive?

evhub7 Sep 2019 18:11 UTC

74 points

11 comments8 min readLW link

AI Could Defeat All Of Us Combined

HoldenKarnofsky9 Jun 2022 15:50 UTC

170 points

42 comments17 min readLW link

(www.cold-takes.com)

“Endgame safety” for AGI

Steven Byrnes24 Jan 2023 14:15 UTC

84 points

10 comments6 min readLW link

Devil’s Advocate: Adverse Selection Against Conscientiousness

lionhearted (Sebastian Marshall)28 May 2023 17:53 UTC

10 points

2 comments1 min readLW link

Being at peace with Doom

Johannes C. Mayer9 Apr 2023 14:53 UTC

24 points

11 comments4 min readLW link

How good is humanity at coordination?

Buck21 Jul 2020 20:01 UTC

81 points

44 comments3 min readLW link

My Objections to “We’re All Gonna Die with Eliezer Yudkowsky”

Quintin Pope21 Mar 2023 0:06 UTC

364 points

210 comments39 min readLW link

Request to AGI organizations: Share your views on pausing AI progress

Akash and simeon_c

11 Apr 2023 17:30 UTC

141 points

11 comments1 min readLW link

Architects of Our Own Demise: We Should Stop Developing AI

Roko26 Oct 2023 0:36 UTC

168 points

73 comments3 min readLW link

Did Bengio and Tegmark lose a debate about AI x-risk against LeCun and Mitchell?

Karl von Wendt25 Jun 2023 16:59 UTC

106 points

50 comments7 min readLW link

On Solving Problems Before They Appear: The Weird Epistemologies of Alignment

adamShimi11 Oct 2021 8:20 UTC

107 points

10 comments15 min readLW link

Intent alignment should not be the goal for AGI x-risk reduction

John Nay26 Oct 2022 1:24 UTC

1 point

10 comments3 min readLW link

The Hidden Complexity of Wishes

Eliezer Yudkowsky24 Nov 2007 0:12 UTC

163 points

142 comments7 min readLW link

Truthful LMs as a warm-up for aligned AGI

Jacob_Hilton17 Jan 2022 16:49 UTC

65 points

14 comments13 min readLW link

Soft takeoff can still lead to decisive strategic advantage

Daniel Kokotajlo23 Aug 2019 16:39 UTC

122 points

47 comments8 min readLW link 4 reviews

Announcing Apollo Research

Marius Hobbhahn, beren, Lee Sharkey, Lucius Bushnaq, Dan Braun, Mikita Balesni and Jérémy Scheurer

30 May 2023 16:17 UTC

214 points

10 comments8 min readLW link

Alexander and Yudkowsky on AGI goals

Scott Alexander and Eliezer Yudkowsky

24 Jan 2023 21:09 UTC

174 points

52 comments26 min readLW link

Should we postpone AGI until we reach safety?

otto.barten18 Nov 2020 15:43 UTC

27 points

36 comments3 min readLW link

A challenge for AGI organizations, and a challenge for readers

Rob Bensinger and Eliezer Yudkowsky

1 Dec 2022 23:11 UTC

296 points

33 comments2 min readLW link

DL towards the unaligned Recursive Self-Optimization attractor

jacob_cannell18 Dec 2021 2:15 UTC

32 points

22 comments4 min readLW link

An Appeal to AI Superintelligence: Reasons to Preserve Humanity

James_Miller18 Mar 2023 16:22 UTC

30 points

72 comments12 min readLW link

[Question] How likely are scenarios where AGI ends up overtly or de facto torturing us? How likely are scenarios where AGI prevents us from committing suicide or dying?

JohnGreer28 Mar 2023 18:00 UTC

11 points

4 comments1 min readLW link

[Question] First and Last Questions for GPT-5*

Mitchell_Porter24 Nov 2023 5:03 UTC

14 points

5 comments1 min readLW link

Response to Oren Etzioni’s “How to know if artificial intelligence is about to destroy civilization”

Daniel Kokotajlo27 Feb 2020 18:10 UTC

27 points

5 comments8 min readLW link

“Why can’t you just turn it off?”

Roko19 Nov 2023 14:46 UTC

39 points

25 comments1 min readLW link

Survey: What (de)motivates you about AI risk?

Daniel_Friedrich3 Aug 2022 19:17 UTC

1 point

0 comments1 min readLW link

(forms.gle)

Catastrophic Risks from AI #5: Rogue AIs

Dan H, Mantas Mazeika and ThomasW

27 Jun 2023 22:06 UTC

15 points

0 comments22 min readLW link

(arxiv.org)

Catastrophic Risks from AI #1: Introduction

Dan H, Mantas Mazeika and ThomasW

22 Jun 2023 17:09 UTC

40 points

1 comment5 min readLW link

(arxiv.org)

Why don’t singularitarians bet on the creation of AGI by buying stocks?

John_Maxwell11 Mar 2020 16:27 UTC

43 points

20 comments4 min readLW link

Google’s Ethical AI team and AI Safety

magfrump20 Feb 2021 9:42 UTC

12 points

16 comments7 min readLW link

[Question] Why don’t quantilizers also cut off the upper end of the distribution?

Alex_Altair15 May 2023 1:40 UTC

25 points

2 comments1 min readLW link

[Linkpost] Existential Risk Analysis in Empirical Research Papers

Dan H2 Jul 2022 0:09 UTC

40 points

0 comments1 min readLW link

(arxiv.org)

Confusion about neuroscience/cognitive science as a danger for AI Alignment

Samuel Nellessen22 Jun 2022 17:59 UTC

2 points

1 comment3 min readLW link

(snellessen.com)

Rogue AGI Embodies Valuable Intellectual Property

Mark Xu and CarlShulman

3 Jun 2021 20:37 UTC

71 points

9 comments3 min readLW link

Shapes of Mind and Pluralism in Alignment

adamShimi13 Aug 2022 10:01 UTC

33 points

2 comments2 min readLW link

Behavioral Sufficient Statistics for Goal-Directedness

adamShimi11 Mar 2021 15:01 UTC

21 points

12 comments9 min readLW link

[Question] Should people build productizations of open source AI models?

lc2 Nov 2023 1:26 UTC

21 points

0 comments1 min readLW link

Alex Turner’s Research, Comprehensive Information Gathering

adamShimi23 Jun 2021 9:44 UTC

15 points

3 comments3 min readLW link

[Question] Does the Structure of an algorithm matter for AI Risk and/or consciousness?

Logan Zoellner3 Dec 2021 18:31 UTC

7 points

4 comments1 min readLW link

AI Safety is Dropping the Ball on Clown Attacks

trevor22 Oct 2023 20:09 UTC

51 points

69 comments34 min readLW link

A conversation about Katja’s counterarguments to AI risk

Matthew Barnett and Ege Erdil

18 Oct 2022 18:40 UTC

43 points

9 comments33 min readLW link

Bayeswatch 7: Wildfire

lsusr8 Sep 2021 5:35 UTC

48 points

6 comments3 min readLW link

Why the technological singularity by AGI may never happen

hippke3 Sep 2021 14:19 UTC

5 points

14 comments1 min readLW link

Approaches to gradient hacking

adamShimi14 Aug 2021 15:16 UTC

16 points

8 comments8 min readLW link

Announcing AISIC 2022 - the AI Safety Israel Conference, October 19-20

Davidmanheim21 Sep 2022 19:32 UTC

13 points

0 comments1 min readLW link

Epistemic Strategies of Selection Theorems

adamShimi18 Oct 2021 8:57 UTC

33 points

1 comment12 min readLW link

Refine’s Third Blog Post Day/Week

adamShimi17 Sep 2022 17:03 UTC

18 points

0 comments1 min readLW link

Empowerment is (almost) All We Need

jacob_cannell23 Oct 2022 21:48 UTC

59 points

44 comments17 min readLW link

AISN #24: Kissinger Urges US-China Cooperation on AI, China’s New AI Law, US Export Controls, International Institutions, and Open Source AI

aogara, Dan H and Corin Katzke

18 Oct 2023 17:06 UTC

14 points

0 comments6 min readLW link

(newsletter.safe.ai)

Podcast: Shoshannah Tekofsky on skilling up in AI safety, visiting Berkeley, and developing novel research ideas

Akash25 Nov 2022 20:47 UTC

37 points

2 comments9 min readLW link

Poster Session on AI Safety

Neil Crawford12 Nov 2022 3:50 UTC

7 points

6 comments1 min readLW link

[Question] Why are we sure that AI will “want” something?

shminux16 Sep 2022 20:35 UTC

31 points

57 comments1 min readLW link

Non-Adversarial Goodhart and AI Risks

Davidmanheim27 Mar 2018 1:39 UTC

22 points

11 comments6 min readLW link

Projects I would like to see (possibly at AI Safety Camp)

Linda Linsefors27 Sep 2023 21:27 UTC

16 points

9 comments4 min readLW link

Microdooms averted by working on AI Safety

nikola17 Sep 2023 21:46 UTC

30 points

2 comments3 min readLW link

(forum.effectivealtruism.org)

Ten Levels of AI Alignment Difficulty

Sammy Martin3 Jul 2023 20:20 UTC

97 points

10 comments12 min readLW link

Catastrophic Risks from AI #6: Discussion and FAQ

Dan H, Mantas Mazeika and ThomasW

27 Jun 2023 23:23 UTC

24 points

1 comment13 min readLW link

(arxiv.org)

If you wish to make an apple pie, you must first become dictator of the universe

jasoncrawford5 Jul 2023 18:14 UTC

27 points

9 comments13 min readLW link

(rootsofprogress.org)

Catastrophic Risks from AI #4: Organizational Risks

Dan H, Mantas Mazeika and ThomasW

26 Jun 2023 19:36 UTC

23 points

0 comments21 min readLW link

(arxiv.org)

The Control Problem: Unsolved or Unsolvable?

Remmelt2 Jun 2023 15:42 UTC

43 points

45 comments11 min readLW link

But exactly how complex and fragile?

KatjaGrace3 Nov 2019 18:20 UTC

82 points

32 comments3 min readLW link 1 review

(meteuphoric.com)

Risks from AI Overview: Summary

Dan H, Mantas Mazeika and ThomasW

18 Aug 2023 1:21 UTC

25 points

0 comments13 min readLW link

(www.safe.ai)

Apply to lead a project during the next virtual AI Safety Camp

Linda Linsefors and Remmelt

13 Sep 2023 13:29 UTC

19 points

0 comments5 min readLW link

(aisafety.camp)

Beyond Hyperanthropomorphism

PointlessOne21 Aug 2022 17:55 UTC

3 points

17 comments1 min readLW link

(studio.ribbonfarm.com)

Some abstract, non-technical reasons to be non-maximally-pessimistic about AI alignment

Rob Bensinger12 Dec 2021 2:08 UTC

70 points

35 comments7 min readLW link

All AGI Safety questions welcome (especially basic ones) [~monthly thread]

Robert Miles1 Nov 2022 23:23 UTC

68 points

105 comments2 min readLW link

Announcement: AI alignment prize winners and next round

cousin_it15 Jan 2018 14:33 UTC

80 points

68 comments2 min readLW link

Using Brain-Computer Interfaces to get more data for AI alignment

Robbo7 Nov 2021 0:00 UTC

42 points

10 comments7 min readLW link

Epistemic Strategies of Safety-Capabilities Tradeoffs

adamShimi22 Oct 2021 8:22 UTC

5 points

0 comments6 min readLW link

Is progress in ML-assisted theorem-proving beneficial?

mako yass28 Sep 2021 1:54 UTC

11 points

3 comments1 min readLW link

[Book Review] “The Alignment Problem” by Brian Christian

lsusr20 Sep 2021 6:36 UTC

70 points

16 comments6 min readLW link

Some disjunctive reasons for urgency on AI risk

Wei Dai15 Feb 2019 20:43 UTC

36 points

24 comments1 min readLW link

Interview with Skynet

lsusr30 Sep 2021 2:20 UTC

49 points

1 comment2 min readLW link

Drexler on AI Risk

PeterMcCluskey1 Feb 2019 5:11 UTC

35 points

10 comments9 min readLW link

(www.bayesianinvestor.com)

All AGI safety questions welcome (especially basic ones) [Sept 2022]

plex8 Sep 2022 11:56 UTC

22 points

48 comments2 min readLW link

Mechanism Design for AI Safety—Reading Group Curriculum

Rubi J. Hudson25 Oct 2022 3:54 UTC

15 points

3 comments1 min readLW link

The Fusion Power Generator Scenario

johnswentworth8 Aug 2020 18:31 UTC

139 points

29 comments3 min readLW link

AI Alignment 2018-19 Review

Rohin Shah28 Jan 2020 2:19 UTC

126 points

6 comments35 min readLW link

Brainstorming additional AI risk reduction ideas

John_Maxwell14 Jun 2012 7:55 UTC

19 points

37 comments1 min readLW link

Information warfare historically revolved around human conduits

trevor28 Aug 2023 18:54 UTC

37 points

7 comments3 min readLW link

What if we approach AI safety like a technical engineering safety problem

zeshen20 Aug 2022 10:29 UTC

32 points

4 comments7 min readLW link

25 Min Talk on MetaEthical.AI with Questions from Stuart Armstrong

June Ku29 Apr 2021 15:38 UTC

21 points

7 comments1 min readLW link

Learning societal values from law as part of an AGI alignment strategy

John Nay21 Oct 2022 2:03 UTC

5 points

18 comments54 min readLW link

DeepMind alignment team opinions on AGI ruin arguments

Vika12 Aug 2022 21:06 UTC

375 points

36 comments14 min readLW link

The alignment problem from a deep learning perspective

Richard_Ngo10 Aug 2022 22:46 UTC

97 points

13 comments27 min readLW link

April drafts

AI Impacts1 Apr 2021 18:10 UTC

49 points

2 comments1 min readLW link

(aiimpacts.org)

Catastrophic Risks from AI #3: AI Race

Dan H, Mantas Mazeika and ThomasW

23 Jun 2023 19:21 UTC

18 points

9 comments29 min readLW link

(arxiv.org)

Robustness to Scaling Down: More Important Than I Thought

adamShimi23 Jul 2022 11:40 UTC

37 points

5 comments3 min readLW link

Request: stop advancing AI capabilities

So8res26 May 2023 17:42 UTC

154 points

23 comments1 min readLW link

The Alignment Problem

lsusr11 Jul 2022 3:03 UTC

46 points

18 comments3 min readLW link

Difficulties in making powerful aligned AI

DanielFilan14 May 2023 20:50 UTC

41 points

1 comment10 min readLW link

(danielfilan.com)

My Overview of the AI Alignment Landscape: A Bird’s Eye View

Neel Nanda15 Dec 2021 23:44 UTC

126 points

9 comments15 min readLW link

AI Fire Alarm Scenarios

PeterMcCluskey28 Dec 2021 2:20 UTC

10 points

0 comments6 min readLW link

(www.bayesianinvestor.com)

An unaligned benchmark

paulfchristiano17 Nov 2018 15:51 UTC

31 points

0 comments9 min readLW link

The strategy-stealing assumption

paulfchristiano16 Sep 2019 15:23 UTC

84 points

46 comments12 min readLW link 3 reviews

A plea for solutionism on AI safety

jasoncrawford9 Jun 2023 16:29 UTC

72 points

6 comments6 min readLW link

(rootsofprogress.org)

Why and When Interpretability Work is Dangerous

NicholasKross28 May 2023 0:27 UTC

20 points

7 comments8 min readLW link

(www.thinkingmuchbetter.com)

AI Safety Newsletter #6: Examples of AI safety progress, Yoshua Bengio proposes a ban on AI agents, and lessons from nuclear arms control

Dan H, Akash and aogara

16 May 2023 15:14 UTC

31 points

0 comments6 min readLW link

(newsletter.safe.ai)

Will Artificial Superintelligence Kill Us?

James_Miller23 May 2023 16:27 UTC

33 points

2 comments22 min readLW link

Activation additions in a simple MNIST network

Garrett Baker18 May 2023 2:49 UTC

26 points

0 comments2 min readLW link

The case for removing alignment and ML research from the training dataset

beren30 May 2023 20:54 UTC

48 points

8 comments5 min readLW link

Systems that cannot be unsafe cannot be safe

Davidmanheim2 May 2023 8:53 UTC

62 points

27 comments2 min readLW link

Stuxnet, not Skynet: Humanity’s disempowerment by AI

Roko4 Nov 2023 22:23 UTC

103 points

22 comments6 min readLW link

A Case for the Least Forgiving Take On Alignment

Thane Ruthenis2 May 2023 21:34 UTC

89 points

71 comments22 min readLW link

[Linkpost] Biden-Harris Executive Order on AI

beren30 Oct 2023 15:20 UTC

6 points

0 comments1 min readLW link

RA Bounty: Looking for feedback on screenplay about AI Risk

Writer26 Oct 2023 13:23 UTC

30 points

6 comments1 min readLW link

Quote quiz: “drifting into dependence”

jasoncrawford27 Apr 2023 15:13 UTC

7 points

6 comments1 min readLW link

(rootsofprogress.org)

[Question] How much do personal biases in risk assessment affect assessment of AI risks?

Gordon Seidoh Worley3 May 2023 6:12 UTC

10 points

8 comments1 min readLW link

Minimum Viable Exterminator

Richard Horvath29 May 2023 16:32 UTC

14 points

5 comments5 min readLW link

AISN #23: New OpenAI Models, News from Anthropic, and Representation Engineering

aogara and Dan H

4 Oct 2023 17:37 UTC

15 points

2 comments5 min readLW link

(newsletter.safe.ai)

AI Takeover Scenario with Scaled LLMs

simeon_c16 Apr 2023 23:28 UTC

42 points

15 comments8 min readLW link

Financial Times: We must slow down the race to God-like AI

trevor13 Apr 2023 19:55 UTC

103 points

17 comments16 min readLW link

(www.ft.com)

All images from the WaitButWhy sequence on AI

trevor8 Apr 2023 7:36 UTC

72 points

5 comments2 min readLW link

Reliability, Security, and AI risk: Notes from infosec textbook chapter 1

Akash7 Apr 2023 15:47 UTC

34 points

1 comment4 min readLW link

“warning about ai doom” is also “announcing capabilities progress to noobs”

the gears to ascension8 Apr 2023 23:42 UTC

16 points

5 comments3 min readLW link

All AGI Safety questions welcome (especially basic ones) [April 2023]

steven04618 Apr 2023 4:21 UTC

57 points

88 comments2 min readLW link

Distilled—AGI Safety from First Principles

Harrison G29 May 2022 0:57 UTC

10 points

1 comment14 min readLW link

[AN #80]: Why AI risk might be solved without additional intervention from longtermists

Rohin Shah2 Jan 2020 18:20 UTC

36 points

95 comments10 min readLW link

(mailchi.mp)

Why I don’t believe in doom

mukashi7 Jun 2022 23:49 UTC

6 points

30 comments4 min readLW link

[Link] Sarah Constantin: “Why I am Not An AI Doomer”

lbThingrb12 Apr 2023 1:52 UTC

61 points

13 comments1 min readLW link

(sarahconstantin.substack.com)

Top lesson from GPT: we will probably destroy humanity “for the lulz” as soon as we are able.

shminux16 Apr 2023 20:27 UTC

65 points

28 comments1 min readLW link

Another plausible scenario of AI risk: AI builds military infrastructure while collaborating with humans, defects later.

avturchin10 Jun 2022 17:24 UTC

10 points

2 comments1 min readLW link

Chad Jones paper modeling AI and x-risk vs. growth

jasoncrawford26 Apr 2023 20:07 UTC

39 points

7 comments2 min readLW link

(web.stanford.edu)

On A List of Lethalities

Zvi13 Jun 2022 12:30 UTC

160 points

48 comments54 min readLW link

(thezvi.wordpress.com)

We’re Not Ready: thoughts on “pausing” and responsible scaling policies

HoldenKarnofsky27 Oct 2023 15:19 UTC

188 points

29 comments8 min readLW link

Thoughts on Robin Hanson’s AI Impacts interview

Steven Byrnes24 Nov 2019 1:40 UTC

25 points

3 comments7 min readLW link

AISN #25: White House Executive Order on AI, UK AI Safety Summit, and Progress on Voluntary Evaluations of AI Risks

aogara and Dan H

31 Oct 2023 19:34 UTC

35 points

1 comment6 min readLW link

(newsletter.safe.ai)

The other side of the tidal wave

KatjaGrace3 Nov 2023 5:40 UTC

164 points

77 comments1 min readLW link

(worldspiritsockpuppet.com)

Will GPT-5 be able to self-improve?

Nathan Helm-Burger29 Apr 2023 17:34 UTC

17 points

22 comments3 min readLW link

Are we there yet?

theflowerpot20 Jun 2022 11:19 UTC

2 points

2 comments1 min readLW link

AI Safety Newsletter #5: Geoffrey Hinton speaks out on AI risk, the White House meets with AI labs, and Trojan attacks on language models

Dan H, Akash and aogara

9 May 2023 15:26 UTC

28 points

1 comment4 min readLW link

(newsletter.safe.ai)

Let’s build a fire alarm for AGI

chaosmage15 May 2023 9:16 UTC

−2 points

0 comments2 min readLW link

Activation additions in a small residual network

Garrett Baker22 May 2023 20:28 UTC

22 points

4 comments3 min readLW link

AI Safety Newsletter #7: Disinformation, Governance Recommendations for AI labs, and Senate Hearings on AI

Dan H, Akash and aogara

23 May 2023 21:47 UTC

25 points

0 comments6 min readLW link

(newsletter.safe.ai)

[Question] Suggestions of posts on the AF to review

adamShimi16 Feb 2021 12:40 UTC

56 points

20 comments1 min readLW link

Hands-On Experience Is Not Magic

Thane Ruthenis27 May 2023 16:57 UTC

20 points

14 comments5 min readLW link

AI Safety Newsletter #1 [CAIS Linkpost]

Akash, Dan H, aogara and ozhang

10 Apr 2023 20:18 UTC

45 points

0 comments4 min readLW link

(newsletter.safe.ai)

A moral backlash against AI will probably slow down AGI development

geoffreymiller7 Jun 2023 20:39 UTC

48 points

10 comments14 min readLW link

Catastrophic Risks from AI #2: Malicious Use

Dan H, Mantas Mazeika and ThomasW

22 Jun 2023 17:10 UTC

37 points

1 comment17 min readLW link

(arxiv.org)

Comparing Four Approaches to Inner Alignment

Lucas Teixeira29 Jul 2022 21:06 UTC

35 points

1 comment9 min readLW link

Clarifying some key hypotheses in AI alignment

Ben Cottier and Rohin Shah

15 Aug 2019 21:29 UTC

79 points

12 comments9 min readLW link

My research agenda in agent foundations

Alex_Altair28 Jun 2023 18:00 UTC

70 points

6 comments11 min readLW link

Levels of safety for AI and other technologies

jasoncrawford28 Jun 2023 18:35 UTC

16 points

0 comments2 min readLW link

(rootsofprogress.org)

Douglas Hofstadter changes his mind on Deep Learning & AI risk (June 2023)?

gwern3 Jul 2023 0:48 UTC

401 points

52 comments7 min readLW link

(www.youtube.com)

Oversight Misses 100% of Thoughts The AI Does Not Think

johnswentworth12 Aug 2022 16:30 UTC

90 points

50 comments1 min readLW link

Arguments against existential risk from AI, part 2

Nina Rimsky10 Jul 2023 8:25 UTC

6 points

0 comments5 min readLW link

(ninarimsky.substack.com)

Winners of AI Alignment Awards Research Contest

Akash and OliviaJ

13 Jul 2023 16:14 UTC

114 points

2 comments12 min readLW link

(alignmentawards.com)

Thoughts on ‘List of Lethalities’

Alex Lawsen 17 Aug 2022 18:33 UTC

27 points

0 comments10 min readLW link

Thoughts on sharing information about language model capabilities

paulfchristiano31 Jul 2023 16:04 UTC

188 points

34 comments11 min readLW link

The problem/solution matrix: Calculating the probability of AI safety “on the back of an envelope”

John_Maxwell20 Oct 2019 8:03 UTC

22 points

4 comments2 min readLW link

Paper: On measuring situational awareness in LLMs

Owain_Evans, Daniel Kokotajlo, Mikita Balesni, Tomek Korbak, lberglund, Asa Cooper Stickland, Meg and Maximilian Kaufmann

4 Sep 2023 12:54 UTC

101 points

15 comments5 min readLW link

(arxiv.org)

[Question] Measure of complexity allowed by the laws of the universe and relative theory?

dr_s7 Sep 2023 12:21 UTC

8 points

22 comments1 min readLW link

Environmental Structure Can Cause Instrumental Convergence

TurnTrout22 Jun 2021 22:26 UTC

71 points

43 comments16 min readLW link

(arxiv.org)

What I Learned Running Refine

adamShimi24 Nov 2022 14:49 UTC

107 points

5 comments4 min readLW link

Sam Altman and Ezra Klein on the AI Revolution

Zack_M_Davis27 Jun 2021 4:53 UTC

38 points

17 comments1 min readLW link

(www.nytimes.com)

Framing approaches to alignment and the hard problem of AI cognition

ryan_greenblatt15 Dec 2021 19:06 UTC

16 points

15 comments27 min readLW link

A shift in arguments for AI risk

Richard_Ngo28 May 2019 13:47 UTC

32 points

7 comments1 min readLW link

(fragile-credences.github.io)

Ngo and Yudkowsky on alignment difficulty

Eliezer Yudkowsky and Richard_Ngo

15 Nov 2021 20:31 UTC

248 points

148 comments99 min readLW link 1 review

All AGI Safety questions welcome (especially basic ones) [~monthly thread]

mwatkins and Robert Miles

26 Jan 2023 21:01 UTC

39 points

80 comments2 min readLW link

Using blinders to help you see things for what they are

Adam Zerner11 Nov 2021 7:07 UTC

13 points

2 comments2 min readLW link

“Taking AI Risk Seriously” (thoughts by Critch)

Raemon29 Jan 2018 9:27 UTC

110 points

68 comments13 min readLW link

Drug addicts and deceptively aligned agents—a comparative analysis

Jan5 Nov 2021 21:42 UTC

42 points

2 comments12 min readLW link

(universalprior.substack.com)

Results from the language model hackathon

Esben Kran10 Oct 2022 8:29 UTC

22 points

1 comment4 min readLW link

Work on Security Instead of Friendliness?

Wei Dai21 Jul 2012 18:28 UTC

65 points

107 comments2 min readLW link

[Question] Did AI pioneers not worry much about AI risks?

lisperati9 Feb 2020 19:58 UTC

42 points

9 comments1 min readLW link

Critiquing “What failure looks like”

Grue_Slinky27 Dec 2019 23:59 UTC

35 points

6 comments3 min readLW link

A guide to Iterated Amplification & Debate

Rafael Harth15 Nov 2020 17:14 UTC

72 points

9 comments15 min readLW link

Wizards and prophets of AI [draft for comment]

jasoncrawford31 Mar 2023 20:22 UTC

16 points

11 comments6 min readLW link

[Question] Conditional on the first AGI being aligned correctly, is a good outcome even still likely?

iamthouthouarti6 Sep 2021 17:30 UTC

2 points

1 comment1 min readLW link

AI Doom Is Not (Only) Disjunctive

NickGabs30 Mar 2023 1:42 UTC

12 points

0 comments5 min readLW link

Orthogonality is expensive

beren3 Apr 2023 10:20 UTC

33 points

8 comments3 min readLW link

What would a compute monitoring plan look like? [Linkpost]

Akash26 Mar 2023 19:33 UTC

157 points

9 comments4 min readLW link

(arxiv.org)

[Question] What are good alignment conference papers?

adamShimi28 Aug 2021 13:35 UTC

12 points

2 comments1 min readLW link

Counterarguments to the basic AI x-risk case

KatjaGrace14 Oct 2022 13:00 UTC

360 points

123 comments34 min readLW link

(aiimpacts.org)

How Josiah became an AI safety researcher

Neil Crawford6 Sep 2022 17:17 UTC

4 points

0 comments1 min readLW link

Deceptive Alignment

evhub, Chris van Merwijk, Vlad Mikulik, Joar Skalse and Scott Garrabrant

5 Jun 2019 20:16 UTC

117 points

20 comments17 min readLW link

AGI Safety Literature Review (Everitt, Lea & Hutter 2018)

Kaj_Sotala4 May 2018 8:56 UTC

14 points

1 comment1 min readLW link

(arxiv.org)

Applications for AI Safety Camp 2022 Now Open!

adamShimi17 Nov 2021 21:42 UTC

47 points

3 comments1 min readLW link

Thinking soberly about the context and consequences of Friendly AI

Mitchell_Porter16 Oct 2012 4:33 UTC

21 points

39 comments1 min readLW link

AI Safety Microgrant Round

Chris_Leong14 Nov 2022 4:25 UTC

22 points

1 comment1 min readLW link

Summary of the Acausal Attack Issue for AIXI

Diffractor13 Dec 2021 8:16 UTC

14 points

6 comments4 min readLW link

Complex Systems are Hard to Control

jsteinhardt4 Apr 2023 0:00 UTC

42 points

5 comments10 min readLW link

(bounded-regret.ghost.io)

[Question] Why not constrain wetlabs instead of AI?

Lone Pine21 Mar 2023 18:02 UTC

13 points

10 comments1 min readLW link

The Wizard of Oz Problem: How incentives and narratives can skew our perception of AI developments

Akash20 Mar 2023 20:44 UTC

16 points

3 comments6 min readLW link

Three Stories for How AGI Comes Before FAI

John_Maxwell17 Sep 2019 23:26 UTC

27 points

5 comments6 min readLW link

No One-Size-Fit-All Epistemic Strategy

adamShimi20 Aug 2022 12:56 UTC

23 points

1 comment2 min readLW link

Less Realistic Tales of Doom

Mark Xu6 May 2021 23:01 UTC

113 points

13 comments4 min readLW link

Why AI Safety is Hard

Simon Möller22 Mar 2023 10:44 UTC

3 points

0 comments6 min readLW link

ChatGPT (and now GPT4) is very easily distracted from its rules

dmcs15 Mar 2023 17:55 UTC

178 points

42 comments1 min readLW link

Relaxed adversarial training for inner alignment

evhub10 Sep 2019 23:03 UTC

65 points

27 comments27 min readLW link

Six AI Risk/Strategy Ideas

Wei Dai27 Aug 2019 0:40 UTC

64 points

17 comments4 min readLW link 1 review

My current uncertainties regarding AI, alignment, and the end of the world

dominicq14 Nov 2021 14:08 UTC

2 points

3 comments2 min readLW link

Stanford Encyclopedia of Philosophy on AI ethics and superintelligence

Kaj_Sotala2 May 2020 7:35 UTC

43 points

19 comments7 min readLW link

(plato.stanford.edu)

I Think Eliezer Should Go on Glenn Beck

Lao Mein30 Jun 2023 3:12 UTC

25 points

21 comments1 min readLW link

The Overton Window widens: Examples of AI risk in the media

Akash23 Mar 2023 17:10 UTC

107 points

24 comments6 min readLW link

Review of “Fun with +12 OOMs of Compute”

adamShimi, Joe_Collman and Gyrodiot

28 Mar 2021 14:55 UTC

63 points

21 comments8 min readLW link 1 review

New survey: 46% of Americans are concerned about extinction from AI; 69% support a six-month pause in AI development

Akash5 Apr 2023 1:26 UTC

46 points

9 comments1 min readLW link

(today.yougov.com)

DeepMind and Google Brain are merging [Linkpost]

Akash20 Apr 2023 18:47 UTC

55 points

5 comments1 min readLW link

(www.deepmind.com)

Questions about Conjecure’s CoEm proposal

Akash, CharlotteS and NicholasKees

9 Mar 2023 19:32 UTC

51 points

4 comments2 min readLW link

Contra “Strong Coherence”

DragonGod4 Mar 2023 20:05 UTC

39 points

24 comments1 min readLW link

[Linkpost] Scott Alexander reacts to OpenAI’s latest post

Akash11 Mar 2023 22:24 UTC

27 points

0 comments5 min readLW link

(astralcodexten.substack.com)

The Preference Fulfillment Hypothesis

Kaj_Sotala26 Feb 2023 10:55 UTC

66 points

62 comments11 min readLW link

Full Transcript: Eliezer Yudkowsky on the Bankless podcast

remember and Andrea_Miotti

23 Feb 2023 12:34 UTC

138 points

89 comments75 min readLW link

Confusions in My Model of AI Risk

peterbarnett7 Jul 2022 1:05 UTC

22 points

9 comments5 min readLW link

Evil autocomplete: Existential Risk and Next-Token Predictors

Yitz28 Feb 2023 8:47 UTC

9 points

3 comments5 min readLW link

An AI risk argument that resonates with NYTimes readers

Julian Bradshaw12 Mar 2023 23:09 UTC

199 points

13 comments1 min readLW link

Clarifying “What failure looks like”

Sam Clarke20 Sep 2020 20:40 UTC

95 points

14 comments17 min readLW link

[RETRACTED] It’s time for EA leadership to pull the short-timelines fire alarm.

Not Relevant8 Apr 2022 16:07 UTC

111 points

163 comments4 min readLW link

All AGI Safety questions welcome (especially basic ones) [May 2023]

steven04618 May 2023 22:30 UTC

33 points

44 comments2 min readLW link

My Overview of the AI Alignment Landscape: Threat Models

Neel Nanda25 Dec 2021 23:07 UTC

52 points

3 comments28 min readLW link

Artificial Intelligence: A Modern Approach (4th edition) on the Alignment Problem

Zack_M_Davis17 Sep 2020 2:23 UTC

72 points

12 comments5 min readLW link

(aima.cs.berkeley.edu)

More Is Different for AI

jsteinhardt4 Jan 2022 19:30 UTC

137 points

22 comments3 min readLW link

(bounded-regret.ghost.io)

4 ways to think about democratizing AI [GovAI Linkpost]

Akash13 Feb 2023 18:06 UTC

24 points

4 comments1 min readLW link

(www.governance.ai)

Challenges with Breaking into MIRI-Style Research

Chris_Leong17 Jan 2022 9:23 UTC

75 points

15 comments3 min readLW link

AXRP Episode 13 - First Principles of AGI Safety with Richard Ngo

DanielFilan31 Mar 2022 5:20 UTC

24 points

1 comment48 min readLW link

Many important technologies start out as science fiction before becoming real

trevor10 Feb 2023 9:36 UTC

26 points

2 comments2 min readLW link

Bankless Podcast: 159 - We’re All Gonna Die with Eliezer Yudkowsky

bayesed20 Feb 2023 16:42 UTC

83 points

54 comments1 min readLW link

(www.youtube.com)

Plan for mediocre alignment of brain-like [model-based RL] AGI

Steven Byrnes13 Mar 2023 14:11 UTC

63 points

24 comments12 min readLW link

Many AI governance proposals have a tradeoff between usefulness and feasibility

Akash and Carson Ezell

3 Feb 2023 18:49 UTC

22 points

2 comments2 min readLW link

[Linkpost] TIME article: DeepMind’s CEO Helped Take AI Mainstream. Now He’s Urging Caution

Akash21 Jan 2023 16:51 UTC

56 points

2 comments3 min readLW link

(time.com)

Against a General Factor of Doom

Jeffrey Heninger23 Nov 2022 16:50 UTC

61 points

18 comments5 min readLW link

(aiimpacts.org)

ea.domains—Domains Free to a Good Home

plex12 Jan 2023 13:32 UTC

24 points

0 comments1 min readLW link

[Question] Would (myopic) general public good producers significantly accelerate the development of AGI?

mako yass2 Mar 2022 23:47 UTC

25 points

10 comments1 min readLW link

How we could stumble into AI catastrophe

HoldenKarnofsky13 Jan 2023 16:20 UTC

64 points

18 comments18 min readLW link

(www.cold-takes.com)

How I Formed My Own Views About AI Safety

Neel Nanda27 Feb 2022 18:50 UTC

64 points

6 comments13 min readLW link

(www.neelnanda.io)

What Failure Looks Like: Distilling the Discussion

Ben Pace29 Jul 2020 21:49 UTC

81 points

14 comments7 min readLW link

World-Model Interpretability Is All We Need

Thane Ruthenis14 Jan 2023 19:37 UTC

29 points

21 comments21 min readLW link

Thoughts on refusing harmful requests to large language models

William_S19 Jan 2023 19:49 UTC

30 points

4 comments2 min readLW link

Talk to me about your summer/career plans

Akash31 Jan 2023 18:29 UTC

31 points

3 comments2 min readLW link

Taboo P(doom)

NathanBarnard3 Feb 2023 10:37 UTC

11 points

10 comments1 min readLW link

How evals might (or might not) prevent catastrophic risks from AI

Akash7 Feb 2023 20:16 UTC

40 points

0 comments9 min readLW link

Uber Self-Driving Crash

jefftk7 Nov 2019 15:00 UTC

110 points

1 comment2 min readLW link

(www.jefftk.com)

It Looks Like You’re Trying To Take Over The World

gwern9 Mar 2022 16:35 UTC

396 points

119 comments1 min readLW link

(www.gwern.net)

AI Safety Info Distillation Fellowship

Robert Miles and mwatkins

17 Feb 2023 16:16 UTC

47 points

3 comments3 min readLW link

Paradigm-building: Introduction

Cameron Berg8 Feb 2022 0:06 UTC

28 points

0 comments2 min readLW link

AMA Conjecture, A New Alignment Startup

adamShimi9 Apr 2022 9:43 UTC

47 points

42 comments1 min readLW link

Paradigm-building from first principles: Effective altruism, AGI, and alignment

Cameron Berg8 Feb 2022 16:12 UTC

26 points

5 comments14 min readLW link

AI Safety “Success Stories”

Wei Dai7 Sep 2019 2:54 UTC

116 points

27 comments4 min readLW link 1 review

Reply to Holden on ‘Tool AI’

Eliezer Yudkowsky12 Jun 2012 18:00 UTC

152 points

356 comments17 min readLW link

AI #1: Sydney and Bing

Zvi21 Feb 2023 14:00 UTC

170 points

44 comments61 min readLW link

(thezvi.wordpress.com)

Incentives and Selection: A Missing Frame From AI Threat Discussions?

DragonGod26 Feb 2023 1:18 UTC

11 points

16 comments2 min readLW link

Racing through a minefield: the AI deployment problem

HoldenKarnofsky22 Dec 2022 16:10 UTC

36 points

2 comments13 min readLW link

(www.cold-takes.com)

AI #2

Zvi2 Mar 2023 14:50 UTC

66 points

18 comments55 min readLW link

(thezvi.wordpress.com)

Contra Hanson on AI Risk

Liron4 Mar 2023 8:02 UTC

36 points

23 comments8 min readLW link

Against ubiquitous alignment taxes

beren6 Mar 2023 19:50 UTC

56 points

10 comments2 min readLW link

Disentangling arguments for the importance of AI safety

Richard_Ngo21 Jan 2019 12:41 UTC

133 points

23 comments8 min readLW link

An overview of 11 proposals for building safe advanced AI

evhub29 May 2020 20:38 UTC

209 points

36 comments38 min readLW link 2 reviews

ARC tests to see if GPT-4 can escape human control; GPT-4 failed to do so

Christopher King15 Mar 2023 0:29 UTC

116 points

22 comments2 min readLW link

Podcast: Tamera Lanham on AI risk, threat models, alignment proposals, externalized reasoning oversight, and working at Anthropic

Akash20 Dec 2022 21:39 UTC

18 points

2 comments11 min readLW link

Risks from Learned Optimization: Introduction

evhub, Chris van Merwijk, Vlad Mikulik, Joar Skalse and Scott Garrabrant

31 May 2019 23:44 UTC

183 points

42 comments12 min readLW link 3 reviews

Risks from Learned Optimization: Conclusion and Related Work

evhub, Chris van Merwijk, Vlad Mikulik, Joar Skalse and Scott Garrabrant

7 Jun 2019 19:53 UTC

82 points

5 comments6 min readLW link

My thoughts on OpenAI’s alignment plan

Akash30 Dec 2022 19:33 UTC

55 points

3 comments20 min readLW link

Thoughts on AGI safety from the top

jylin042 Feb 2022 20:06 UTC

36 points

3 comments32 min readLW link

Why I’m Worried About AI

peterbarnett23 May 2022 21:13 UTC

22 points

2 comments12 min readLW link

Complex Systems for AI Safety [Pragmatic AI Safety #3]

Dan H and ThomasW

24 May 2022 0:00 UTC

57 points

2 comments21 min readLW link

The Inner Alignment Problem

evhub, Chris van Merwijk, Vlad Mikulik, Joar Skalse and Scott Garrabrant

4 Jun 2019 1:20 UTC

102 points

17 comments13 min readLW link

Conditions for Mesa-Optimization

evhub, Chris van Merwijk, Vlad Mikulik, Joar Skalse and Scott Garrabrant

1 Jun 2019 20:52 UTC

80 points

48 comments12 min readLW link

Four lenses on AI risks

jasoncrawford28 Mar 2023 21:52 UTC

23 points

5 comments3 min readLW link

(rootsofprogress.org)

AI risk hub in Singapore?

Daniel Kokotajlo29 Oct 2020 11:45 UTC

57 points

18 comments4 min readLW link

Will working here advance AGI? Help us not destroy the world!

Yonatan Cale29 May 2022 11:42 UTC

30 points

46 comments1 min readLW link

Some conceptual highlights from “Disjunctive Scenarios of Catastrophic AI Risk”

Kaj_Sotala12 Feb 2018 12:30 UTC

33 points

4 comments6 min readLW link

(kajsotala.fi)

Perform Tractable Research While Avoiding Capabilities Externalities [Pragmatic AI Safety #4]

Dan H and ThomasW

30 May 2022 20:25 UTC

51 points

3 comments25 min readLW link

Confused why a “capabilities research is good for alignment progress” position isn’t discussed more

Kaj_Sotala2 Jun 2022 21:41 UTC

129 points

27 comments4 min readLW link

I’m trying out “asteroid mindset”

Alex_Altair3 Jun 2022 13:35 UTC

90 points

5 comments4 min readLW link

Epistemological Vigilance for Alignment

adamShimi6 Jun 2022 0:27 UTC

60 points

11 comments10 min readLW link

A Quick Guide to Confronting Doom

Ruby13 Apr 2022 19:30 UTC

240 points

33 comments2 min readLW link

n=3 AI Risk Quick Math and Reasoning

lionhearted (Sebastian Marshall)7 Apr 2023 20:27 UTC

6 points

3 comments4 min readLW link

AI Safety Seems Hard to Measure

HoldenKarnofsky8 Dec 2022 19:50 UTC

69 points

6 comments14 min readLW link

(www.cold-takes.com)

Debate on Instrumental Convergence between LeCun, Russell, Bengio, Zador, and More

Ben Pace4 Oct 2019 4:08 UTC

221 points

61 comments15 min readLW link 2 reviews

Where’s the foom?

Fergus Fettes11 Apr 2023 15:50 UTC

34 points

27 comments2 min readLW link

AI Neorealism: a threat model & success criterion for existential safety

davidad15 Dec 2022 13:42 UTC

56 points

1 comment3 min readLW link

Wentworth and Larsen on buying time

Akash, Thomas Larsen and johnswentworth

9 Jan 2023 21:31 UTC

73 points

6 comments12 min readLW link

A (EtA: quick) note on terminology: AI Alignment != AI x-safety

David Scott Krueger (formerly: capybaralet)8 Feb 2023 22:33 UTC

46 points

20 comments1 min readLW link

How dangerous is human-level AI?

Alex_Altair10 Jun 2022 17:38 UTC

21 points

4 comments8 min readLW link

“Carefully Bootstrapped Alignment” is organizationally hard

Raemon17 Mar 2023 18:00 UTC

246 points

20 comments11 min readLW link

Reframing the burden of proof: Companies should prove that models are safe (rather than expecting auditors to prove that models are dangerous)

Akash25 Apr 2023 18:49 UTC

27 points

11 comments3 min readLW link

(childrenoficarus.substack.com)

Talking publicly about AI risk

Jan_Kulveit21 Apr 2023 11:28 UTC

173 points

8 comments6 min readLW link

Sensor Exposure can Compromise the Human Brain in the 2020s

trevor26 Oct 2023 3:31 UTC

17 points

2 comments10 min readLW link

The Main Sources of AI Risk?

Daniel Kokotajlo and Wei Dai

21 Mar 2019 18:28 UTC

115 points

26 comments2 min readLW link

Continuity Assumptions

Jan_Kulveit13 Jun 2022 21:31 UTC

33 points

13 comments4 min readLW link

Slow motion videos as AI risk intuition pumps

Andrew_Critch14 Jun 2022 19:31 UTC

232 points

40 comments2 min readLW link

Alignment Risk Doesn’t Require Superintelligence

JustisMills15 Jun 2022 3:12 UTC

35 points

4 comments2 min readLW link

[Question] Has there been any work on attempting to use Pascal’s Mugging to make an AGI behave?

Chris_Leong15 Jun 2022 8:33 UTC

7 points

17 comments1 min readLW link

The AI Safety Game (UPDATED)

Daniel Kokotajlo5 Dec 2020 10:27 UTC

44 points

10 comments3 min readLW link

Memes and Rational Decisions

inferential9 Jan 2015 6:42 UTC

35 points

18 comments10 min readLW link

An Increasingly Manipulative Newsfeed

Michaël Trazzi1 Jul 2019 15:26 UTC

62 points

16 comments5 min readLW link

Reviews of “Is power-seeking AI an existential risk?”

Joe Carlsmith16 Dec 2021 20:48 UTC

78 points

20 comments1 min readLW link

Thoughts on the Feasibility of Prosaic AGI Alignment?

iamthouthouarti21 Aug 2020 23:25 UTC

8 points

10 comments1 min readLW link

Universality and the “Filter”

maggiehayes16 Dec 2021 0:47 UTC

10 points

2 comments11 min readLW link

New AI risks research institute at Oxford University

lukeprog16 Nov 2011 18:52 UTC

36 points

10 comments1 min readLW link

Alignment Newsletter #13: 07/02/18

Rohin Shah2 Jul 2018 16:10 UTC

70 points

12 comments8 min readLW link

(mailchi.mp)

How can I reduce existential risk from AI?

lukeprog13 Nov 2012 21:56 UTC

63 points

92 comments8 min readLW link

Reframing the Problem of AI Progress

Wei Dai12 Apr 2012 19:31 UTC

32 points

47 comments1 min readLW link

HIRING: Inform and shape a new project on AI safety at Partnership on AI

madhu_lika7 Dec 2021 19:37 UTC

1 point

0 comments1 min readLW link

Modeling Failure Modes of High-Level Machine Intelligence

Ben Cottier, Daniel_Eth and Sammy Martin

6 Dec 2021 13:54 UTC

54 points

1 comment12 min readLW link

[LINK] NYT Article about Existential Risk from AI

[deleted]28 Jan 2013 10:37 UTC

38 points

23 comments1 min readLW link

The genie knows, but doesn’t care

Rob Bensinger6 Sep 2013 6:42 UTC

120 points

495 comments8 min readLW link

AI Safety Unconference NeurIPS 2022

Orpheus7 Nov 2022 15:39 UTC

25 points

0 comments1 min readLW link

(aisafetyevents.org)

4 Key Assumptions in AI Safety

Prometheus7 Nov 2022 10:50 UTC

20 points

5 comments7 min readLW link

Muehlhauser-Goertzel Dialogue, Part 1

lukeprog16 Mar 2012 17:12 UTC

42 points

161 comments33 min readLW link

How to measure FLOP/s for Neural Networks empirically?

Marius Hobbhahn29 Nov 2021 15:18 UTC

16 points

5 comments7 min readLW link

HIRING: Inform and shape a new project on AI safety at Partnership on AI

Madhulika Srikumar24 Nov 2021 8:27 UTC

6 points

0 comments1 min readLW link

Q&A with Stan Franklin on risks from AI

XiXiDu11 Jun 2011 15:22 UTC

36 points

10 comments2 min readLW link

AI Summer Fellows Program

colm21 Mar 2018 15:32 UTC

21 points

0 comments1 min readLW link

Responses to Catastrophic AGI Risk: A Survey

lukeprog8 Jul 2013 14:33 UTC

17 points

8 comments1 min readLW link

AI Tracker: monitoring current and near-future risks from superscale models

Edouard Harris and Jeremie Harris

23 Nov 2021 19:16 UTC

64 points

13 comments3 min readLW link

(aitracker.org)

Breaking Oracles: superrationality and acausal trade

Stuart_Armstrong25 Nov 2019 10:40 UTC

25 points

15 comments1 min readLW link

AI as a Civilizational Risk Part 6/6: What can be done

PashaKamyshev3 Nov 2022 19:48 UTC

2 points

4 comments4 min readLW link

Super intelligent AIs that don’t require alignment

Yair Halberstadt16 Nov 2021 19:55 UTC

10 points

2 comments6 min readLW link

Let’s talk about “Convergent Rationality”

David Scott Krueger (formerly: capybaralet)12 Jun 2019 21:53 UTC

41 points

33 comments6 min readLW link

AI Safety Research Camp—Project Proposal

David_Kristoffersson2 Feb 2018 4:25 UTC

29 points

11 comments8 min readLW link

Two Stupid AI Alignment Ideas

aphyer16 Nov 2021 16:13 UTC

24 points

3 comments4 min readLW link

Worldview iPeople—Future Fund’s AI Worldview Prize

Toni MUENDEL28 Oct 2022 1:53 UTC

−22 points

4 comments9 min readLW link

Will the world’s elites navigate the creation of AI just fine?

lukeprog31 May 2013 18:49 UTC

36 points

266 comments2 min readLW link

POWERplay: An open-source toolchain to study AI power-seeking

Edouard Harris24 Oct 2022 20:03 UTC

26 points

0 comments1 min readLW link

(github.com)

What would we do if alignment were futile?

Grant Demaree14 Nov 2021 8:09 UTC

75 points

39 comments3 min readLW link

Algo trading is a central example of AI risk

Vanessa Kosoy28 Jul 2018 20:31 UTC

27 points

5 comments1 min readLW link

Course recommendations for Friendliness researchers

Louie9 Jan 2013 14:33 UTC

96 points

112 comments10 min readLW link

Response to Katja Grace’s AI x-risk counterarguments

Erik Jenner and Johannes Treutlein

19 Oct 2022 1:17 UTC

76 points

18 comments15 min readLW link

Q&A with experts on risks from AI #1

XiXiDu8 Jan 2012 11:46 UTC

45 points

67 comments9 min readLW link

Provably Honest—A First Step

Srijanak De5 Nov 2022 19:18 UTC

10 points

2 comments8 min readLW link

Hardcode the AGI to need our approval indefinitely?

MichaelStJules11 Nov 2021 7:04 UTC

2 points

2 comments1 min readLW link

Evaluating the feasibility of SI’s plan

JoshuaFox10 Jan 2013 8:17 UTC

39 points

187 comments4 min readLW link

What are red flags for Neural Network suffering?

Marius Hobbhahn8 Nov 2021 12:51 UTC

29 points

15 comments12 min readLW link

What I would like the SIAI to publish

XiXiDu1 Nov 2010 14:07 UTC

36 points

225 comments3 min readLW link

What is the most evil AI that we could build, today?

ThomasJ1 Nov 2021 19:58 UTC

−2 points

14 comments1 min readLW link

Misalignment-by-default in multi-agent systems

Edouard Harris and simonsdsuo

13 Oct 2022 15:38 UTC

19 points

8 comments20 min readLW link

(www.gladstone.ai)

Truthful and honest AI

abergal, Nick_Beckstead and Owain_Evans

29 Oct 2021 7:28 UTC

42 points

1 comment13 min readLW link

The Evil AI Overlord List

Stuart_Armstrong20 Nov 2012 17:02 UTC

44 points

80 comments1 min readLW link

Opportunities for individual donors in AI safety

Alex Flint31 Mar 2018 18:37 UTC

30 points

3 comments11 min readLW link

A rant against robots

Lê Nguyên Hoang14 Jan 2020 22:03 UTC

65 points

7 comments5 min readLW link

Transcription of Eliezer’s January 2010 video Q&A

curiousepic14 Nov 2011 17:02 UTC

112 points

9 comments56 min readLW link

AMA on Truthful AI: Owen Cotton-Barratt, Owain Evans & co-authors

Owain_Evans22 Oct 2021 16:23 UTC

31 points

15 comments1 min readLW link

Possible miracles

Akash and Thomas Larsen

9 Oct 2022 18:17 UTC

62 points

33 comments8 min readLW link

1hr talk: Intro to AGI safety

Steven Byrnes18 Jun 2019 21:41 UTC

36 points

4 comments24 min readLW link

Truthful AI: Developing and governing AI that does not lie

Owain_Evans, owencb and Lukas Finnveden

18 Oct 2021 18:37 UTC

81 points

9 comments10 min readLW link

Allegory On AI Risk, Game Theory, and Mithril

James_Miller13 Feb 2017 20:41 UTC

45 points

57 comments3 min readLW link

Three AI Safety Related Ideas

Wei Dai13 Dec 2018 21:32 UTC

68 points

38 comments2 min readLW link

[Linkpost] “Blueprint for an AI Bill of Rights”—Office of Science and Technology Policy, USA (2022)

Fer32dwt34r3dfsz5 Oct 2022 16:42 UTC

9 points

4 comments2 min readLW link

(www.whitehouse.gov)

Boolean Primitives for Coupled Optimizers

Paul Bricman7 Oct 2022 18:02 UTC

9 points

0 comments8 min readLW link

The Dark Side of Cognition Hypothesis

Cameron Berg3 Oct 2021 20:10 UTC

19 points

1 comment16 min readLW link

[Question] Any further work on AI Safety Success Stories?

Krieger2 Oct 2022 9:53 UTC

8 points

6 comments1 min readLW link

Distribution Shifts and The Importance of AI Safety

Leon Lang29 Sep 2022 22:38 UTC

17 points

2 comments12 min readLW link

A brief review of the reasons multi-objective RL could be important in AI Safety Research

Ben Smith29 Sep 2021 17:09 UTC

31 points

7 comments10 min readLW link

A toy model of the treacherous turn

Stuart_Armstrong8 Jan 2016 12:58 UTC

42 points

13 comments6 min readLW link

AI Alignment Prize: Round 2 due March 31, 2018

Zvi12 Mar 2018 12:10 UTC

28 points

2 comments3 min readLW link

(thezvi.wordpress.com)

Oren’s Field Guide of Bad AGI Outcomes

Oren Montano26 Sep 2022 4:06 UTC

0 points

0 comments1 min readLW link

[Question] Why Do AI researchers Rate the Probability of Doom So Low?

Aorou24 Sep 2022 2:33 UTC

7 points

6 comments3 min readLW link

AI takeoff story: a continuation of progress by other means

Edouard Harris27 Sep 2021 15:55 UTC

76 points

13 comments10 min readLW link

AI Risk Intro 2: Solving The Problem

CallumMcDougall and LRudL

22 Sep 2022 13:55 UTC

22 points

0 comments27 min readLW link

Here Be AGI Dragons

Oren Montano21 Sep 2022 22:28 UTC

−1 points

3 comments5 min readLW link

On unfixably unsafe AGI architectures

Steven Byrnes19 Feb 2020 21:16 UTC

33 points

8 comments5 min readLW link

Investigating AI Takeover Scenarios

Sammy Martin17 Sep 2021 18:47 UTC

27 points

1 comment27 min readLW link

How truthful is GPT-3? A benchmark for language models

Owain_Evans16 Sep 2021 10:09 UTC

56 points

24 comments6 min readLW link

Conceptual issues in AI safety: the paradigmatic gap

vedevazz24 Jun 2018 15:09 UTC

33 points

0 comments1 min readLW link

(www.foldl.me)

The alignment problem in different capability regimes

Buck9 Sep 2021 19:46 UTC

88 points

12 comments5 min readLW link

Responding to ‘Beyond Hyperanthropomorphism’

ukc1001414 Sep 2022 20:37 UTC

8 points

0 comments16 min readLW link

Distinguishing AI takeover scenarios

Sam Clarke and Sammy Martin

8 Sep 2021 16:19 UTC

72 points

11 comments14 min readLW link

Dreams of Friendliness

Eliezer Yudkowsky31 Aug 2008 1:20 UTC

26 points

81 comments9 min readLW link

Beware of black boxes in AI alignment research

cousin_it18 Jan 2018 15:07 UTC

39 points

10 comments1 min readLW link

I Vouch For MIRI

Zvi17 Dec 2017 17:50 UTC

37 points

9 comments5 min readLW link

(thezvi.wordpress.com)

Precise P(doom) isn’t very important for prioritization or strategy

harsimony14 Sep 2022 17:19 UTC

14 points

6 comments1 min readLW link

Emily Brontë on: Psychology Required for Serious™ AGI Safety Research

robertzk14 Sep 2022 14:47 UTC

2 points

0 comments1 min readLW link

Capability and Agency as Cornerstones of AI risk — My current model

wilm15 Sep 2022 8:25 UTC

10 points

4 comments12 min readLW link

Understanding Conjecture: Notes from Connor Leahy interview

Akash15 Sep 2022 18:37 UTC

105 points

23 comments15 min readLW link

[Question] Would a Misaligned SSI Really Kill Us All?

DragonGod14 Sep 2022 12:15 UTC

6 points

7 comments6 min readLW link

Risk aversion and GPT-3

hatta_afiq13 Sep 2022 20:50 UTC

1 point

0 comments1 min readLW link

[Question] Updates on FLI’s Value Aligment Map?

Fer32dwt34r3dfsz17 Sep 2022 22:27 UTC

17 points

4 comments1 min readLW link

Summaries: Alignment Fundamentals Curriculum

Leon Lang18 Sep 2022 13:08 UTC

44 points

3 comments1 min readLW link

(docs.google.com)

Leveraging Legal Informatics to Align AI

John Nay18 Sep 2022 20:39 UTC

11 points

0 comments3 min readLW link

(forum.effectivealtruism.org)

Representational Tethers: Tying AI Latents To Human Ones

Paul Bricman16 Sep 2022 14:45 UTC

30 points

0 comments16 min readLW link

[Linkpost] A survey on over 300 works about interpretability in deep networks

scasper12 Sep 2022 19:07 UTC

97 points

7 comments2 min readLW link

(arxiv.org)

AI Safety field-building projects I’d like to see

Akash11 Sep 2022 23:43 UTC

44 points

7 comments6 min readLW link

Interlude: But Who Optimizes The Optimizer?

Paul Bricman23 Sep 2022 15:30 UTC

15 points

0 comments10 min readLW link

The Governance Problem and the “Pretty Good” X-Risk

Zach Stein-Perlman29 Aug 2021 18:00 UTC

5 points

2 comments11 min readLW link

On Generality

Oren Montano26 Sep 2022 4:06 UTC

2 points

0 comments5 min readLW link

Qualitative Strategies of Friendliness

Eliezer Yudkowsky30 Aug 2008 2:12 UTC

30 points

56 comments12 min readLW link

(Structural) Stability of Coupled Optimizers

Paul Bricman30 Sep 2022 11:28 UTC

25 points

0 comments10 min readLW link

AI Risk Intro 1: Advanced AI Might Be Very Bad

CallumMcDougall and LRudL

11 Sep 2022 10:57 UTC

46 points

13 comments30 min readLW link

Eli’s review of “Is power-seeking AI an existential risk?”

elifland30 Sep 2022 12:21 UTC

67 points

0 comments3 min readLW link

(docs.google.com)

Ideological Inference Engines: Making Deontology Differentiable*

Paul Bricman12 Sep 2022 12:00 UTC

6 points

0 comments14 min readLW link

Announcing the AI Safety Nudge Competition to Help Beat Procrastination

Marc Carauleanu1 Oct 2022 1:49 UTC

10 points

0 comments1 min readLW link

Oversight Leagues: The Training Game as a Feature

Paul Bricman9 Sep 2022 10:08 UTC

20 points

6 comments10 min readLW link

Generative, Episodic Objectives for Safe AI

Michael Glass5 Oct 2022 23:18 UTC

11 points

3 comments8 min readLW link

Could you have stopped Chernobyl?

Carlos Ramirez27 Aug 2021 1:48 UTC

29 points

17 comments8 min readLW link

Book review: Architects of Intelligence by Martin Ford (2018)

Ofer11 Aug 2020 17:30 UTC

15 points

0 comments2 min readLW link

What does it mean for an AGI to be ‘safe’?

So8res7 Oct 2022 4:13 UTC

72 points

29 comments3 min readLW link

It’s (not) how you use it

Eleni Angelou7 Sep 2022 17:15 UTC

8 points

1 comment2 min readLW link

Community Building for Graduate Students: A Targeted Approach

Neil Crawford6 Sep 2022 17:17 UTC

6 points

0 comments4 min readLW link

Instrumental convergence in single-agent systems

Edouard Harris and simonsdsuo

12 Oct 2022 12:24 UTC

31 points

4 comments8 min readLW link

(www.gladstone.ai)

Cataloguing Priors in Theory and Practice

Paul Bricman13 Oct 2022 12:36 UTC

13 points

8 comments7 min readLW link

A Game About AI Alignment (& Meta-Ethics): What Are the Must Haves?

JonathanErhardt5 Sep 2022 7:55 UTC

18 points

13 comments2 min readLW link

Instrumental convergence: scale and physical interactions

Edouard Harris and simonsdsuo

14 Oct 2022 15:50 UTC

15 points

0 comments17 min readLW link

(www.gladstone.ai)

Power-Seeking AI and Existential Risk

Antonio Franca11 Oct 2022 22:50 UTC

6 points

0 comments9 min readLW link

A gentle apocalypse

pchvykov16 Aug 2021 5:03 UTC

3 points

5 comments3 min readLW link

Niceness is unnatural

So8res13 Oct 2022 1:30 UTC

121 points

19 comments8 min readLW link

Greed Is the Root of This Evil

Thane Ruthenis13 Oct 2022 20:40 UTC

18 points

7 comments8 min readLW link

2019 AI Alignment Literature Review and Charity Comparison

Larks19 Dec 2019 3:00 UTC

130 points

18 comments62 min readLW link

Should ethicists be inside or outside a profession?

Eliezer Yudkowsky12 Dec 2018 1:40 UTC

88 points

7 comments9 min readLW link

Sticky goals: a concrete experiment for understanding deceptive alignment

evhub2 Sep 2022 21:57 UTC

39 points

13 comments3 min readLW link

[Question] How easy is it to supervise processes vs outcomes?

Noosphere8918 Oct 2022 17:48 UTC

3 points

0 comments1 min readLW link

Agency engineering: is AI-alignment “to human intent” enough?

catubc2 Sep 2022 18:14 UTC

9 points

10 comments6 min readLW link

Alignment is hard. Communicating that, might be harder

Eleni Angelou1 Sep 2022 16:57 UTC

7 points

8 comments3 min readLW link

Worlds Where Iterative Design Fails

johnswentworth30 Aug 2022 20:48 UTC

181 points

27 comments10 min readLW link

Three scenarios of pseudo-alignment

Eleni Angelou3 Sep 2022 12:47 UTC

9 points

0 comments3 min readLW link

Are Generative World Models a Mesa-Optimization Risk?

Thane Ruthenis29 Aug 2022 18:37 UTC

13 points

2 comments3 min readLW link

[Question] What would you expect a massive multimodal online federated learner to be capable of?

Aryeh Englander27 Aug 2022 17:31 UTC

13 points

4 comments1 min readLW link

AI Researchers On AI Risk

Scott Alexander22 May 2015 11:16 UTC

19 points

0 comments16 min readLW link

AI as a Civilizational Risk Part 2/6: Behavioral Modification

PashaKamyshev30 Oct 2022 16:57 UTC

9 points

0 comments10 min readLW link

AI as a Civilizational Risk Part 3/6: Anti-economy and Signal Pollution

PashaKamyshev31 Oct 2022 17:03 UTC

7 points

4 comments14 min readLW link

AI as a Civilizational Risk Part 4/6: Bioweapons and Philosophy of Modification

PashaKamyshev1 Nov 2022 20:50 UTC

7 points

1 comment8 min readLW link

AI as a Civilizational Risk Part 5/6: Relationship between C-risk and X-risk

PashaKamyshev3 Nov 2022 2:19 UTC

2 points

0 comments7 min readLW link

Mauhn Releases AI Safety Documentation

Berg Severens3 Jul 2021 21:23 UTC

4 points

0 comments1 min readLW link

Am I secretly excited for AI getting weird?

porby29 Oct 2022 22:16 UTC

112 points

4 comments4 min readLW link

My (naive) take on Risks from Learned Optimization

artkpv31 Oct 2022 10:59 UTC

7 points

0 comments5 min readLW link

Clarifying AI X-risk

zac_kenton, Rohin Shah, David Lindner, Vikrant Varma, Vika, Mary Phuong, Ramana Kumar and Elliot Catt

1 Nov 2022 11:03 UTC

125 points

23 comments4 min readLW link

Threat Model Literature Review

zac_kenton, Rohin Shah, David Lindner, Vikrant Varma, Vika, Mary Phuong, Ramana Kumar and Elliot Catt

1 Nov 2022 11:03 UTC

73 points

4 comments25 min readLW link

a casual intro to AI doom and alignment

Tamsin Leake1 Nov 2022 16:38 UTC

18 points

0 comments4 min readLW link

(carado.moe)

[Question] What are some claims or opinions about multi-multi delegation you’ve seen in the memeplex that you think deserve scrutiny?

Quinn27 Jun 2021 17:44 UTC

17 points

6 comments2 min readLW link

Help Understanding Preferences And Evil

Netcentrica27 Aug 2022 3:42 UTC

6 points

7 comments2 min readLW link

Why do we post our AI safety plans on the Internet?

Peter S. Park3 Nov 2022 16:02 UTC

4 points

4 comments11 min readLW link

My summary of “Pragmatic AI Safety”

Eleni Angelou5 Nov 2022 12:54 UTC

3 points

0 comments5 min readLW link

Annual AGI Benchmarking Event

Lawrence Phillips27 Aug 2022 0:06 UTC

24 points

3 comments2 min readLW link

(www.metaculus.com)

Loss of control of AI is not a likely source of AI x-risk

squek7 Nov 2022 18:44 UTC

−6 points

0 comments5 min readLW link

Taking the parameters which seem to matter and rotating them until they don’t

Garrett Baker26 Aug 2022 18:26 UTC

120 points

48 comments1 min readLW link

Value Formation: An Overarching Model

Thane Ruthenis15 Nov 2022 17:16 UTC

26 points

20 comments34 min readLW link

Is AI Gain-of-Function research a thing?

MadHatter12 Nov 2022 2:33 UTC

9 points

2 comments2 min readLW link

AI Risk in Terms of Unstable Nuclear Software

Thane Ruthenis26 Aug 2022 18:49 UTC

30 points

1 comment6 min readLW link

The limited upside of interpretability

Peter S. Park15 Nov 2022 18:46 UTC

13 points

11 comments1 min readLW link

It Looks Like You’re Trying To Take Over The Narrative

George3d624 Aug 2022 13:36 UTC

3 points

20 comments9 min readLW link

(www.epistem.ink)

The Alignment Problem Needs More Positive Fiction

Netcentrica21 Aug 2022 22:01 UTC

5 points

2 comments5 min readLW link

Survey on AI existential risk scenarios

Sam Clarke, apc and Jonas Schuett

8 Jun 2021 17:12 UTC

63 points

11 comments7 min readLW link

Conjecture: a retrospective after 8 months of work

Connor Leahy, Sid Black, Gabriel Alfour and Chris Scammell

23 Nov 2022 17:10 UTC

185 points

9 comments8 min readLW link

Conjecture Second Hiring Round

Connor Leahy, Sid Black, Gabriel Alfour and Chris Scammell

23 Nov 2022 17:11 UTC

92 points

0 comments1 min readLW link

Corrigibility Via Thought-Process Deference

Thane Ruthenis24 Nov 2022 17:06 UTC

17 points

5 comments9 min readLW link

‘Dumb’ AI observes and manipulates controllers

Stuart_Armstrong13 Jan 2015 13:35 UTC

52 points

19 comments2 min readLW link

2017 AI Safety Literature Review and Charity Comparison

Larks24 Dec 2017 18:52 UTC

41 points

5 comments23 min readLW link

And the AI would have got away with it too, if...

Stuart_Armstrong22 May 2019 21:35 UTC

75 points

7 comments1 min readLW link

Conversation with Paul Christiano

abergal11 Sep 2019 23:20 UTC

44 points

6 comments30 min readLW link

(aiimpacts.org)

Steering systems

Max H4 Apr 2023 0:56 UTC

43 points

1 comment15 min readLW link

ICA Simulacra

Ozyrus5 Apr 2023 6:41 UTC

26 points

2 comments7 min readLW link

Against sacrificing AI transparency for generality gains

Ape in the coat7 May 2023 6:52 UTC

3 points

0 comments2 min readLW link

Superintelligence will outsmart us or it isn’t superintelligence

Neil 3 Apr 2023 15:01 UTC

−7 points

4 comments1 min readLW link

Do we have a plan for the “first critical try” problem?

Christopher King3 Apr 2023 16:27 UTC

−3 points

14 comments1 min readLW link

Towards empathy in RL agents and beyond: Insights from cognitive science for AI Alignment

Marc Carauleanu3 Apr 2023 19:59 UTC

14 points

6 comments1 min readLW link

(clipchamp.com)

Strategies to Prevent AI Annihilation

lastchanceformankind4 Apr 2023 8:59 UTC

−2 points

0 comments4 min readLW link

[Question] Isn’t safe AGI impossible?

lefoenix5 Apr 2023 4:01 UTC

1 point

0 comments1 min readLW link

AGI deployment as an act of aggression

dr_s5 Apr 2023 6:39 UTC

27 points

29 comments13 min readLW link

[Question] Daisy-chaining epsilon-step verifiers

Decaeneus6 Apr 2023 2:07 UTC

2 points

0 comments1 min readLW link

One Does Not Simply Replace the Humans

JerkyTreats6 Apr 2023 20:56 UTC

9 points

3 comments4 min readLW link

(www.lesswrong.com)

OpenAI: Our approach to AI safety

g-w15 Apr 2023 20:26 UTC

1 point

1 comment1 min readLW link

(openai.com)

Williams-Beuren Syndrome: Frendly Mutations

Takk5 Apr 2023 20:59 UTC

−1 points

1 comment1 min readLW link

Yoshua Bengio: “Slowing down development of AI systems passing the Turing test”

Roman Leventov6 Apr 2023 3:31 UTC

49 points

2 comments5 min readLW link

(yoshuabengio.org)

Risks from GPT-4 Byproduct of Recursively Optimizing AIs

ben hayum7 Apr 2023 0:02 UTC

72 points

9 comments10 min readLW link

(forum.effectivealtruism.org)

A decade of lurking, a month of posting

Max H9 Apr 2023 0:21 UTC

70 points

4 comments5 min readLW link

Alignment of AutoGPT agents

Ozyrus12 Apr 2023 12:54 UTC

14 points

1 comment4 min readLW link

Why I’m not worried about imminent doom

Ariel Kwiatkowski10 Apr 2023 15:31 UTC

7 points

1 comment4 min readLW link

Measuring artificial intelligence on human benchmarks is naive

Anomalous11 Apr 2023 11:34 UTC

11 points

4 comments1 min readLW link

(forum.effectivealtruism.org)

In favor of accelerating problems you’re trying to solve

Christopher King11 Apr 2023 18:15 UTC

2 points

2 comments4 min readLW link

AI Risk US Presidential Candidate

Simon Berens11 Apr 2023 19:31 UTC

5 points

3 comments1 min readLW link

Open-source LLMs may prove Bostrom’s vulnerable world hypothesis

Roope Ahvenharju15 Apr 2023 19:16 UTC

1 point

1 comment1 min readLW link

Artificial Intelligence as exit strategy from the age of acute existential risk

Arturo Macias12 Apr 2023 14:48 UTC

−7 points

15 comments7 min readLW link

AGI goal space is big, but narrowing might not be as hard as it seems.

Jacy Reese Anthis12 Apr 2023 19:03 UTC

15 points

0 comments3 min readLW link

Polluting the agentic commons

hamandcheese13 Apr 2023 17:42 UTC

7 points

4 comments2 min readLW link

(www.secondbest.ca)

The Virus—Short Story

Michael Soareverix13 Apr 2023 18:18 UTC

4 points

0 comments4 min readLW link

On the possibility of impossibility of AGI Long-Term Safety

Roman Yen13 May 2023 18:38 UTC

4 points

1 comment9 min readLW link

Speculation on mapping the moral landscape for future Ai Alignment

Sven Heinz (Welwordion)16 Apr 2023 13:43 UTC

1 point

0 comments1 min readLW link

On urgency, priority and collective reaction to AI-Risks: Part I

Denreik16 Apr 2023 19:14 UTC

−10 points

15 comments5 min readLW link

AGI Clinics: A Safe Haven for Humanity’s First Encounters with Superintelligence

portr.17 Apr 2023 1:52 UTC

−5 points

1 comment1 min readLW link

Defining Boundaries on Outcomes

Takk7 Jun 2023 17:41 UTC

1 point

0 comments1 min readLW link

No, really, it predicts next tokens.

simon18 Apr 2023 3:47 UTC

57 points

37 comments3 min readLW link

Prediction: any uncontrollable AI will turn earth into a giant computer

Karl von Wendt17 Apr 2023 12:30 UTC

9 points

8 comments3 min readLW link

What is your timelines for ADI (artificial disempowering intelligence)?

Christopher King17 Apr 2023 17:01 UTC

3 points

3 comments2 min readLW link

Green goo is plausible

anithite18 Apr 2023 0:04 UTC

56 points

29 comments4 min readLW link

World and Mind in Artificial Intelligence: arguments against the AI pause

Arturo Macias18 Apr 2023 14:40 UTC

1 point

0 comments1 min readLW link

(forum.effectivealtruism.org)

AI Safety Newsletter #2: ChaosGPT, Natural Selection, and AI Safety in the Media

ozhang, Dan H, Akash and aogara

18 Apr 2023 18:44 UTC

30 points

0 comments4 min readLW link

(newsletter.safe.ai)

I Believe I Know Why AI Models Hallucinate

Richard Aragon19 Apr 2023 21:07 UTC

−10 points

6 comments7 min readLW link

(turingssolutions.com)

[Crosspost] Organizing a debate with experts and MPs to raise AI xrisk awareness: a possible blueprint

otto.barten19 Apr 2023 11:45 UTC

8 points

0 comments4 min readLW link

(forum.effectivealtruism.org)

How to express this system for ethically aligned AGI as a Mathematical formula?

Oliver Siegel19 Apr 2023 20:13 UTC

−1 points

0 comments1 min readLW link

[Question] Is there any literature on using socialization for AI alignment?

Nathan112319 Apr 2023 22:16 UTC

10 points

9 comments2 min readLW link

How does AI Risk Affect the Simulation Hypothesis?

amelia20 Apr 2023 3:16 UTC

6 points

9 comments2 min readLW link

Stability AI releases StableLM, an open-source ChatGPT counterpart

Ozyrus20 Apr 2023 6:04 UTC

11 points

3 comments1 min readLW link

(github.com)

Ideas for studies on AGI risk

dr_s20 Apr 2023 18:17 UTC

5 points

1 comment11 min readLW link

Proposal: Using Monte Carlo tree search instead of RLHF for alignment research

Christopher King20 Apr 2023 19:57 UTC

2 points

7 comments3 min readLW link

Notes on “the hot mess theory of AI misalignment”

JakubK21 Apr 2023 10:07 UTC

13 points

0 comments5 min readLW link

(sohl-dickstein.github.io)

The Security Mindset, S-Risk and Publishing Prosaic Alignment Research

lukemarks22 Apr 2023 14:36 UTC

39 points

7 comments6 min readLW link

A great talk for AI noobs (according to an AI noob)

dov23 Apr 2023 5:34 UTC

10 points

1 comment1 min readLW link

(forum.effectivealtruism.org)

Paths to failure

Karl von Wendt and mespa

25 Apr 2023 8:03 UTC

29 points

1 comment8 min readLW link

A concise sum-up of the basic argument for AI doom

Mergimio H. Doefevmil24 Apr 2023 17:37 UTC

11 points

6 comments2 min readLW link

A response to Conjecture’s CoEm proposal

Kristian Freed24 Apr 2023 17:23 UTC

7 points

0 comments4 min readLW link

A Proposal for AI Alignment: Using Directly Opposing Models

Arne B27 Apr 2023 18:05 UTC

0 points

5 comments3 min readLW link

Making Nanobots isn’t a one-shot process, even for an artificial superintelligance

dankrad25 Apr 2023 0:39 UTC

20 points

13 comments6 min readLW link

My Assessment of the Chinese AI Safety Community

Lao Mein25 Apr 2023 4:21 UTC

239 points

92 comments3 min readLW link

Briefly how I’ve updated since ChatGPT

rime25 Apr 2023 14:47 UTC

48 points

2 comments2 min readLW link

Freedom Is All We Need

Leo Glisic27 Apr 2023 0:09 UTC

−1 points

8 comments10 min readLW link

What’s in a Name? Are you really an “AI Pessimist”?

amelia23 Oct 2023 22:41 UTC

−2 points

4 comments3 min readLW link

Halloween Problem

Saint Blasphemer24 Oct 2023 16:46 UTC

−10 points

1 comment1 min readLW link

Announcing #AISummitTalks featuring Professor Stuart Russell and many others

otto.barten24 Oct 2023 10:11 UTC

17 points

1 comment1 min readLW link

[Question] What if AGI had its own universe to maybe wreck?

mseale26 Oct 2023 17:49 UTC

−1 points

2 comments1 min readLW link

Responsible Scaling Policies Are Risk Management Done Wrong

simeon_c25 Oct 2023 23:46 UTC

112 points

33 comments22 min readLW link

(www.navigatingrisks.ai)

[Thought Experiment] Tomorrow’s Echo—The future of synthetic companionship.

Vimal Naran26 Oct 2023 17:54 UTC

−7 points

2 comments2 min readLW link

[untitled post]

NeuralSystem_e5e127 Apr 2023 17:37 UTC

3 points

0 comments1 min readLW link

Response to “Coordinated pausing: An evaluation-based coordination scheme for frontier AI developers”

Matthew Wearden30 Oct 2023 17:27 UTC

5 points

2 comments6 min readLW link

(matthewwearden.co.uk)

Charbel-Raphaël and Lucius discuss Interpretability

Mateusz Bagiński, Charbel-Raphaël and Lucius Bushnaq

30 Oct 2023 5:50 UTC

102 points

7 comments21 min readLW link

An International Manhattan Project for Artificial Intelligence

Glenn Clayton27 Apr 2023 17:34 UTC

−9 points

2 comments5 min readLW link

Focus on existential risk is a distraction from the real issues. A false fallacy

Nik Samoylov30 Oct 2023 23:42 UTC

−19 points

11 comments2 min readLW link

Saying the quiet part out loud: trading off x-risk for personal immortality

disturbance2 Nov 2023 17:43 UTC

78 points

90 comments5 min readLW link

The 6D effect: When companies take risks, one email can be very powerful.

scasper4 Nov 2023 20:08 UTC

243 points

40 comments3 min readLW link

AI as Super-Demagogue

RationalDino5 Nov 2023 21:21 UTC

−2 points

9 comments9 min readLW link

Symbiotic self-alignment of AIs.

Spiritus Dei7 Nov 2023 17:18 UTC

1 point

0 comments3 min readLW link

Scalable And Transferable Black-Box Jailbreaks For Language Models Via Persona Modulation

Soroush Pour, rusheb, Quentin FEUILLADE--MONTIXI, Arush and scasper

7 Nov 2023 17:59 UTC

35 points

2 comments2 min readLW link

(arxiv.org)

The Social Alignment Problem

irving28 Apr 2023 14:16 UTC

96 points

13 comments8 min readLW link

Do you want a first-principled preparedness guide to prepare yourself and loved ones for potential catastrophes?

Ulrik Horn14 Nov 2023 12:13 UTC

9 points

5 comments15 min readLW link

[Question] Realistic near-future scenarios of AI doom understandable for non-techy people?

RomanS28 Apr 2023 14:45 UTC

4 points

4 comments1 min readLW link

[Question] AI Safety orgs- what’s your biggest bottleneck right now?

Kabir Kumar16 Nov 2023 2:02 UTC

1 point

0 comments1 min readLW link

We Should Talk About This More. Epistemic World Collapse as Imminent Safety Risk of Generative AI.

Joerg Weiss16 Nov 2023 18:46 UTC

11 points

2 comments29 min readLW link

On excluding dangerous information from training

ShayBenMoshe17 Nov 2023 11:14 UTC

23 points

4 comments3 min readLW link

Killswitch

Junio18 Nov 2023 22:53 UTC

1 point

0 comments3 min readLW link

Ilya: The AI scientist shaping the world

David Varga20 Nov 2023 13:09 UTC

11 points

0 comments4 min readLW link

A Guide to Forecasting AI Science Capabilities

Eleni Angelou29 Apr 2023 23:24 UTC

6 points

1 comment4 min readLW link

The two paragraph argument for AI risk

CronoDAS25 Nov 2023 2:01 UTC

17 points

6 comments1 min readLW link

AISC 2024 - Project Summaries

NickyP27 Nov 2023 22:32 UTC

39 points

3 comments18 min readLW link

Rethink Priorities: Seeking Expressions of Interest for Special Projects Next Year

kierangreig29 Nov 2023 13:59 UTC

4 points

0 comments5 min readLW link

Support me in a Week-Long Picketing Campaign Near OpenAI’s HQ: Seeking Support and Ideas from the LessWrong Community

Percy30 Apr 2023 17:48 UTC

−26 points

15 comments1 min readLW link

Thoughts on “AI is easy to control” by Pope & Belrose

Steven Byrnes1 Dec 2023 17:30 UTC

163 points

42 comments13 min readLW link

The benefits and risks of optimism (about AI safety)

Karl von Wendt3 Dec 2023 12:45 UTC

−5 points

4 comments5 min readLW link

A call for a quantitative report card for AI bioterrorism threat models

Juno4 Dec 2023 6:35 UTC

3 points

0 comments10 min readLW link

[Question] Accuracy of arguments that are seen as ridiculous and intuitively false but don’t have good counter-arguments

Christopher King29 Apr 2023 23:58 UTC

30 points

39 comments1 min readLW link

Call for submissions: Choice of Futures survey questions

c.trout30 Apr 2023 6:59 UTC

4 points

0 comments2 min readLW link

(airtable.com)

[Question] Does agency necessarily imply self-preservation instinct?

Mislav Jurić1 May 2023 16:06 UTC

5 points

8 comments1 min readLW link

Shah (DeepMind) and Leahy (Conjecture) Discuss Alignment Cruxes

OliviaJ, Rohin Shah, Connor Leahy and Andrea_Miotti

1 May 2023 16:47 UTC

93 points

10 comments30 min readLW link

AI Safety Newsletter #4: AI and Cybersecurity, Persuasive AIs, Weaponization, and Geoffrey Hinton talks AI risks

ozhang, Dan H, Akash and aogara

2 May 2023 18:41 UTC

32 points

0 comments5 min readLW link

(newsletter.safe.ai)

Averting Catastrophe: Decision Theory for COVID-19, Climate Change, and Potential Disasters of All Kinds

JakubK2 May 2023 22:50 UTC

10 points

0 comments1 min readLW link

Regulate or Compete? The China Factor in U.S. AI Policy (NAIR #2)

charles_m5 May 2023 17:43 UTC

2 points

1 comment7 min readLW link

(navigatingairisks.substack.com)

But What If We Actually Want To Maximize Paperclips?

snerx25 May 2023 7:13 UTC

−17 points

6 comments7 min readLW link

Formalizing the “AI x-risk is unlikely because it is ridiculous” argument

Christopher King3 May 2023 18:56 UTC

40 points

17 comments3 min readLW link

We don’t need AGI for an amazing future

Karl von Wendt4 May 2023 12:10 UTC

18 points

32 comments5 min readLW link

White House Announces “New Actions to Promote Responsible AI Innovation”

lberglund4 May 2023 12:15 UTC

54 points

18 comments3 min readLW link

(www.whitehouse.gov)

[Question] Why not use active SETI to prevent AI Doom?

RomanS5 May 2023 14:41 UTC

13 points

13 comments1 min readLW link

CHAT Diplomacy: LLMs and National Security

JohnBuridan5 May 2023 19:45 UTC

25 points

5 comments7 min readLW link

Is “red” for GPT-4 the same as “red” for you?

Yusuke Hayashi6 May 2023 17:55 UTC

9 points

6 comments2 min readLW link

Oh, Think of the Bananas

Jeffs1 Jun 2023 6:46 UTC

3 points

0 comments2 min readLW link

TED talk by Eliezer Yudkowsky: Unleashing the Power of Artificial Intelligence

bayesed7 May 2023 5:45 UTC

49 points

36 comments1 min readLW link

(www.youtube.com)

Annotated reply to Bengio’s “AI Scientists: Safe and Useful AI?”

Roman Leventov8 May 2023 21:26 UTC

18 points

2 comments7 min readLW link

(yoshuabengio.org)

H-JEPA might be technically alignable in a modified form

Roman Leventov8 May 2023 23:04 UTC

12 points

2 comments7 min readLW link

[Question] How much of a concern are open-source LLMs in the short, medium and long terms?

JavierCC10 May 2023 9:14 UTC

5 points

0 comments1 min readLW link

AGI-Automated Interpretability is Suicide

__RicG__10 May 2023 14:20 UTC

22 points

33 comments7 min readLW link

[Question] Is “brittle alignment” good enough?

the8thbit23 May 2023 17:35 UTC

9 points

5 comments3 min readLW link

[Question] AI interpretability could be harmful?

Roman Leventov10 May 2023 20:43 UTC

13 points

2 comments1 min readLW link

[Question] How should we think about the decision relevance of models estimating p(doom)?

Mo Putera11 May 2023 4:16 UTC

11 points

1 comment3 min readLW link

A more grounded idea of AI risk

Iknownothing11 May 2023 9:48 UTC

3 points

4 comments1 min readLW link

Separating the “control problem” from the “alignment problem”

Yi-Yang11 May 2023 9:41 UTC

10 points

0 comments4 min readLW link

Alignment, Goals, and The Gut-Head Gap: A Review of Ngo. et al.

Violet Hour11 May 2023 18:06 UTC

18 points

2 comments13 min readLW link

[Question] Term/Category for AI with Neutral Impact?

isomic11 May 2023 22:00 UTC

6 points

1 comment1 min readLW link

Un-unpluggability—can’t we just unplug it?

Oliver Sourbut15 May 2023 13:23 UTC

26 points

10 comments12 min readLW link

(www.oliversourbut.net)

Formulating the AI Doom Argument for Analytic Philosophers

JonathanErhardt12 May 2023 7:54 UTC

13 points

0 comments2 min readLW link

The way AGI wins could look very stupid

Christopher King12 May 2023 16:34 UTC

42 points

22 comments1 min readLW link

PCAST Working Group on Generative AI Invites Public Input

Christopher King13 May 2023 22:49 UTC

7 points

0 comments1 min readLW link

(terrytao.wordpress.com)

Coordination by common knowledge to prevent uncontrollable AI

Karl von Wendt14 May 2023 13:37 UTC

10 points

2 comments9 min readLW link

[Question] What projects and efforts are there to promote AI safety research?

Christopher King24 May 2023 0:33 UTC

4 points

0 comments1 min readLW link

AI Risk & Policy Forecasts from Metaculus & FLI’s AI Pathways Workshop

_will_16 May 2023 18:06 UTC

11 points

4 comments8 min readLW link

Tyler Cowen’s challenge to develop an ‘actual mathematical model’ for AI X-Risk

Joe Brenton16 May 2023 11:57 UTC

5 points

4 comments1 min readLW link

GPT as an “Intelligence Forklift.”

boazbarak19 May 2023 21:15 UTC

46 points

27 comments3 min readLW link

Proposal: we should start referring to the risk from unaligned AI as a type of accident risk

Christopher King16 May 2023 15:18 UTC

22 points

6 comments2 min readLW link

[untitled post]

[Error communicating with LW2 server]20 May 2023 3:08 UTC

1 point

0 comments1 min readLW link

Confusions and updates on STEM AI

Eleni Angelou19 May 2023 21:34 UTC

21 points

0 comments3 min readLW link

A&I (Rihanna ‘S&M’ parody lyrics)

nahoj21 May 2023 22:34 UTC

−3 points

0 comments2 min readLW link

We Shouldn’t Expect AI to Ever be Fully Rational

OneManyNone18 May 2023 17:09 UTC

19 points

31 comments6 min readLW link

[Crosspost] A recent write-up of the case for AI (existential) risk

Timsey18 May 2023 13:13 UTC

6 points

0 comments19 min readLW link

The Polarity Problem [Draft]

Dan H, cdkg and Simon Goldstein

23 May 2023 21:05 UTC

24 points

3 comments44 min readLW link

A flaw in the A.G.I. Ruin Argument

Cole Wyeth19 May 2023 19:40 UTC

0 points

6 comments3 min readLW link

(colewyeth.com)

 Yoshua Bengio: How Rogue AIs may Arise

harfe23 May 2023 18:28 UTC

87 points

12 comments18 min readLW link

(yoshuabengio.org)

A rejection of the Orthogonality Thesis

ArisC24 May 2023 16:37 UTC

−2 points

11 comments2 min readLW link

(medium.com)

Two ideas for alignment, perpetual mutual distrust and induction

APaleBlueDot25 May 2023 0:56 UTC

1 point

2 comments4 min readLW link

The Genie in the Bottle: An Introduction to AI Alignment and Risk

Snorkelfarsan25 May 2023 16:30 UTC

2 points

0 comments25 min readLW link

DeepMind: Model evaluation for extreme risks

Zach Stein-Perlman25 May 2023 3:00 UTC

94 points

11 comments1 min readLW link

(arxiv.org)

Aligning an H-JEPA agent via training on the outputs of an LLM-based “exemplary actor”

Roman Leventov29 May 2023 11:08 UTC

12 points

10 comments30 min readLW link

An LLM-based “exemplary actor”

Roman Leventov29 May 2023 11:12 UTC

16 points

0 comments12 min readLW link

[Question] Why is violence against AI labs a taboo?

ArisC26 May 2023 8:00 UTC

−21 points

63 comments1 min readLW link

[Question] What’s your viewpoint on the likelihood of GPT-5 being able to autonomously create, train, and implement an AI superior to GPT-5?

Super AGI26 May 2023 1:43 UTC

6 points

15 comments1 min readLW link

Inference from a Mathematical Description of an Existing Alignment Research: a proposal for an outer alignment research program

Christopher King2 Jun 2023 21:54 UTC

7 points

4 comments16 min readLW link

AI X-risk is a possible solution to the Fermi Paradox

magic9mushroom30 May 2023 17:42 UTC

10 points

10 comments2 min readLW link

Hands of gods

Anders L28 May 2023 15:15 UTC

1 point

0 comments9 min readLW link

(woodfromeden.substack.com)

Proposed Alignment Technique: OSNR (Output Sanitization via Noising and Reconstruction) for Safer Usage of Potentially Misaligned AGI

sudo29 May 2023 1:35 UTC

14 points

9 comments6 min readLW link

Without a trajectory change, the development of AGI is likely to go badly

Max H29 May 2023 23:42 UTC

16 points

2 comments13 min readLW link

On the Impossibility of Intelligent Paperclip Maximizers

Michael Simkin29 May 2023 16:55 UTC

−17 points

5 comments4 min readLW link

Winners-take-how-much?

YonatanK29 May 2023 21:56 UTC

1 point

2 comments3 min readLW link

An Analysis of the ‘Digital Gaia’ Proposal from a Safety Perspective

lukemarks31 May 2023 12:21 UTC

8 points

1 comment4 min readLW link

Limiting factors to predict AI take-off speed

Alfonso Pérez Escudero31 May 2023 23:19 UTC

1 point

0 comments6 min readLW link

A Double-Feature on The Extropians

Maxwell Tabarrok3 Jun 2023 18:27 UTC

58 points

4 comments1 min readLW link

Intrinsic vs. Extrinsic Alignment

Alfonso Pérez Escudero1 Jun 2023 1:06 UTC

1 point

1 comment3 min readLW link

How will they feed us

meijer19731 Jun 2023 8:49 UTC

4 points

3 comments5 min readLW link

Unpredictability and the Increasing Difficulty of AI Alignment for Increasingly Intelligent AI

Max_He-Ho31 May 2023 22:25 UTC

5 points

2 comments20 min readLW link

The unspoken but ridiculous assumption of AI doom: the hidden doom assumption

Christopher King1 Jun 2023 17:01 UTC

−9 points

1 comment3 min readLW link

Open Source LLMs Can Now Actively Lie

Josh Levy1 Jun 2023 22:03 UTC

4 points

0 comments3 min readLW link

Proposal: labs should precommit to pausing if an AI argues for itself to be improved

NickGabs2 Jun 2023 22:31 UTC

3 points

3 comments4 min readLW link

[FICTION] Prometheus Rising: The Emergence of an AI Consciousness

Super AGI10 Jun 2023 4:41 UTC

−13 points

0 comments9 min readLW link

Andrew Ng wants to have a conversation about extinction risk from AI

Leon Lang5 Jun 2023 22:29 UTC

32 points

2 comments1 min readLW link

(twitter.com)

The (local) unit of intelligence is FLOPs

boazbarak5 Jun 2023 18:23 UTC

40 points

7 comments5 min readLW link

Non-loss of control AGI-related catastrophes are out of control too

Yi-Yang, Mo Putera and zeshen

12 Jun 2023 12:01 UTC

0 points

3 comments24 min readLW link

A Playbook for AI Risk Reduction (focused on misaligned AI)

HoldenKarnofsky6 Jun 2023 18:05 UTC

89 points

41 comments14 min readLW link

Using Consensus Mechanisms as an approach to Alignment

Prometheus10 Jun 2023 23:38 UTC

9 points

2 comments6 min readLW link

Agentic Mess (A Failure Story)

Karl von Wendt, Sofia Bharadia, PeterDrotos, Artem Korotkov, mespa and mruwnik

6 Jun 2023 13:09 UTC

44 points

5 comments13 min readLW link

Manifold Predicted the AI Extinction Statement and CAIS Wanted it Deleted

David Chee12 Jun 2023 15:54 UTC

70 points

14 comments12 min readLW link

Current AI harms are also sci-fi

Christopher King8 Jun 2023 17:49 UTC

26 points

3 comments1 min readLW link

[FICTION] Unboxing Elysium: An AI’S Escape

Super AGI10 Jun 2023 4:41 UTC

−14 points

4 comments14 min readLW link

Why AI may not save the World

Alberto Zannoni9 Jun 2023 17:42 UTC

0 points

0 comments4 min readLW link

(a16z.com)

[Question] AI Rights: In your view, what would be required for an AGI to gain rights and protections from the various Governments of the World?

Super AGI9 Jun 2023 1:24 UTC

13 points

22 comments1 min readLW link

Introduction to Towards Causal Foundations of Safe AGI

tom4everitt, Lewis Hammond, Francis Rhys Ward, RyanCarey, James Fox, mattmacdermott and sbenthall

12 Jun 2023 17:55 UTC

67 points

6 comments4 min readLW link

Aligned Objectives Prize Competition

Prometheus15 Jun 2023 12:42 UTC

8 points

0 comments2 min readLW link

(app.impactmarkets.io)

Lightning Post: Things people in AI Safety should stop talking about

Prometheus20 Jun 2023 15:00 UTC

23 points

6 comments2 min readLW link

A Friendly Face (Another Failure Story)

Karl von Wendt, Sofia Bharadia, PeterDrotos, Artem Korotkov, mespa and mruwnik

20 Jun 2023 10:31 UTC

65 points

21 comments16 min readLW link

Exploring Last-Resort Measures for AI Alignment: Humanity’s Extinction Switch

0xPetra23 Jun 2023 17:01 UTC

7 points

0 comments2 min readLW link

I just watched don’t look up.

ATheCoder23 Jun 2023 21:22 UTC

0 points

5 comments2 min readLW link

AI-Plans.com—a contributable compendium

Iknownothing25 Jun 2023 14:40 UTC

39 points

7 comments4 min readLW link

(ai-plans.com)

Cheat sheet of AI X-risk

amaury lorin29 Jun 2023 4:28 UTC

18 points

1 comment7 min readLW link

Challenge proposal: smallest possible self-hardening backdoor for RLHF

Christopher King29 Jun 2023 16:56 UTC

7 points

0 comments2 min readLW link

Biosafety Regulations (BMBL) and their relevance for AI

Štěpán Los29 Jun 2023 19:22 UTC

4 points

0 comments4 min readLW link

Metaphors for AI, and why I don’t like them

boazbarak28 Jun 2023 22:47 UTC

32 points

18 comments12 min readLW link

AGI & War

Calecute29 Jun 2023 22:20 UTC

9 points

1 comment1 min readLW link

George Hotz on AI safety: ~”centralized power is bad”

Chipmonk30 Jun 2023 5:00 UTC

14 points

5 comments1 min readLW link

(www.youtube.com)

AI Incident Sharing—Best practices from other fields and a comprehensive list of existing platforms

Štěpán Los28 Jun 2023 17:21 UTC

20 points

0 comments4 min readLW link

AI Safety without Alignment: How humans can WIN against AI

vicchain29 Jun 2023 17:53 UTC

1 point

1 comment2 min readLW link

Quantitative cruxes in Alignment

Martín Soto2 Jul 2023 20:38 UTC

20 points

0 comments23 min readLW link

Sources of evidence in Alignment

Martín Soto2 Jul 2023 20:38 UTC

20 points

0 comments11 min readLW link

An AGI kill switch with defined security properties

Peterpiper5 Jul 2023 17:40 UTC

−5 points

6 comments1 min readLW link

How I Learned To Stop Worrying And Love The Shoggoth

Peter Merel12 Jul 2023 17:47 UTC

10 points

9 comments5 min readLW link

OpenAI Launches Superalignment Taskforce

Zvi11 Jul 2023 13:00 UTC

140 points

38 comments49 min readLW link

(thezvi.wordpress.com)

Do you feel that AGI Alignment could be achieved in a Type 0 civilization?

Super AGI6 Jul 2023 4:52 UTC

−2 points

1 comment1 min readLW link

An Overview of AI risks—the Flyer

Charbel-Raphaël, Jonathan Claybrough and tchauvin

17 Jul 2023 12:03 UTC

20 points

0 comments1 min readLW link

(docs.google.com)

ACI#4: Seed AI is the new Perpetual Motion Machine

Akira Pyinya8 Jul 2023 1:17 UTC

−7 points

0 comments6 min readLW link

Announcing AI Alignment workshop at the ALIFE 2023 conference

rorygreig8 Jul 2023 13:52 UTC

11 points

0 comments1 min readLW link

(humanvaluesandartificialagency.com)

“Reframing Superintelligence” + LLMs + 4 years

Eric Drexler10 Jul 2023 13:42 UTC

116 points

8 comments12 min readLW link

Gearing Up for Long Timelines in a Hard World

Dalcy14 Jul 2023 6:11 UTC

11 points

0 comments4 min readLW link

[Question] What criterion would you use to select companies likely to cause AI doom?

amaury lorin13 Jul 2023 20:31 UTC

8 points

4 comments1 min readLW link

Why was the AI Alignment community so unprepared for this moment?

Ras151315 Jul 2023 0:26 UTC

118 points

64 comments2 min readLW link

Simple alignment plan that maybe works

Iknownothing18 Jul 2023 22:48 UTC

4 points

8 comments1 min readLW link

Runaway Optimizers in Mind Space

silentbob16 Jul 2023 14:26 UTC

16 points

0 comments12 min readLW link

Quick Thoughts on Language Models

RohanS18 Jul 2023 20:38 UTC

6 points

0 comments4 min readLW link

[Question] Why don’t we consider large forms of social organization (economic and political forms, in particular) to qualify as AGI and Transformative AI?

T L16 Jul 2023 18:54 UTC

1 point

0 comments2 min readLW link

Proof of posteriority: a defense against AI-generated misinformation

jchan17 Jul 2023 12:04 UTC

31 points

3 comments5 min readLW link

[Crosspost] An AI Pause Is Humanity’s Best Bet For Preventing Extinction (TIME)

otto.barten24 Jul 2023 10:07 UTC

12 points

0 comments7 min readLW link

(time.com)

All AGI Safety questions welcome (especially basic ones) [July 2023]

smallsilo20 Jul 2023 20:20 UTC

38 points

42 comments2 min readLW link

(forum.effectivealtruism.org)

Supplementary Alignment Insights Through a Highly Controlled Shutdown Incentive

Justausername23 Jul 2023 16:08 UTC

4 points

1 comment3 min readLW link

Autonomous Alignment Oversight Framework (AAOF)

Justausername25 Jul 2023 10:25 UTC

−9 points

0 comments4 min readLW link

A response to the Richards et al.’s “The Illusion of AI’s Existential Risk”

Harrison Fell26 Jul 2023 17:34 UTC

1 point

0 comments10 min readLW link

[Question] Have you ever considered taking the ‘Turing Test’ yourself?

Super AGI27 Jul 2023 3:48 UTC

2 points

6 comments1 min readLW link

Evaluating Superhuman Models with Consistency Checks

Daniel Paleka and Lukas Fluri

1 Aug 2023 7:51 UTC

14 points

2 comments9 min readLW link

(arxiv.org)

3 levels of threat obfuscation

HoldenKarnofsky2 Aug 2023 14:58 UTC

69 points

14 comments7 min readLW link

4 types of AGI selection, and how to constrain them

Remmelt8 Aug 2023 10:02 UTC

−4 points

3 comments3 min readLW link

Embedding Ethical Priors into AI Systems: A Bayesian Approach

Justausername3 Aug 2023 15:31 UTC

−5 points

3 comments21 min readLW link

Ilya Sutskever’s thoughts on AI safety (July 2023): a transcript with my comments

mishka10 Aug 2023 19:07 UTC

21 points

3 comments5 min readLW link

Some alignment ideas

SelonNerias10 Aug 2023 17:51 UTC

1 point

0 comments11 min readLW link

Seeking Input to AI Safety Book for non-technical audience

Darren McKee10 Aug 2023 17:58 UTC

10 points

4 comments1 min readLW link

Existentially relevant thought experiment: To kill or not to kill, a sniper, a man and a button.

AlexFromSafeTransition14 Aug 2023 10:53 UTC

−18 points

6 comments4 min readLW link

We Should Prepare for a Larger Representation of Academia in AI Safety

Leon Lang13 Aug 2023 18:03 UTC

89 points

13 comments5 min readLW link

Decomposing independent generalizations in neural networks via Hessian analysis

Dmitry Vaintrob and Nina Rimsky

14 Aug 2023 17:04 UTC

80 points

3 comments1 min readLW link

AISN #19: US-China Competition on AI Chips, Measuring Language Agent Developments, Economic Analysis of Language Model Propaganda, and White House AI Cyber Challenge

aogara and Dan H

15 Aug 2023 16:10 UTC

21 points

0 comments5 min readLW link

(newsletter.safe.ai)

Ideas for improving epistemics in AI safety outreach

mic21 Aug 2023 19:55 UTC

64 points

6 comments3 min readLW link

Mesa-Optimization: Explain it like I’m 10 Edition

brook26 Aug 2023 23:04 UTC

20 points

1 comment6 min readLW link

Enhancing Corrigibility in AI Systems through Robust Feedback Loops

Justausername24 Aug 2023 3:53 UTC

1 point

0 comments6 min readLW link

Reflections on “Making the Atomic Bomb”

boazbarak17 Aug 2023 2:48 UTC

50 points

7 comments8 min readLW link

AI Regulation May Be More Important Than AI Alignment For Existential Safety

otto.barten24 Aug 2023 11:41 UTC

59 points

38 comments5 min readLW link

When discussing AI doom barriers propose specific plausible scenarios

anithite18 Aug 2023 4:06 UTC

5 points

0 comments3 min readLW link

2 unusual reasons for why we can avoid being turned into paperclips

Artem Panush19 Aug 2023 10:28 UTC

1 point

0 comments4 min readLW link

[Question] Clarifying how misalignment can arise from scaling LLMs

Util19 Aug 2023 14:16 UTC

3 points

1 comment1 min readLW link

Will AI kill everyone? Here’s what the godfathers of AI have to say [RA video]

Writer19 Aug 2023 17:29 UTC

56 points

8 comments1 min readLW link

(youtu.be)

Report on Frontier Model Training

YafahEdelman30 Aug 2023 20:02 UTC

119 points

18 comments21 min readLW link

(docs.google.com)

Ramble on STUFF: intelligence, simulation, AI, doom, default mode, the usual

Bill Benzon26 Aug 2023 15:49 UTC

5 points

0 comments4 min readLW link

The Game of Dominance

Karl von Wendt27 Aug 2023 11:04 UTC

24 points

15 comments6 min readLW link

A Letter to the Editor of MIT Technology Review

Jeffs30 Aug 2023 16:59 UTC

0 points

0 comments1 min readLW link

The Epistemic Authority of Deep Learning Pioneers

Dylan Bowman29 Aug 2023 18:14 UTC

8 points

2 comments3 min readLW link

AISN #20: LLM Proliferation, AI Deception, and Continuing Drivers of AI Capabilities

aogara and Dan H

29 Aug 2023 15:07 UTC

12 points

0 comments8 min readLW link

(newsletter.safe.ai)

Video essay: How Will We Know When AI is Conscious?

JanPro6 Sep 2023 18:10 UTC

11 points

7 comments1 min readLW link

(www.youtube.com)

AISN #21: Google DeepMind’s GPT-4 Competitor, Military Investments in Autonomous Drones, The UK AI Safety Summit, and Case Studies in AI Policy

aogara and Dan H

5 Sep 2023 15:03 UTC

15 points

0 comments5 min readLW link

(newsletter.safe.ai)

[Question] What is to be done? (About the profit motive)

Connor Barber8 Sep 2023 19:27 UTC

1 point

21 comments1 min readLW link

How teams went about their research at AI Safety Camp edition 8

Remmelt, Linda Linsefors and Kristi Uustalu

9 Sep 2023 16:34 UTC

28 points

0 comments13 min readLW link

A conversation with Pi, a conversational AI.

Spiritus Dei15 Sep 2023 23:13 UTC

1 point

0 comments1 min readLW link

Jimmy Apples, source of the rumor that OpenAI has achieved AGI internally, is a credible insider.

Jorterder28 Sep 2023 1:20 UTC

−6 points

2 comments1 min readLW link

(twitter.com)

MMLU’s Moral Scenarios Benchmark Doesn’t Measure What You Think it Measures

corey morris27 Sep 2023 17:54 UTC

14 points

2 comments4 min readLW link

(medium.com)

Technical AI Safety Research Landscape [Slides]

Magdalena Wache18 Sep 2023 13:56 UTC

38 points

0 comments4 min readLW link

Immortality or death by AGI

ImmortalityOrDeathByAGI21 Sep 2023 23:59 UTC

46 points

30 comments4 min readLW link

(forum.effectivealtruism.org)

[Linkpost] Mark Zuckerberg confronted about Meta’s Llama 2 AI’s ability to give users detailed guidance on making anthrax—Business Insider

mic26 Sep 2023 12:05 UTC

18 points

11 comments2 min readLW link

(www.businessinsider.com)

“Diamondoid bacteria” nanobots: deadly threat or dead-end? A nanotech investigation

titotal29 Sep 2023 14:01 UTC

144 points

50 comments1 min readLW link

(titotal.substack.com)

I designed an AI safety course (for a philosophy department)

Eleni Angelou23 Sep 2023 22:03 UTC

37 points

15 comments2 min readLW link

Taking features out of superposition with sparse autoencoders more quickly with informed initialization

Pierre Peigné23 Sep 2023 16:21 UTC

29 points

8 comments5 min readLW link

Linkpost: Are Emergent Abilities in Large Language Models just In-Context Learning?

Erich_Grunewald8 Oct 2023 12:14 UTC

11 points

1 comment2 min readLW link

(arxiv.org)

Paper: Identifying the Risks of LM Agents with an LM-Emulated Sandbox—University of Toronto 2023 - Benchmark consisting of 36 high-stakes tools and 144 test cases!

Singularian25019 Oct 2023 0:00 UTC

5 points

0 comments1 min readLW link

A thought experiment to help persuade skeptics that power-seeking AI is plausible

jacobcd5225 Nov 2023 23:26 UTC

1 point

3 comments5 min readLW link

Ideation and Trajectory Modelling in Language Models

NickyP5 Oct 2023 19:21 UTC

14 points

2 comments10 min readLW link

The Gradient – The Artificiality of Alignment

mic8 Oct 2023 4:06 UTC

12 points

1 comment5 min readLW link

(thegradient.pub)

Become a PIBBSS Research Affiliate

Nora_Ammann and DusanDNesic

10 Oct 2023 7:41 UTC

28 points

6 comments6 min readLW link

LoRA Fine-tuning Efficiently Undoes Safety Training from Llama 2-Chat 70B

Simon Lermen and Jeffrey Ladish

12 Oct 2023 19:58 UTC

138 points

28 comments14 min readLW link

Back to the Past to the Future

Prometheus18 Oct 2023 16:51 UTC

5 points

0 comments1 min readLW link

Taxonomy of AI-risk counterarguments

Odd anon16 Oct 2023 0:12 UTC

61 points

13 comments8 min readLW link

AISC team report: Soft-optimization, Bayes and Goodhart

Simon Fischer, benjaminko, jazcarretao, DFNaiff and Jeremy Gillen

27 Jun 2023 6:05 UTC

36 points

0 comments15 min readLW link

Access to AI: a human right?

dmtea25 Jul 2020 9:38 UTC

5 points

3 comments2 min readLW link

Agentic Language Model Memes

FactorialCode1 Aug 2020 18:03 UTC

16 points

1 comment2 min readLW link

Conversation with Paul Christiano

abergal11 Sep 2019 23:20 UTC

44 points

6 comments30 min readLW link

(aiimpacts.org)

Transcription of Eliezer’s January 2010 video Q&A

curiousepic14 Nov 2011 17:02 UTC

112 points

9 comments56 min readLW link

Responses to Catastrophic AGI Risk: A Survey

lukeprog8 Jul 2013 14:33 UTC

17 points

8 comments1 min readLW link

How can I reduce existential risk from AI?

lukeprog13 Nov 2012 21:56 UTC

63 points

92 comments8 min readLW link

Thoughts on Ben Garfinkel’s “How sure are we about this AI stuff?”

David Scott Krueger (formerly: capybaralet)6 Feb 2019 19:09 UTC

25 points

17 comments1 min readLW link

Reframing misaligned AGI’s: well-intentioned non-neurotypical assistants

zhukeepa1 Apr 2018 1:22 UTC

46 points

14 comments2 min readLW link

When is unaligned AI morally valuable?

paulfchristiano25 May 2018 1:57 UTC

73 points

53 comments10 min readLW link

Introducing the AI Alignment Forum (FAQ)

habryka, Ben Pace, Raemon and jimrandomh

29 Oct 2018 21:07 UTC

86 points

8 comments6 min readLW link

Swimming Upstream: A Case Study in Instrumental Rationality

TurnTrout3 Jun 2018 3:16 UTC

76 points

7 comments8 min readLW link

Current AI Safety Roles for Software Engineers

ozziegooen9 Nov 2018 20:57 UTC

70 points

9 comments4 min readLW link

[Question] Why is so much discussion happening in private Google Docs?

Wei Dai12 Jan 2019 2:19 UTC

100 points

22 comments1 min readLW link

Problems in AI Alignment that philosophers could potentially contribute to

Wei Dai17 Aug 2019 17:38 UTC

76 points

14 comments2 min readLW link

Two Neglected Problems in Human-AI Safety

Wei Dai16 Dec 2018 22:13 UTC

89 points

24 comments2 min readLW link

Announcement: AI alignment prize round 4 winners

cousin_it20 Jan 2019 14:46 UTC

74 points

41 comments1 min readLW link

Soon: a weekly AI Safety prerequisites module on LessWrong

null30 Apr 2018 13:23 UTC

35 points

10 comments1 min readLW link

And the AI would have got away with it too, if...

Stuart_Armstrong22 May 2019 21:35 UTC

75 points

7 comments1 min readLW link

2017 AI Safety Literature Review and Charity Comparison

Larks24 Dec 2017 18:52 UTC

41 points

5 comments23 min readLW link

Should ethicists be inside or outside a profession?

Eliezer Yudkowsky12 Dec 2018 1:40 UTC

88 points

7 comments9 min readLW link

I Vouch For MIRI

Zvi17 Dec 2017 17:50 UTC

37 points

9 comments5 min readLW link

(thezvi.wordpress.com)

Beware of black boxes in AI alignment research

cousin_it18 Jan 2018 15:07 UTC

39 points

10 comments1 min readLW link

AI Alignment Prize: Round 2 due March 31, 2018

Zvi12 Mar 2018 12:10 UTC

28 points

2 comments3 min readLW link

(thezvi.wordpress.com)

Three AI Safety Related Ideas

Wei Dai13 Dec 2018 21:32 UTC

68 points

38 comments2 min readLW link

A rant against robots

Lê Nguyên Hoang14 Jan 2020 22:03 UTC

65 points

7 comments5 min readLW link

Opportunities for individual donors in AI safety

Alex Flint31 Mar 2018 18:37 UTC

30 points

3 comments11 min readLW link

Course recommendations for Friendliness researchers

Louie9 Jan 2013 14:33 UTC

96 points

112 comments10 min readLW link

AI Safety Research Camp—Project Proposal

David_Kristoffersson2 Feb 2018 4:25 UTC

29 points

11 comments8 min readLW link

AI Summer Fellows Program

colm21 Mar 2018 15:32 UTC

21 points

0 comments1 min readLW link

The genie knows, but doesn’t care

Rob Bensinger6 Sep 2013 6:42 UTC

120 points

495 comments8 min readLW link

Alignment Newsletter #13: 07/02/18

Rohin Shah2 Jul 2018 16:10 UTC

70 points

12 comments8 min readLW link

(mailchi.mp)

An Increasingly Manipulative Newsfeed

Michaël Trazzi1 Jul 2019 15:26 UTC

62 points

16 comments5 min readLW link

The simple picture on AI safety

Alex Flint27 May 2018 19:43 UTC

31 points

10 comments2 min readLW link

Elon Musk donates $10M to the Future of Life Institute to keep AI beneficial

Paul Crowley15 Jan 2015 16:33 UTC

78 points

52 comments1 min readLW link

Strategic implications of AIs’ ability to coordinate at low cost, for example by merging

Wei Dai25 Apr 2019 5:08 UTC

67 points

46 comments2 min readLW link 1 review

Modeling AGI Safety Frameworks with Causal Influence Diagrams

Ramana Kumar21 Jun 2019 12:50 UTC

43 points

6 comments1 min readLW link

(arxiv.org)

Henry Kissinger: AI Could Mean the End of Human History

ESRogs15 May 2018 20:11 UTC

17 points

12 comments1 min readLW link

(www.theatlantic.com)

Toy model of the AI control problem: animated version

Stuart_Armstrong10 Oct 2017 11:06 UTC

23 points

8 comments1 min readLW link

A Visualization of Nick Bostrom’s Superintelligence

[deleted]23 Jul 2014 0:24 UTC

62 points

28 comments3 min readLW link

AI Alignment Research Overview (by Jacob Steinhardt)

Ben Pace6 Nov 2019 19:24 UTC

44 points

0 comments7 min readLW link

(docs.google.com)

A general model of safety-oriented AI development

Wei Dai11 Jun 2018 21:00 UTC

65 points

8 comments1 min readLW link

Counterfactual Oracles = online supervised learning with random selection of training episodes

Wei Dai10 Sep 2019 8:29 UTC

48 points

26 comments3 min readLW link

Siren worlds and the perils of over-optimised search

Stuart_Armstrong7 Apr 2014 11:00 UTC

76 points

418 comments7 min readLW link

Top 9+2 myths about AI risk

Stuart_Armstrong29 Jun 2015 20:41 UTC

68 points

45 comments2 min readLW link

Rohin Shah on reasons for AI optimism

abergal31 Oct 2019 12:10 UTC

40 points

58 comments1 min readLW link

(aiimpacts.org)

Plausibly, almost every powerful algorithm would be manipulative

Stuart_Armstrong6 Feb 2020 11:50 UTC

38 points

25 comments3 min readLW link

The Magnitude of His Own Folly

Eliezer Yudkowsky30 Sep 2008 11:31 UTC

95 points

127 comments6 min readLW link

AI alignment landscape

paulfchristiano13 Oct 2019 2:10 UTC

40 points

3 comments1 min readLW link

(ai-alignment.com)

Launched: Friendship is Optimal

iceman15 Nov 2012 4:57 UTC

74 points

32 comments1 min readLW link

Friendship is Optimal: A My Little Pony fanfic about an optimization process

iceman8 Sep 2012 6:16 UTC

104 points

151 comments1 min readLW link

Do Earths with slower economic growth have a better chance at FAI?

Eliezer Yudkowsky12 Jun 2013 19:54 UTC

58 points

175 comments4 min readLW link

Idea: Open Access AI Safety Journal

Gordon Seidoh Worley23 Mar 2018 18:27 UTC

28 points

11 comments1 min readLW link

G.K. Chesterton On AI Risk

Scott Alexander1 Apr 2017 19:00 UTC

17 points

0 comments7 min readLW link

The Friendly AI Game

bentarm15 Mar 2011 16:45 UTC

50 points

178 comments1 min readLW link

Q&A with Jürgen Schmidhuber on risks from AI

XiXiDu15 Jun 2011 15:51 UTC

61 points

45 comments4 min readLW link

[Question] What should an Einstein-like figure in Machine Learning do?

Razied5 Aug 2020 23:52 UTC

7 points

4 comments1 min readLW link

Takeaways from safety by default interviews

AI Impacts and abergal

3 Apr 2020 17:20 UTC

28 points

2 comments13 min readLW link

(aiimpacts.org)

Field-Building and Deep Models

Ben Pace13 Jan 2018 21:16 UTC

21 points

12 comments4 min readLW link

Critique my Model: The EV of AGI to Selfish Individuals

ozziegooen8 Apr 2018 20:04 UTC

19 points

9 comments4 min readLW link

‘Dumb’ AI observes and manipulates controllers

Stuart_Armstrong13 Jan 2015 13:35 UTC

52 points

19 comments2 min readLW link

2019 AI Alignment Literature Review and Charity Comparison

Larks19 Dec 2019 3:00 UTC

130 points

18 comments62 min readLW link

Book review: Architects of Intelligence by Martin Ford (2018)

Ofer11 Aug 2020 17:30 UTC

15 points

0 comments2 min readLW link

Qualitative Strategies of Friendliness

Eliezer Yudkowsky30 Aug 2008 2:12 UTC

30 points

56 comments12 min readLW link

Dreams of Friendliness

Eliezer Yudkowsky31 Aug 2008 1:20 UTC

26 points

81 comments9 min readLW link

Conceptual issues in AI safety: the paradigmatic gap

vedevazz24 Jun 2018 15:09 UTC

33 points

0 comments1 min readLW link

(www.foldl.me)

On unfixably unsafe AGI architectures

Steven Byrnes19 Feb 2020 21:16 UTC

33 points

8 comments5 min readLW link

A toy model of the treacherous turn

Stuart_Armstrong8 Jan 2016 12:58 UTC

42 points

13 comments6 min readLW link

Allegory On AI Risk, Game Theory, and Mithril

James_Miller13 Feb 2017 20:41 UTC

45 points

57 comments3 min readLW link

1hr talk: Intro to AGI safety

Steven Byrnes18 Jun 2019 21:41 UTC

36 points

4 comments24 min readLW link

The Evil AI Overlord List

Stuart_Armstrong20 Nov 2012 17:02 UTC

44 points

80 comments1 min readLW link

What I would like the SIAI to publish

XiXiDu1 Nov 2010 14:07 UTC

36 points

225 comments3 min readLW link

Evaluating the feasibility of SI’s plan

JoshuaFox10 Jan 2013 8:17 UTC

39 points

187 comments4 min readLW link

Q&A with experts on risks from AI #1

XiXiDu8 Jan 2012 11:46 UTC

45 points

67 comments9 min readLW link

Algo trading is a central example of AI risk

Vanessa Kosoy28 Jul 2018 20:31 UTC

27 points

5 comments1 min readLW link

Will the world’s elites navigate the creation of AI just fine?

lukeprog31 May 2013 18:49 UTC

36 points

266 comments2 min readLW link

Let’s talk about “Convergent Rationality”

David Scott Krueger (formerly: capybaralet)12 Jun 2019 21:53 UTC

41 points

33 comments6 min readLW link

Breaking Oracles: superrationality and acausal trade

Stuart_Armstrong25 Nov 2019 10:40 UTC

25 points

15 comments1 min readLW link

Q&A with Stan Franklin on risks from AI

XiXiDu11 Jun 2011 15:22 UTC

36 points

10 comments2 min readLW link

Muehlhauser-Goertzel Dialogue, Part 1

lukeprog16 Mar 2012 17:12 UTC

42 points

161 comments33 min readLW link

[LINK] NYT Article about Existential Risk from AI

[deleted]28 Jan 2013 10:37 UTC

38 points

23 comments1 min readLW link

Reframing the Problem of AI Progress

Wei Dai12 Apr 2012 19:31 UTC

32 points

47 comments1 min readLW link

New AI risks research institute at Oxford University

lukeprog16 Nov 2011 18:52 UTC

36 points

10 comments1 min readLW link

Thoughts on the Feasibility of Prosaic AGI Alignment?

iamthouthouarti21 Aug 2020 23:25 UTC

8 points

10 comments1 min readLW link

Memes and Rational Decisions

inferential9 Jan 2015 6:42 UTC

35 points

18 comments10 min readLW link

Levels of AI Self-Improvement

avturchin29 Apr 2018 11:45 UTC

11 points

1 comment39 min readLW link

Optimising Society to Constrain Risk of War from an Artificial Superintelligence

JohnCDraper30 Apr 2020 10:47 UTC

3 points

1 comment51 min readLW link

Some Thoughts on Singularity Strategies

Wei Dai13 Jul 2011 2:41 UTC

41 points

30 comments3 min readLW link

A trick for Safer GPT-N

Razied23 Aug 2020 0:39 UTC

7 points

1 comment2 min readLW link

against “AI risk”

Wei Dai11 Apr 2012 22:46 UTC

35 points

91 comments1 min readLW link

“Smarter than us” is out!

Stuart_Armstrong25 Feb 2014 15:50 UTC

41 points

57 comments1 min readLW link

Analysing: Dangerous messages from future UFAI via Oracles

Stuart_Armstrong22 Nov 2019 14:17 UTC

22 points

16 comments4 min readLW link

Q&A with Abram Demski on risks from AI

XiXiDu17 Jan 2012 9:43 UTC

33 points

71 comments9 min readLW link

Q&A with experts on risks from AI #2

XiXiDu9 Jan 2012 19:40 UTC

22 points

29 comments7 min readLW link

AI Safety Discussion Day

Linda Linsefors15 Sep 2020 14:40 UTC

20 points

0 comments1 min readLW link

A long reply to Ben Garfinkel on Scrutinizing Classic AI Risk Arguments

Søren Elverlin27 Sep 2020 17:51 UTC

17 points

6 comments1 min readLW link

Open-ended ethics of phenomena (a desiderata with universal morality)

Ryo 8 Nov 2023 20:10 UTC

1 point

0 comments8 min readLW link

Online AI Safety Discussion Day

Linda Linsefors8 Oct 2020 12:11 UTC

5 points

0 comments1 min readLW link

Military AI as a Convergent Goal of Self-Improving AI

avturchin13 Nov 2017 12:17 UTC

5 points

3 comments1 min readLW link

Neural program synthesis is a dangerous technology

syllogism12 Jan 2018 16:19 UTC

10 points

6 comments2 min readLW link

New, Brief Popular-Level Introduction to AI Risks and Superintelligence

LyleN23 Jan 2015 15:43 UTC

33 points

3 comments1 min readLW link

FAI Research Constraints and AGI Side Effects

JustinShovelain3 Jun 2015 19:25 UTC

27 points

59 comments7 min readLW link

European Master’s Programs in Machine Learning, Artificial Intelligence, and related fields

Master Programs ML/AI14 Nov 2020 15:51 UTC

33 points

6 comments1 min readLW link

The mind-killer

Paul Crowley2 May 2009 16:49 UTC

29 points

160 comments2 min readLW link

[Question] Should I do it?

MrLight19 Nov 2020 1:08 UTC

−3 points

16 comments2 min readLW link

Rationalising humans: another mugging, but not Pascal’s

Stuart_Armstrong14 Nov 2017 15:46 UTC

7 points

1 comment3 min readLW link

Machine learning could be fundamentally unexplainable

George3d616 Dec 2020 13:32 UTC

26 points

15 comments15 min readLW link

(cerebralab.com)

[Question] What do you make of AGI:unaligned::spaceships:not enough food?

Ronny Fernandez22 Feb 2020 14:14 UTC

4 points

3 comments1 min readLW link

Risk Map of AI Systems

VojtaKovarik and Jan_Kulveit

15 Dec 2020 9:16 UTC

28 points

3 comments8 min readLW link

[Question] Does it become easier, or harder, for the world to coordinate around not building AGI as time goes on?

Eli Tyre29 Jul 2019 22:59 UTC

86 points

31 comments3 min readLW link 2 reviews

Grey Goo Requires AI

harsimony15 Jan 2021 4:45 UTC

8 points

11 comments4 min readLW link

(harsimony.wordpress.com)

AISU 2021

Linda Linsefors30 Jan 2021 17:40 UTC

28 points

2 comments1 min readLW link

Nonperson Predicates

Eliezer Yudkowsky27 Dec 2008 1:47 UTC

53 points

177 comments6 min readLW link

Engaging First Introductions to AI Risk

Rob Bensinger19 Aug 2013 6:26 UTC

31 points

21 comments3 min readLW link

Formal Solution to the Inner Alignment Problem

michaelcohen18 Feb 2021 14:51 UTC

49 points

123 comments2 min readLW link

[Question] What are the biggest current impacts of AI?

Sam Clarke7 Mar 2021 21:44 UTC

15 points

5 comments1 min readLW link

[Question] Is a Self-Iterating AGI Vulnerable to Thompson-style Trojans?

sxae25 Mar 2021 14:46 UTC

15 points

6 comments3 min readLW link

AI oracles on blockchain

Caravaggio6 Apr 2021 20:13 UTC

5 points

0 comments3 min readLW link

What if AGI is near?

Wulky Wilkinsen14 Apr 2021 0:05 UTC

11 points

5 comments1 min readLW link

[Question] Is there anything that can stop AGI development in the near term?

Wulky Wilkinsen22 Apr 2021 20:37 UTC

5 points

5 comments1 min readLW link

[Question] [timeboxed exercise] write me your model of AI human-existential safety and the alignment problems in 15 minutes

Quinn4 May 2021 19:10 UTC

6 points

2 comments1 min readLW link

AI Safety Research Project Ideas

Owain_Evans21 May 2021 13:39 UTC

58 points

2 comments3 min readLW link

Survey on AI existential risk scenarios

Sam Clarke, apc and Jonas Schuett

8 Jun 2021 17:12 UTC

63 points

11 comments7 min readLW link

[Question] What are some claims or opinions about multi-multi delegation you’ve seen in the memeplex that you think deserve scrutiny?

Quinn27 Jun 2021 17:44 UTC

17 points

6 comments2 min readLW link

Mauhn Releases AI Safety Documentation

Berg Severens3 Jul 2021 21:23 UTC

4 points

0 comments1 min readLW link

A gentle apocalypse

pchvykov16 Aug 2021 5:03 UTC

3 points

5 comments3 min readLW link

Could you have stopped Chernobyl?

Carlos Ramirez27 Aug 2021 1:48 UTC

29 points

17 comments8 min readLW link

The Governance Problem and the “Pretty Good” X-Risk

Zach Stein-Perlman29 Aug 2021 18:00 UTC

5 points

2 comments11 min readLW link

Distinguishing AI takeover scenarios

Sam Clarke and Sammy Martin

8 Sep 2021 16:19 UTC

72 points

11 comments14 min readLW link

The alignment problem in different capability regimes

Buck9 Sep 2021 19:46 UTC

88 points

12 comments5 min readLW link

How truthful is GPT-3? A benchmark for language models

Owain_Evans16 Sep 2021 10:09 UTC

56 points

24 comments6 min readLW link

Investigating AI Takeover Scenarios

Sammy Martin17 Sep 2021 18:47 UTC

27 points

1 comment27 min readLW link

AI takeoff story: a continuation of progress by other means

Edouard Harris27 Sep 2021 15:55 UTC

76 points

13 comments10 min readLW link

A brief review of the reasons multi-objective RL could be important in AI Safety Research

Ben Smith29 Sep 2021 17:09 UTC

31 points

7 comments10 min readLW link

The Dark Side of Cognition Hypothesis

Cameron Berg3 Oct 2021 20:10 UTC

19 points

1 comment16 min readLW link

Truthful AI: Developing and governing AI that does not lie

Owain_Evans, owencb and Lukas Finnveden

18 Oct 2021 18:37 UTC

81 points

9 comments10 min readLW link

AMA on Truthful AI: Owen Cotton-Barratt, Owain Evans & co-authors

Owain_Evans22 Oct 2021 16:23 UTC

31 points

15 comments1 min readLW link

Truthful and honest AI

abergal, Nick_Beckstead and Owain_Evans

29 Oct 2021 7:28 UTC

42 points

1 comment13 min readLW link

What is the most evil AI that we could build, today?

ThomasJ1 Nov 2021 19:58 UTC

−2 points

14 comments1 min readLW link

What are red flags for Neural Network suffering?

Marius Hobbhahn8 Nov 2021 12:51 UTC

29 points

15 comments12 min readLW link

Hardcode the AGI to need our approval indefinitely?

MichaelStJules11 Nov 2021 7:04 UTC

2 points

2 comments1 min readLW link

What would we do if alignment were futile?

Grant Demaree14 Nov 2021 8:09 UTC

75 points

39 comments3 min readLW link

Two Stupid AI Alignment Ideas

aphyer16 Nov 2021 16:13 UTC

24 points

3 comments4 min readLW link

Super intelligent AIs that don’t require alignment

Yair Halberstadt16 Nov 2021 19:55 UTC

10 points

2 comments6 min readLW link

AI Tracker: monitoring current and near-future risks from superscale models

Edouard Harris and Jeremie Harris

23 Nov 2021 19:16 UTC

64 points

13 comments3 min readLW link

(aitracker.org)

HIRING: Inform and shape a new project on AI safety at Partnership on AI

Madhulika Srikumar24 Nov 2021 8:27 UTC

6 points

0 comments1 min readLW link

How to measure FLOP/s for Neural Networks empirically?

Marius Hobbhahn29 Nov 2021 15:18 UTC

16 points

5 comments7 min readLW link

Modeling Failure Modes of High-Level Machine Intelligence

Ben Cottier, Daniel_Eth and Sammy Martin

6 Dec 2021 13:54 UTC

54 points

1 comment12 min readLW link

HIRING: Inform and shape a new project on AI safety at Partnership on AI

madhu_lika7 Dec 2021 19:37 UTC

1 point

0 comments1 min readLW link

Universality and the “Filter”

maggiehayes16 Dec 2021 0:47 UTC

10 points

2 comments11 min readLW link

Reviews of “Is power-seeking AI an existential risk?”

Joe Carlsmith16 Dec 2021 20:48 UTC

78 points

20 comments1 min readLW link

2+2: Ontological Framework

Lyrialtus1 Feb 2022 1:07 UTC

−15 points

2 comments12 min readLW link

Can the laws of physics/nature prevent hell?

superads916 Feb 2022 20:39 UTC

−5 points

8 comments2 min readLW link

How harmful are improvements in AI? + Poll

tilmanr and Marius Hobbhahn

15 Feb 2022 18:16 UTC

15 points

4 comments8 min readLW link

Preserving and continuing alignment research through a severe global catastrophe

A_donor6 Mar 2022 18:43 UTC

39 points

11 comments5 min readLW link

Ask AI companies about what they are doing for AI safety?

mic9 Mar 2022 15:14 UTC

51 points

0 comments2 min readLW link

Is There a Valley of Bad Civilizational Adequacy?

lbThingrb11 Mar 2022 19:49 UTC

13 points

1 comment2 min readLW link

[Question] Danger(s) of theorem-proving AI?

Yitz16 Mar 2022 2:47 UTC

8 points

8 comments1 min readLW link

We Are Conjecture, A New Alignment Research Startup

Connor Leahy8 Apr 2022 11:40 UTC

197 points

25 comments4 min readLW link

Is technical AI alignment research a net positive?

cranberry_bear12 Apr 2022 13:07 UTC

6 points

2 comments2 min readLW link

The Peerless

Tamsin Leake13 Apr 2022 1:07 UTC

18 points

2 comments1 min readLW link

(carado.moe)

[Question] Can someone explain to me why MIRI is so pessimistic of our chances of survival?

iamthouthouarti14 Apr 2022 20:28 UTC

10 points

7 comments1 min readLW link

[Question] Convince me that humanity isn’t doomed by AGI

Yitz15 Apr 2022 17:26 UTC

61 points

49 comments1 min readLW link

Reflections on My Own Missing Mood

Lone Pine21 Apr 2022 16:19 UTC

52 points

25 comments5 min readLW link

Code Generation as an AI risk setting

Not Relevant17 Apr 2022 22:27 UTC

91 points

16 comments2 min readLW link

[Question] What is being improved in recursive self improvement?

Lone Pine25 Apr 2022 18:30 UTC

7 points

6 comments1 min readLW link

AI Alternative Futures: Scenario Mapping Artificial Intelligence Risk—Request for Participation (Closed)

Kakili27 Apr 2022 22:07 UTC

10 points

2 comments8 min readLW link

Video and Transcript of Presentation on Existential Risk from Power-Seeking AI

Joe Carlsmith8 May 2022 3:50 UTC

20 points

1 comment29 min readLW link

Interpretability’s Alignment-Solving Potential: Analysis of 7 Scenarios

Evan R. Murphy12 May 2022 20:01 UTC

53 points

0 comments59 min readLW link

Agency As a Natural Abstraction

Thane Ruthenis13 May 2022 18:02 UTC

55 points

9 comments13 min readLW link

[Link post] Promising Paths to Alignment—Connor Leahy | Talk

frances_lorenz14 May 2022 16:01 UTC

34 points

0 comments1 min readLW link

DeepMind’s generalist AI, Gato: A non-technical explainer

frances_lorenz, Nora Belrose and jonmenaster

16 May 2022 21:21 UTC

57 points

6 comments6 min readLW link

Actionable-guidance and roadmap recommendations for the NIST AI Risk Management Framework

Dan H and Tony Barrett

17 May 2022 15:26 UTC

26 points

0 comments3 min readLW link

Why I’m Optimistic About Near-Term AI Risk

harsimony15 May 2022 23:05 UTC

57 points

27 comments1 min readLW link

Pivotal acts using an unaligned AGI?

Simon Fischer21 Aug 2022 17:13 UTC

26 points

3 comments7 min readLW link

Reshaping the AI Industry

Thane Ruthenis29 May 2022 22:54 UTC

147 points

35 comments21 min readLW link

Explaining inner alignment to myself

Jeremy Gillen24 May 2022 23:10 UTC

9 points

2 comments10 min readLW link

A Story of AI Risk: InstructGPT-N

peterbarnett26 May 2022 23:22 UTC

24 points

0 comments8 min readLW link

We will be around in 30 years

mukashi7 Jun 2022 3:47 UTC

12 points

205 comments2 min readLW link

Research Questions from Stained Glass Windows

StefanHex8 Jun 2022 12:38 UTC

4 points

0 comments2 min readLW link

Towards Gears-Level Understanding of Agency

Thane Ruthenis16 Jun 2022 22:00 UTC

23 points

4 comments18 min readLW link

A plausible story about AI risk.

DeLesley Hutchins10 Jun 2022 2:08 UTC

14 points

2 comments4 min readLW link

Summary of “AGI Ruin: A List of Lethalities”

Stephen McAleese10 Jun 2022 22:35 UTC

40 points

2 comments8 min readLW link

Poorly-Aimed Death Rays

Thane Ruthenis11 Jun 2022 18:29 UTC

48 points

5 comments4 min readLW link

Contra EY: Can AGI destroy us without trial & error?