AI Risk

TagLast edit: 16 Jul 2020 10:29 UTC by Ben Pace

AI Risk is analysis of the risks associated with building powerful AI systems.

Superintelligence FAQ

Scott Alexander20 Sep 2016 19:00 UTC

84 points

11 comments27 min readLW link

What failure looks like

paulfchristiano17 Mar 2019 20:18 UTC

287 points

49 comments8 min readLW link 2 reviews

Specification gaming examples in AI

Vika3 Apr 2018 12:30 UTC

39 points

9 comments1 min readLW link 2 reviews

Discussion with Eliezer Yudkowsky on AGI interventions

Rob Bensinger and Eliezer Yudkowsky

11 Nov 2021 3:01 UTC

326 points

256 comments34 min readLW link

Intuitions about goal-directed behavior

Rohin Shah1 Dec 2018 4:25 UTC

50 points

15 comments6 min readLW link

Epistemological Framing for AI Alignment Research

adamShimi8 Mar 2021 22:05 UTC

53 points

7 comments9 min readLW link

AGI Ruin: A List of Lethalities

Eliezer Yudkowsky5 Jun 2022 22:05 UTC

655 points

624 comments30 min readLW link

What can the principal-agent literature tell us about AI risk?

Alexis Carlier8 Feb 2020 21:28 UTC

101 points

31 comments16 min readLW link

Developmental Stages of GPTs

orthonormal26 Jul 2020 22:03 UTC

140 points

74 comments7 min readLW link 1 review

[Question] Will OpenAI’s work unintentionally increase existential risks related to AI?

adamShimi11 Aug 2020 18:16 UTC

50 points

56 comments1 min readLW link

Another (outer) alignment failure story

paulfchristiano7 Apr 2021 20:12 UTC

203 points

37 comments12 min readLW link

How good is humanity at coordination?

Buck21 Jul 2020 20:01 UTC

77 points

43 comments3 min readLW link

A Gym Gridworld Environment for the Treacherous Turn

Michaël Trazzi28 Jul 2018 21:27 UTC

71 points

9 comments3 min readLW link

(github.com)

Are minimal circuits deceptive?

evhub7 Sep 2019 18:11 UTC

56 points

11 comments8 min readLW link

Soft takeoff can still lead to decisive strategic advantage

Daniel Kokotajlo23 Aug 2019 16:39 UTC

117 points

46 comments8 min readLW link 4 reviews

Should we postpone AGI until we reach safety?

otto.barten18 Nov 2020 15:43 UTC

26 points

36 comments3 min readLW link

On Solving Problems Before They Appear: The Weird Epistemologies of Alignment

adamShimi11 Oct 2021 8:20 UTC

93 points

11 comments15 min readLW link

DL towards the unaligned Recursive Self-Optimization attractor

jacob_cannell18 Dec 2021 2:15 UTC

32 points

22 comments4 min readLW link

Truthful LMs as a warm-up for aligned AGI

Jacob_Hilton17 Jan 2022 16:49 UTC

64 points

14 comments13 min readLW link

MIRI announces new “Death With Dignity” strategy

Eliezer Yudkowsky2 Apr 2022 0:43 UTC

324 points

518 comments18 min readLW link

AI Could Defeat All Of Us Combined

HoldenKarnofsky9 Jun 2022 15:50 UTC

162 points

27 comments17 min readLW link

(www.cold-takes.com)

Critiquing “What failure looks like”

Grue_Slinky27 Dec 2019 23:59 UTC

35 points

6 comments3 min readLW link

The Main Sources of AI Risk?

Daniel Kokotajlo and Wei_Dai

21 Mar 2019 18:28 UTC

78 points

22 comments2 min readLW link

Clarifying some key hypotheses in AI alignment

Ben Cottier and Rohin Shah

15 Aug 2019 21:29 UTC

78 points

12 comments9 min readLW link

“Taking AI Risk Seriously” (thoughts by Critch)

Raemon29 Jan 2018 9:27 UTC

111 points

68 comments13 min readLW link

Some conceptual highlights from “Disjunctive Scenarios of Catastrophic AI Risk”

Kaj_Sotala12 Feb 2018 12:30 UTC

33 points

4 comments6 min readLW link

(kajsotala.fi)

Non-Adversarial Goodhart and AI Risks

Davidmanheim27 Mar 2018 1:39 UTC

22 points

11 comments6 min readLW link

Six AI Risk/Strategy Ideas

Wei_Dai27 Aug 2019 0:40 UTC

63 points

18 comments4 min readLW link 1 review

[Question] Did AI pioneers not worry much about AI risks?

lisperati9 Feb 2020 19:58 UTC

42 points

9 comments1 min readLW link

Some disjunctive reasons for urgency on AI risk

Wei_Dai15 Feb 2019 20:43 UTC

36 points

24 comments1 min readLW link

Drexler on AI Risk

PeterMcCluskey1 Feb 2019 5:11 UTC

34 points

10 comments9 min readLW link

(www.bayesianinvestor.com)

A shift in arguments for AI risk

Richard_Ngo28 May 2019 13:47 UTC

32 points

7 comments1 min readLW link

(fragile-credences.github.io)

Disentangling arguments for the importance of AI safety

Richard_Ngo21 Jan 2019 12:41 UTC

128 points

23 comments8 min readLW link

AI Safety “Success Stories”

Wei_Dai7 Sep 2019 2:54 UTC

107 points

27 comments4 min readLW link 1 review

Debate on Instrumental Convergence between LeCun, Russell, Bengio, Zador, and More

Ben Pace4 Oct 2019 4:08 UTC

191 points

57 comments15 min readLW link 2 reviews

[AN #80]: Why AI risk might be solved without additional intervention from longtermists

Rohin Shah2 Jan 2020 18:20 UTC

35 points

94 comments10 min readLW link

(mailchi.mp)

The strategy-stealing assumption

paulfchristiano16 Sep 2019 15:23 UTC

70 points

46 comments12 min readLW link 3 reviews

Thinking soberly about the context and consequences of Friendly AI

Mitchell_Porter16 Oct 2012 4:33 UTC

20 points

39 comments1 min readLW link

Announcement: AI alignment prize winners and next round

cousin_it15 Jan 2018 14:33 UTC

80 points

68 comments2 min readLW link

What Failure Looks Like: Distilling the Discussion

Ben Pace29 Jul 2020 21:49 UTC

74 points

12 comments7 min readLW link

Uber Self-Driving Crash

jefftk7 Nov 2019 15:00 UTC

110 points

1 comment2 min readLW link

(www.jefftk.com)

Reply to Holden on ‘Tool AI’

Eliezer Yudkowsky12 Jun 2012 18:00 UTC

151 points

357 comments17 min readLW link

Stanford Encyclopedia of Philosophy on AI ethics and superintelligence

Kaj_Sotala2 May 2020 7:35 UTC

43 points

19 comments7 min readLW link

(plato.stanford.edu)

AGI Safety Literature Review (Everitt, Lea & Hutter 2018)

Kaj_Sotala4 May 2018 8:56 UTC

13 points

1 comment1 min readLW link

(arxiv.org)

Response to Oren Etzioni’s “How to know if artificial intelligence is about to destroy civilization”

Daniel Kokotajlo27 Feb 2020 18:10 UTC

27 points

5 comments8 min readLW link

Why don’t singularitarians bet on the creation of AGI by buying stocks?

John_Maxwell11 Mar 2020 16:27 UTC

43 points

20 comments4 min readLW link

The problem/solution matrix: Calculating the probability of AI safety “on the back of an envelope”

John_Maxwell20 Oct 2019 8:03 UTC

22 points

4 comments2 min readLW link

Three Stories for How AGI Comes Before FAI

John_Maxwell17 Sep 2019 23:26 UTC

27 points

8 comments6 min readLW link

Brainstorming additional AI risk reduction ideas

John_Maxwell14 Jun 2012 7:55 UTC

19 points

37 comments1 min readLW link

AI Alignment 2018-19 Review

Rohin Shah28 Jan 2020 2:19 UTC

125 points

6 comments35 min readLW link

The Fusion Power Generator Scenario

johnswentworth8 Aug 2020 18:31 UTC

132 points

29 comments3 min readLW link

A guide to Iterated Amplification & Debate

Rafael Harth15 Nov 2020 17:14 UTC

66 points

9 comments15 min readLW link

Work on Security Instead of Friendliness?

Wei_Dai21 Jul 2012 18:28 UTC

51 points

107 comments2 min readLW link

An unaligned benchmark

paulfchristiano17 Nov 2018 15:51 UTC

31 points

0 comments9 min readLW link

Artificial Intelligence: A Modern Approach (4th edition) on the Alignment Problem

Zack_M_Davis17 Sep 2020 2:23 UTC

72 points

12 comments5 min readLW link

(aima.cs.berkeley.edu)

Clarifying “What failure looks like”

Sam Clarke20 Sep 2020 20:40 UTC

88 points

14 comments17 min readLW link

Relaxed adversarial training for inner alignment

evhub10 Sep 2019 23:03 UTC

57 points

22 comments27 min readLW link

An overview of 11 proposals for building safe advanced AI

evhub29 May 2020 20:38 UTC

184 points

34 comments38 min readLW link 2 reviews

Risks from Learned Optimization: Introduction

evhub, Chris van Merwijk, vlad_m, Joar Skalse and Scott Garrabrant

31 May 2019 23:44 UTC

156 points

42 comments12 min readLW link 3 reviews

Risks from Learned Optimization: Conclusion and Related Work

evhub, Chris van Merwijk, vlad_m, Joar Skalse and Scott Garrabrant

7 Jun 2019 19:53 UTC

73 points

4 comments6 min readLW link

Deceptive Alignment

evhub, Chris van Merwijk, vlad_m, Joar Skalse and Scott Garrabrant

5 Jun 2019 20:16 UTC

84 points

11 comments17 min readLW link

The Inner Alignment Problem

evhub, Chris van Merwijk, vlad_m, Joar Skalse and Scott Garrabrant

4 Jun 2019 1:20 UTC

90 points

17 comments13 min readLW link

Conditions for Mesa-Optimization

evhub, Chris van Merwijk, vlad_m, Joar Skalse and Scott Garrabrant

1 Jun 2019 20:52 UTC

68 points

47 comments12 min readLW link

AI risk hub in Singapore?

Daniel Kokotajlo29 Oct 2020 11:45 UTC

51 points

18 comments4 min readLW link

Thoughts on Robin Hanson’s AI Impacts interview

Steven Byrnes24 Nov 2019 1:40 UTC

25 points

3 comments7 min readLW link

The AI Safety Game (UPDATED)

Daniel Kokotajlo5 Dec 2020 10:27 UTC

41 points

9 comments3 min readLW link

[Question] Suggestions of posts on the AF to review

adamShimi16 Feb 2021 12:40 UTC

56 points

20 comments1 min readLW link

Google’s Ethical AI team and AI Safety

magfrump20 Feb 2021 9:42 UTC

12 points

16 comments7 min readLW link

Behavioral Sufficient Statistics for Goal-Directedness

adamShimi11 Mar 2021 15:01 UTC

21 points

12 comments9 min readLW link

Review of “Fun with +12 OOMs of Compute”

adamShimi, Joe_Collman and Gyrodiot

28 Mar 2021 14:55 UTC

57 points

20 comments8 min readLW link

April drafts

AI Impacts1 Apr 2021 18:10 UTC

49 points

2 comments1 min readLW link

(aiimpacts.org)

25 Min Talk on MetaEthical.AI with Questions from Stuart Armstrong

June Ku29 Apr 2021 15:38 UTC

21 points

7 comments1 min readLW link

Less Realistic Tales of Doom

Mark Xu6 May 2021 23:01 UTC

100 points

13 comments4 min readLW link

Rogue AGI Embodies Valuable Intellectual Property

Mark Xu and CarlShulman

3 Jun 2021 20:37 UTC

69 points

9 comments3 min readLW link

Environmental Structure Can Cause Instrumental Convergence

TurnTrout22 Jun 2021 22:26 UTC

71 points

44 comments16 min readLW link

(arxiv.org)

Alex Turner’s Research, Comprehensive Information Gathering

adamShimi23 Jun 2021 9:44 UTC

15 points

3 comments3 min readLW link

Sam Altman and Ezra Klein on the AI Revolution

Zack_M_Davis27 Jun 2021 4:53 UTC

38 points

17 comments1 min readLW link

(www.nytimes.com)

Approaches to gradient hacking

adamShimi14 Aug 2021 15:16 UTC

16 points

7 comments8 min readLW link

[Question] What are good alignment conference papers?

adamShimi28 Aug 2021 13:35 UTC

12 points

2 comments1 min readLW link

Why the technological singularity by AGI may never happen

hippke3 Sep 2021 14:19 UTC

5 points

14 comments1 min readLW link

[Question] Conditional on the first AGI being aligned correctly, is a good outcome even still likely?

iamthouthouarti6 Sep 2021 17:30 UTC

2 points

1 comment1 min readLW link

Bayeswatch 7: Wildfire

lsusr8 Sep 2021 5:35 UTC

47 points

6 comments3 min readLW link

[Book Review] “The Alignment Problem” by Brian Christian

lsusr20 Sep 2021 6:36 UTC

69 points

16 comments6 min readLW link

Is progress in ML-assisted theorem-proving beneficial?

MakoYass28 Sep 2021 1:54 UTC

10 points

2 comments1 min readLW link

Interview with Skynet

lsusr30 Sep 2021 2:20 UTC

49 points

1 comment2 min readLW link

Epistemic Strategies of Selection Theorems

adamShimi18 Oct 2021 8:57 UTC

32 points

1 comment12 min readLW link

Epistemic Strategies of Safety-Capabilities Tradeoffs

adamShimi22 Oct 2021 8:22 UTC

5 points

0 comments6 min readLW link

Drug addicts and deceptively aligned agents—a comparative analysis

Jan5 Nov 2021 21:42 UTC

41 points

2 comments12 min readLW link

(universalprior.substack.com)

Using Brain-Computer Interfaces to get more data for AI alignment

Robbo7 Nov 2021 0:00 UTC

37 points

10 comments7 min readLW link

Using blinders to help you see things for what they are

adamzerner11 Nov 2021 7:07 UTC

13 points

2 comments2 min readLW link

My current uncertainties regarding AI, alignment, and the end of the world

dominicq14 Nov 2021 14:08 UTC

2 points

3 comments2 min readLW link

Ngo and Yudkowsky on alignment difficulty

Eliezer Yudkowsky and Richard_Ngo

15 Nov 2021 20:31 UTC

231 points

143 comments99 min readLW link

Applications for AI Safety Camp 2022 Now Open!

adamShimi17 Nov 2021 21:42 UTC

47 points

3 comments1 min readLW link

[Question] Does the Structure of an algorithm matter for AI Risk and/or consciousness?

Logan Zoellner3 Dec 2021 18:31 UTC

7 points

5 comments1 min readLW link

Framing approaches to alignment and the hard problem of AI cognition

ryan_greenblatt15 Dec 2021 19:06 UTC

6 points

15 comments27 min readLW link

Some abstract, non-technical reasons to be non-maximally-pessimistic about AI alignment

Rob Bensinger12 Dec 2021 2:08 UTC

55 points

37 comments6 min readLW link

Summary of the Acausal Attack Issue for AIXI

Diffractor13 Dec 2021 8:16 UTC

13 points

6 comments4 min readLW link

My Overview of the AI Alignment Landscape: A Bird’s Eye View

Neel Nanda15 Dec 2021 23:44 UTC

104 points

9 comments16 min readLW link

My Overview of the AI Alignment Landscape: Threat Models

Neel Nanda25 Dec 2021 23:07 UTC

38 points

4 comments28 min readLW link

AI Fire Alarm Scenarios

PeterMcCluskey28 Dec 2021 2:20 UTC

10 points

0 comments6 min readLW link

(www.bayesianinvestor.com)

More Is Different for AI

jsteinhardt4 Jan 2022 19:30 UTC

135 points

21 comments3 min readLW link

(bounded-regret.ghost.io)

Challenges with Breaking into MIRI-Style Research

Chris_Leong17 Jan 2022 9:23 UTC

69 points

15 comments3 min readLW link

Thoughts on AGI safety from the top

jylin042 Feb 2022 20:06 UTC

21 points

2 comments32 min readLW link

Paradigm-building from first principles: Effective altruism, AGI, and alignment

Cameron Berg8 Feb 2022 16:12 UTC

24 points

5 comments14 min readLW link

Paradigm-building: Introduction

Cameron Berg8 Feb 2022 0:06 UTC

24 points

0 comments2 min readLW link

How I Formed My Own Views About AI Safety

Neel Nanda27 Feb 2022 18:50 UTC

62 points

5 comments13 min readLW link

(www.neelnanda.io)

[Question] Would (myopic) general public good producers significantly accelerate the development of AGI?

MakoYass2 Mar 2022 23:47 UTC

24 points

10 comments1 min readLW link

AXRP Episode 13 - First Principles of AGI Safety with Richard Ngo

DanielFilan31 Mar 2022 5:20 UTC

24 points

1 comment48 min readLW link

[RETRACTED] It’s time for EA leadership to pull the short-timelines fire alarm.

Not Relevant8 Apr 2022 16:07 UTC

111 points

164 comments4 min readLW link

AMA Conjecture, A New Alignment Startup

adamShimi9 Apr 2022 9:43 UTC

43 points

40 comments1 min readLW link

Why I’m Worried About AI

peterbarnett23 May 2022 21:13 UTC

21 points

2 comments12 min readLW link

Complex Systems for AI Safety [Pragmatic AI Safety #3]

Dan Hendrycks and ThomasWoodside

24 May 2022 0:00 UTC

30 points

0 comments21 min readLW link

Will working here advance AGI? Help us not destroy the world!

Yonatan Cale29 May 2022 11:42 UTC

29 points

43 comments1 min readLW link

Distilled—AGI Safety from First Principles

Harrison G29 May 2022 0:57 UTC

4 points

0 comments14 min readLW link

Perform Tractable Research While Avoiding Capabilities Externalities [Pragmatic AI Safety #4]

Dan Hendrycks and ThomasWoodside

30 May 2022 20:25 UTC

33 points

3 comments25 min readLW link

Artificial Intelligence Safety for the Averagely Intelligent

Salsabila Mahdi1 Jun 2022 6:24 UTC

5 points

2 comments1 min readLW link

Confused why a “capabilities research is good for alignment progress” position isn’t discussed more

Kaj_Sotala2 Jun 2022 21:41 UTC

126 points

26 comments4 min readLW link

I’m trying out “asteroid mindset”

Alex_Altair3 Jun 2022 13:35 UTC

83 points

5 comments4 min readLW link

Epistemological Vigilance for Alignment

adamShimi6 Jun 2022 0:27 UTC

45 points

10 comments11 min readLW link

A Quick Guide to Confronting Doom

Ruby13 Apr 2022 19:30 UTC

222 points

36 comments2 min readLW link

Why I don’t believe in doom

mukashi7 Jun 2022 23:49 UTC

6 points

30 comments4 min readLW link

Open Problems in AI X-Risk [PAIS #5]

Dan Hendrycks and ThomasWoodside

10 Jun 2022 2:08 UTC

41 points

3 comments35 min readLW link

Another plausible scenario of AI risk: AI builds military infrastructure while collaborating with humans, defects later.

avturchin10 Jun 2022 17:24 UTC

10 points

2 comments1 min readLW link

How dangerous is human-level AI?

Alex_Altair10 Jun 2022 17:38 UTC

20 points

4 comments8 min readLW link

On A List of Lethalities

Zvi13 Jun 2022 12:30 UTC

147 points

47 comments54 min readLW link

(thezvi.wordpress.com)

Continuity Assumptions

Jan_Kulveit13 Jun 2022 21:31 UTC

24 points

13 comments4 min readLW link

Slow motion videos as AI risk intuition pumps

Andrew_Critch14 Jun 2022 19:31 UTC

193 points

35 comments2 min readLW link

Alignment Risk Doesn’t Require Superintelligence

JustisMills15 Jun 2022 3:12 UTC

35 points

4 comments2 min readLW link

[Question] Has there been any work on attempting to use Pascal’s Mugging to make an AGI behave?

Chris_Leong15 Jun 2022 8:33 UTC

7 points

17 comments1 min readLW link

Where I agree and disagree with Eliezer

paulfchristiano19 Jun 2022 19:15 UTC

644 points

169 comments20 min readLW link

Are we there yet?

theflowerpot20 Jun 2022 11:19 UTC

2 points

2 comments1 min readLW link

Confusion about neuroscience/cognitive science as a danger for AI Alignment

Samuel Nellessen22 Jun 2022 17:59 UTC

2 points

1 comment3 min readLW link

(snellessen.com)

Access to AI: a human right?

dmtea25 Jul 2020 9:38 UTC

5 points

3 comments2 min readLW link

Agentic Language Model Memes

FactorialCode1 Aug 2020 18:03 UTC

16 points

1 comment2 min readLW link

Conversation with Paul Christiano

abergal11 Sep 2019 23:20 UTC

44 points

6 comments30 min readLW link

(aiimpacts.org)

Transcription of Eliezer’s January 2010 video Q&A

curiousepic14 Nov 2011 17:02 UTC

110 points

9 comments56 min readLW link

Responses to Catastrophic AGI Risk: A Survey

lukeprog8 Jul 2013 14:33 UTC

17 points

8 comments1 min readLW link

How can I reduce existential risk from AI?

lukeprog13 Nov 2012 21:56 UTC

63 points

92 comments8 min readLW link

Thoughts on Ben Garfinkel’s “How sure are we about this AI stuff?”

capybaralet6 Feb 2019 19:09 UTC

25 points

17 comments1 min readLW link

Reframing misaligned AGI’s: well-intentioned non-neurotypical assistants

zhukeepa1 Apr 2018 1:22 UTC

46 points

14 comments2 min readLW link

When is unaligned AI morally valuable?

paulfchristiano25 May 2018 1:57 UTC

61 points

52 comments10 min readLW link

Introducing the AI Alignment Forum (FAQ)

habryka, Ben Pace, Raemon and jimrandomh

29 Oct 2018 21:07 UTC

86 points

8 comments6 min readLW link

Swimming Upstream: A Case Study in Instrumental Rationality

TurnTrout3 Jun 2018 3:16 UTC

68 points

7 comments8 min readLW link

Current AI Safety Roles for Software Engineers

ozziegooen9 Nov 2018 20:57 UTC

70 points

9 comments4 min readLW link

[Question] Why is so much discussion happening in private Google Docs?

Wei_Dai12 Jan 2019 2:19 UTC

100 points

22 comments1 min readLW link

Problems in AI Alignment that philosophers could potentially contribute to

Wei_Dai17 Aug 2019 17:38 UTC

74 points

14 comments2 min readLW link

Two Neglected Problems in Human-AI Safety

Wei_Dai16 Dec 2018 22:13 UTC

82 points

24 comments2 min readLW link

Announcement: AI alignment prize round 4 winners

cousin_it20 Jan 2019 14:46 UTC

74 points

41 comments1 min readLW link

Soon: a weekly AI Safety prerequisites module on LessWrong

null30 Apr 2018 13:23 UTC

35 points

10 comments1 min readLW link

And the AI would have got away with it too, if...

Stuart_Armstrong22 May 2019 21:35 UTC

75 points

7 comments1 min readLW link

2017 AI Safety Literature Review and Charity Comparison

Larks24 Dec 2017 18:52 UTC

41 points

5 comments23 min readLW link

Should ethicists be inside or outside a profession?

Eliezer Yudkowsky12 Dec 2018 1:40 UTC

83 points

6 comments9 min readLW link

I Vouch For MIRI

Zvi17 Dec 2017 17:50 UTC

36 points

9 comments5 min readLW link

(thezvi.wordpress.com)

Beware of black boxes in AI alignment research

cousin_it18 Jan 2018 15:07 UTC

39 points

10 comments1 min readLW link

AI Alignment Prize: Round 2 due March 31, 2018

Zvi12 Mar 2018 12:10 UTC

28 points

2 comments3 min readLW link

(thezvi.wordpress.com)

Three AI Safety Related Ideas

Wei_Dai13 Dec 2018 21:32 UTC

68 points

38 comments2 min readLW link

A rant against robots

Lê Nguyên Hoang14 Jan 2020 22:03 UTC

64 points

7 comments5 min readLW link

Opportunities for individual donors in AI safety

Alex Flint31 Mar 2018 18:37 UTC

27 points

3 comments11 min readLW link

But exactly how complex and fragile?

KatjaGrace3 Nov 2019 18:20 UTC

73 points

32 comments3 min readLW link 1 review

(meteuphoric.com)

Course recommendations for Friendliness researchers

Louie9 Jan 2013 14:33 UTC

96 points

112 comments10 min readLW link

AI Safety Research Camp—Project Proposal

David_Kristoffersson2 Feb 2018 4:25 UTC

29 points

11 comments8 min readLW link

AI Summer Fellows Program

colm21 Mar 2018 15:32 UTC

21 points

0 comments1 min readLW link

The genie knows, but doesn’t care

Rob Bensinger6 Sep 2013 6:42 UTC

89 points

519 comments8 min readLW link

Alignment Newsletter #13: 07/02/18

Rohin Shah2 Jul 2018 16:10 UTC

70 points

12 comments8 min readLW link

(mailchi.mp)

An Increasingly Manipulative Newsfeed

Michaël Trazzi1 Jul 2019 15:26 UTC

61 points

16 comments5 min readLW link

The simple picture on AI safety

Alex Flint27 May 2018 19:43 UTC

28 points

10 comments2 min readLW link

Elon Musk donates $10M to the Future of Life Institute to keep AI beneficial

Paul Crowley15 Jan 2015 16:33 UTC

78 points

52 comments1 min readLW link

Strategic implications of AIs’ ability to coordinate at low cost, for example by merging

Wei_Dai25 Apr 2019 5:08 UTC

60 points

45 comments2 min readLW link 1 review

Modeling AGI Safety Frameworks with Causal Influence Diagrams

Ramana Kumar21 Jun 2019 12:50 UTC

43 points

6 comments1 min readLW link

(arxiv.org)

Henry Kissinger: AI Could Mean the End of Human History

ESRogs15 May 2018 20:11 UTC

17 points

12 comments1 min readLW link

(www.theatlantic.com)

Toy model of the AI control problem: animated version

Stuart_Armstrong10 Oct 2017 11:06 UTC

25 points

8 comments1 min readLW link

A Visualization of Nick Bostrom’s Superintelligence

[deleted]23 Jul 2014 0:24 UTC

62 points

28 comments3 min readLW link

AI Alignment Research Overview (by Jacob Steinhardt)

Ben Pace6 Nov 2019 19:24 UTC

43 points

0 comments7 min readLW link

(docs.google.com)

A general model of safety-oriented AI development

Wei_Dai11 Jun 2018 21:00 UTC

65 points

8 comments1 min readLW link

Counterfactual Oracles = online supervised learning with random selection of training episodes

Wei_Dai10 Sep 2019 8:29 UTC

44 points

26 comments3 min readLW link

Siren worlds and the perils of over-optimised search

Stuart_Armstrong7 Apr 2014 11:00 UTC

73 points

417 comments7 min readLW link

Top 9+2 myths about AI risk

Stuart_Armstrong29 Jun 2015 20:41 UTC

65 points

46 comments2 min readLW link

Rohin Shah on reasons for AI optimism

abergal31 Oct 2019 12:10 UTC

40 points

58 comments1 min readLW link

(aiimpacts.org)

Plausibly, almost every powerful algorithm would be manipulative

Stuart_Armstrong6 Feb 2020 11:50 UTC

38 points

25 comments3 min readLW link

The Magnitude of His Own Folly

Eliezer Yudkowsky30 Sep 2008 11:31 UTC

74 points

128 comments6 min readLW link

AI alignment landscape

paulfchristiano13 Oct 2019 2:10 UTC

40 points

3 comments1 min readLW link

(ai-alignment.com)

Launched: Friendship is Optimal

iceman15 Nov 2012 4:57 UTC

72 points

32 comments1 min readLW link

Friendship is Optimal: A My Little Pony fanfic about an optimization process

iceman8 Sep 2012 6:16 UTC

109 points

152 comments1 min readLW link

Do Earths with slower economic growth have a better chance at FAI?

Eliezer Yudkowsky12 Jun 2013 19:54 UTC

54 points

176 comments4 min readLW link

Idea: Open Access AI Safety Journal

G Gordon Worley III23 Mar 2018 18:27 UTC

28 points

11 comments1 min readLW link

G.K. Chesterton On AI Risk

Scott Alexander1 Apr 2017 19:00 UTC

15 points

0 comments7 min readLW link

The Hidden Complexity of Wishes

Eliezer Yudkowsky24 Nov 2007 0:12 UTC

130 points

135 comments7 min readLW link

The Friendly AI Game

bentarm15 Mar 2011 16:45 UTC

50 points

178 comments1 min readLW link

Q&A with Jürgen Schmidhuber on risks from AI

XiXiDu15 Jun 2011 15:51 UTC

57 points

45 comments4 min readLW link

[Question] What should an Einstein-like figure in Machine Learning do?

Razied5 Aug 2020 23:52 UTC

3 points

3 comments1 min readLW link

Takeaways from safety by default interviews

AI Impacts and abergal

3 Apr 2020 17:20 UTC

23 points

2 comments13 min readLW link

(aiimpacts.org)

Field-Building and Deep Models

Ben Pace13 Jan 2018 21:16 UTC

21 points

12 comments4 min readLW link

Critique my Model: The EV of AGI to Selfish Individuals

ozziegooen8 Apr 2018 20:04 UTC

19 points

9 comments4 min readLW link

‘Dumb’ AI observes and manipulates controllers

Stuart_Armstrong13 Jan 2015 13:35 UTC

52 points

19 comments2 min readLW link

2019 AI Alignment Literature Review and Charity Comparison

Larks19 Dec 2019 3:00 UTC

130 points

18 comments62 min readLW link

Book review: Architects of Intelligence by Martin Ford (2018)

ofer11 Aug 2020 17:30 UTC

15 points

0 comments2 min readLW link

Qualitative Strategies of Friendliness

Eliezer Yudkowsky30 Aug 2008 2:12 UTC

29 points

56 comments12 min readLW link

Dreams of Friendliness

Eliezer Yudkowsky31 Aug 2008 1:20 UTC

31 points

81 comments9 min readLW link

Conceptual issues in AI safety: the paradigmatic gap

vedevazz24 Jun 2018 15:09 UTC

33 points

0 comments1 min readLW link

(www.foldl.me)

On unfixably unsafe AGI architectures

Steven Byrnes19 Feb 2020 21:16 UTC

30 points

8 comments5 min readLW link

A toy model of the treacherous turn

Stuart_Armstrong8 Jan 2016 12:58 UTC

36 points

13 comments6 min readLW link

Allegory On AI Risk, Game Theory, and Mithril

James_Miller13 Feb 2017 20:41 UTC

45 points

57 comments3 min readLW link

1hr talk: Intro to AGI safety

Steven Byrnes18 Jun 2019 21:41 UTC

34 points

4 comments24 min readLW link

The Evil AI Overlord List

Stuart_Armstrong20 Nov 2012 17:02 UTC

44 points

80 comments1 min readLW link

What I would like the SIAI to publish

XiXiDu1 Nov 2010 14:07 UTC

36 points

225 comments3 min readLW link

Evaluating the feasibility of SI’s plan

JoshuaFox10 Jan 2013 8:17 UTC

38 points

188 comments4 min readLW link

Q&A with experts on risks from AI #1

XiXiDu8 Jan 2012 11:46 UTC

45 points

67 comments9 min readLW link

Algo trading is a central example of AI risk

Vanessa Kosoy28 Jul 2018 20:31 UTC

27 points

5 comments1 min readLW link

Will the world’s elites navigate the creation of AI just fine?

lukeprog31 May 2013 18:49 UTC

36 points

266 comments2 min readLW link

Let’s talk about “Convergent Rationality”

capybaralet12 Jun 2019 21:53 UTC

36 points

33 comments6 min readLW link

Breaking Oracles: superrationality and acausal trade

Stuart_Armstrong25 Nov 2019 10:40 UTC

25 points

15 comments1 min readLW link

Q&A with Stan Franklin on risks from AI

XiXiDu11 Jun 2011 15:22 UTC

36 points

10 comments2 min readLW link

Muehlhauser-Goertzel Dialogue, Part 1

lukeprog16 Mar 2012 17:12 UTC

42 points

161 comments33 min readLW link

[LINK] NYT Article about Existential Risk from AI

[deleted]28 Jan 2013 10:37 UTC

38 points

23 comments1 min readLW link

Reframing the Problem of AI Progress

Wei_Dai12 Apr 2012 19:31 UTC

32 points

47 comments1 min readLW link

New AI risks research institute at Oxford University

lukeprog16 Nov 2011 18:52 UTC

36 points

10 comments1 min readLW link

Thoughts on the Feasibility of Prosaic AGI Alignment?

iamthouthouarti21 Aug 2020 23:25 UTC

8 points

10 comments1 min readLW link

Memes and Rational Decisions

inferential9 Jan 2015 6:42 UTC

35 points

17 comments10 min readLW link

Levels of AI Self-Improvement

avturchin29 Apr 2018 11:45 UTC

11 points

0 comments39 min readLW link

Optimising Society to Constrain Risk of War from an Artificial Superintelligence

JohnCDraper30 Apr 2020 10:47 UTC

3 points

1 comment51 min readLW link

Some Thoughts on Singularity Strategies

Wei_Dai13 Jul 2011 2:41 UTC

40 points

29 comments3 min readLW link

A trick for Safer GPT-N

Razied23 Aug 2020 0:39 UTC

7 points

1 comment2 min readLW link

against “AI risk”

Wei_Dai11 Apr 2012 22:46 UTC

35 points

91 comments1 min readLW link

“Smarter than us” is out!

Stuart_Armstrong25 Feb 2014 15:50 UTC

41 points

57 comments1 min readLW link

Analysing: Dangerous messages from future UFAI via Oracles

Stuart_Armstrong22 Nov 2019 14:17 UTC

22 points

16 comments4 min readLW link

Q&A with Abram Demski on risks from AI

XiXiDu17 Jan 2012 9:43 UTC

33 points

71 comments9 min readLW link

Q&A with experts on risks from AI #2

XiXiDu9 Jan 2012 19:40 UTC

22 points

29 comments7 min readLW link

AI Safety Discussion Day

Linda Linsefors15 Sep 2020 14:40 UTC

20 points

0 comments1 min readLW link

A long reply to Ben Garfinkel on Scrutinizing Classic AI Risk Arguments

Søren Elverlin27 Sep 2020 17:51 UTC

17 points

6 comments1 min readLW link

Online AI Safety Discussion Day

Linda Linsefors8 Oct 2020 12:11 UTC

5 points

0 comments1 min readLW link

Military AI as a Convergent Goal of Self-Improving AI

avturchin13 Nov 2017 12:17 UTC

5 points

3 comments1 min readLW link

Neural program synthesis is a dangerous technology

syllogism12 Jan 2018 16:19 UTC

9 points

6 comments2 min readLW link

New, Brief Popular-Level Introduction to AI Risks and Superintelligence

LyleN23 Jan 2015 15:43 UTC

33 points

3 comments1 min readLW link

FAI Research Constraints and AGI Side Effects

JustinShovelain3 Jun 2015 19:25 UTC

26 points

59 comments7 min readLW link

European Master’s Programs in Machine Learning, Artificial Intelligence, and related fields

Master Programs ML/AI14 Nov 2020 15:51 UTC

32 points

8 comments1 min readLW link

The mind-killer

Paul Crowley2 May 2009 16:49 UTC

29 points

160 comments2 min readLW link

[Question] Should I do it?

MrLight19 Nov 2020 1:08 UTC

−3 points

16 comments2 min readLW link

Rationalising humans: another mugging, but not Pascal’s

Stuart_Armstrong14 Nov 2017 15:46 UTC

7 points

1 comment3 min readLW link

Machine learning could be fundamentally unexplainable

George16 Dec 2020 13:32 UTC

26 points

15 comments15 min readLW link

(cerebralab.com)

[Question] What do you make of AGI:unaligned::spaceships:not enough food?

Ronny22 Feb 2020 14:14 UTC

4 points

3 comments1 min readLW link

Risk Map of AI Systems

VojtaKovarik and Jan_Kulveit

15 Dec 2020 9:16 UTC

25 points

3 comments8 min readLW link

Edge of the Cliff

akaTrickster5 Jan 2021 17:21 UTC

1 point

0 comments5 min readLW link

[Question] Does it become easier, or harder, for the world to coordinate around not building AGI as time goes on?

Eli Tyre29 Jul 2019 22:59 UTC

86 points

31 comments3 min readLW link 2 reviews

Grey Goo Requires AI

harsimony15 Jan 2021 4:45 UTC

8 points

11 comments4 min readLW link

(harsimony.wordpress.com)

AISU 2021

Linda Linsefors30 Jan 2021 17:40 UTC

28 points

2 comments1 min readLW link

Nonperson Predicates

Eliezer Yudkowsky27 Dec 2008 1:47 UTC

49 points

176 comments6 min readLW link

Engaging First Introductions to AI Risk

Rob Bensinger19 Aug 2013 6:26 UTC

31 points

21 comments3 min readLW link

Formal Solution to the Inner Alignment Problem

michaelcohen18 Feb 2021 14:51 UTC

46 points

123 comments2 min readLW link

[Question] What are the biggest current impacts of AI?

Sam Clarke7 Mar 2021 21:44 UTC

15 points

5 comments1 min readLW link

[Question] Is a Self-Iterating AGI Vulnerable to Thompson-style Trojans?

sxae25 Mar 2021 14:46 UTC

15 points

7 comments3 min readLW link

AI oracles on blockchain

Caravaggio6 Apr 2021 20:13 UTC

5 points

0 comments3 min readLW link

What if AGI is near?

Wulky Wilkinsen14 Apr 2021 0:05 UTC

11 points

5 comments1 min readLW link

[Question] Is there anything that can stop AGI development in the near term?

Wulky Wilkinsen22 Apr 2021 20:37 UTC

5 points

5 comments1 min readLW link

[Question] [timeboxed exercise] write me your model of AI human-existential safety and the alignment problems in 15 minutes

Quinn4 May 2021 19:10 UTC

6 points

2 comments1 min readLW link

AI Safety Research Project Ideas

Owain_Evans21 May 2021 13:39 UTC

58 points

2 comments3 min readLW link

Survey on AI existential risk scenarios

Sam Clarke, Alexis Carlier and Jonas Schuett

8 Jun 2021 17:12 UTC

60 points

11 comments7 min readLW link

[Question] What are some claims or opinions about multi-multi delegation you’ve seen in the memeplex that you think deserve scrutiny?

Quinn27 Jun 2021 17:44 UTC

16 points

6 comments2 min readLW link

Mauhn Releases AI Safety Documentation

Berg Severens3 Jul 2021 21:23 UTC

4 points

0 comments1 min readLW link

A gentle apocalypse

pchvykov16 Aug 2021 5:03 UTC

3 points

5 comments3 min readLW link

Could you have stopped Chernobyl?

Carlos Ramirez27 Aug 2021 1:48 UTC

28 points

17 comments8 min readLW link

The Governance Problem and the “Pretty Good” X-Risk

Zach Stein-Perlman29 Aug 2021 18:00 UTC

5 points

2 comments12 min readLW link

Distinguishing AI takeover scenarios

Sam Clarke and Sammy Martin

8 Sep 2021 16:19 UTC

62 points

11 comments14 min readLW link

The alignment problem in different capability regimes

Buck9 Sep 2021 19:46 UTC

87 points

12 comments5 min readLW link

How truthful is GPT-3? A benchmark for language models

Owain_Evans16 Sep 2021 10:09 UTC

54 points

24 comments6 min readLW link

Investigating AI Takeover Scenarios

Sammy Martin17 Sep 2021 18:47 UTC

27 points

1 comment27 min readLW link

AI takeoff story: a continuation of progress by other means

Edouard Harris27 Sep 2021 15:55 UTC

74 points

13 comments10 min readLW link

A brief review of the reasons multi-objective RL could be important in AI Safety Research

Ben Smith29 Sep 2021 17:09 UTC

27 points

7 comments10 min readLW link

The Dark Side of Cognition Hypothesis

Cameron Berg3 Oct 2021 20:10 UTC

19 points

1 comment16 min readLW link

Truthful AI: Developing and governing AI that does not lie

Owain_Evans, owencb and Lanrian

18 Oct 2021 18:37 UTC

81 points

9 comments10 min readLW link

AMA on Truthful AI: Owen Cotton-Barratt, Owain Evans & co-authors

Owain_Evans22 Oct 2021 16:23 UTC

31 points

15 comments1 min readLW link

Truthful and honest AI

abergal, Nick_Beckstead and Owain_Evans

29 Oct 2021 7:28 UTC

41 points

1 comment13 min readLW link

What is the most evil AI that we could build, today?

ThomasJ1 Nov 2021 19:58 UTC

−2 points

14 comments1 min readLW link

What are red flags for Neural Network suffering?

Marius Hobbhahn8 Nov 2021 12:51 UTC

26 points

15 comments12 min readLW link

Hardcode the AGI to need our approval indefinitely?

MichaelStJules11 Nov 2021 7:04 UTC

2 points

2 comments1 min readLW link

What would we do if alignment were futile?

Grant Demaree14 Nov 2021 8:09 UTC

73 points

43 comments3 min readLW link

Two Stupid AI Alignment Ideas

aphyer16 Nov 2021 16:13 UTC

24 points

3 comments4 min readLW link

Super intelligent AIs that don’t require alignment

Yair Halberstadt16 Nov 2021 19:55 UTC

10 points

2 comments6 min readLW link

AI Tracker: monitoring current and near-future risks from superscale models

Edouard Harris and Jeremie Harris

23 Nov 2021 19:16 UTC

62 points

13 comments3 min readLW link

(aitracker.org)

HIRING: Inform and shape a new project on AI safety at Partnership on AI

Madhulika Srikumar24 Nov 2021 8:27 UTC

6 points

0 comments1 min readLW link

How to measure FLOP/s for Neural Networks empirically?

Marius Hobbhahn29 Nov 2021 15:18 UTC

15 points

3 comments7 min readLW link

Modeling Failure Modes of High-Level Machine Intelligence

Ben Cottier, Daniel_Eth and Sammy Martin

6 Dec 2021 13:54 UTC

54 points

1 comment12 min readLW link

HIRING: Inform and shape a new project on AI safety at Partnership on AI

madhu_lika7 Dec 2021 19:37 UTC

1 point

0 comments1 min readLW link

Universality and the “Filter”

maggiehayes16 Dec 2021 0:47 UTC

10 points

3 comments11 min readLW link

Reviews of “Is power-seeking AI an existential risk?”

Joe Carlsmith16 Dec 2021 20:48 UTC

67 points

20 comments1 min readLW link

2+2: Ontological Framework

Lyrialtus1 Feb 2022 1:07 UTC

−15 points

2 comments15 min readLW link

Can the laws of physics/nature prevent hell?

superads916 Feb 2022 20:39 UTC

−7 points

10 comments2 min readLW link

How harmful are improvements in AI? + Poll

tilker and Marius Hobbhahn

15 Feb 2022 18:16 UTC

15 points

4 comments8 min readLW link

Preserving and continuing alignment research through a severe global catastrophe

A_donor6 Mar 2022 18:43 UTC

36 points

11 comments5 min readLW link

Ask AI companies about what they are doing for AI safety?

Michael Chen9 Mar 2022 15:14 UTC

50 points

0 comments2 min readLW link

Is There a Valley of Bad Civilizational Adequacy?

lbThingrb11 Mar 2022 19:49 UTC

13 points

1 comment2 min readLW link

[Question] Danger(s) of theorem-proving AI?

Yitz16 Mar 2022 2:47 UTC

7 points

9 comments1 min readLW link

It Looks Like You’re Trying To Take Over The World

gwern9 Mar 2022 16:35 UTC

376 points

124 comments1 min readLW link

(www.gwern.net)

We Are Conjecture, A New Alignment Research Startup

Connor Leahy8 Apr 2022 11:40 UTC

170 points

24 comments4 min readLW link

Clippy’s modest proposal

Daphne_W11 Apr 2022 21:00 UTC

5 points

13 comments9 min readLW link

Is technical AI alignment research a net positive?

cranberry_bear12 Apr 2022 13:07 UTC

4 points

2 comments2 min readLW link

The Peerless

carado13 Apr 2022 1:07 UTC

3 points

2 comments1 min readLW link

(carado.moe)

[Question] Can someone explain to me why MIRI is so pessimistic of our chances of survival?

iamthouthouarti14 Apr 2022 20:28 UTC

10 points

7 comments1 min readLW link

[Question] Convince me that humanity isn’t doomed by AGI

Yitz15 Apr 2022 17:26 UTC

59 points

53 comments1 min readLW link

Reflections on My Own Missing Mood

Conor Sullivan21 Apr 2022 16:19 UTC

51 points

25 comments5 min readLW link

Eight Short Studies On Excuses

Scott Alexander20 Apr 2010 23:01 UTC

645 points

244 comments10 min readLW link

Code Generation as an AI risk setting

Not Relevant17 Apr 2022 22:27 UTC

91 points

16 comments2 min readLW link

[Question] What is being improved in recursive self improvement?

Conor Sullivan25 Apr 2022 18:30 UTC

6 points

7 comments1 min readLW link

AI Alternative Futures: Scenario Mapping Artificial Intelligence Risk—Request for Participation (Edit)

Kakili27 Apr 2022 22:07 UTC

10 points

2 comments9 min readLW link

Video and Transcript of Presentation on Existential Risk from Power-Seeking AI

Joe Carlsmith8 May 2022 3:50 UTC

20 points

1 comment29 min readLW link

Interpretability’s Alignment-Solving Potential: Analysis of 7 Scenarios

Evan R. Murphy12 May 2022 20:01 UTC

41 points

0 comments59 min readLW link

Agency As a Natural Abstraction

Thane Ruthenis13 May 2022 18:02 UTC

41 points

8 comments13 min readLW link

[Link post] Promising Paths to Alignment—Connor Leahy | Talk

frances_lorenz14 May 2022 16:01 UTC

33 points

0 comments1 min readLW link

DeepMind’s generalist AI, Gato: A non-technical explainer

frances_lorenz, Nora Belrose and jonmenaster

16 May 2022 21:21 UTC

57 points

6 comments6 min readLW link

Actionable-guidance and roadmap recommendations for the NIST AI Risk Management Framework

Dan Hendrycks and Tony Barrett

17 May 2022 15:26 UTC

24 points

0 comments3 min readLW link

In defence of flailing

acylhalide18 Jun 2022 5:26 UTC

10 points

13 comments4 min readLW link

Why I’m Optimistic About Near-Term AI Risk

harsimony15 May 2022 23:05 UTC

47 points

28 comments1 min readLW link

Reshaping the AI Industry

Thane Ruthenis29 May 2022 22:54 UTC

133 points

34 comments21 min readLW link

Explaining inner alignment to myself

Jeremy Gillen24 May 2022 23:10 UTC

6 points

2 comments10 min readLW link

Right now, you’re sitting on a REDONKULOUS opportunity to help solve AGI (and rake in $$$)

Trevor126 May 2022 21:55 UTC

41 points

12 comments2 min readLW link

A Story of AI Risk: InstructGPT-N

peterbarnett26 May 2022 23:22 UTC

21 points

0 comments8 min readLW link

We will be around in 30 years

mukashi7 Jun 2022 3:47 UTC

13 points

205 comments2 min readLW link

Transformer Research Questions from Stained Glass Windows

StefanHex8 Jun 2022 12:38 UTC

4 points

0 comments2 min readLW link

Towards Gears-Level Understanding of Agency

Thane Ruthenis16 Jun 2022 22:00 UTC

7 points

4 comments18 min readLW link

A plausible story about AI risk.

DeLesley Hutchins10 Jun 2022 2:08 UTC

13 points

1 comment4 min readLW link

Summary of “AGI Ruin: A List of Lethalities”

Stephen McAleese10 Jun 2022 22:35 UTC

32 points

2 comments8 min readLW link

Poorly-Aimed Death Rays

Thane Ruthenis11 Jun 2022 18:29 UTC

39 points

5 comments4 min readLW link

Contra EY: Can AGI destroy us without trial & error?

Nikita Sokolsky13 Jun 2022 18:26 UTC

123 points

71 comments15 min readLW link

A Modest Pivotal Act

anonymousaisafety13 Jun 2022 19:24 UTC

−17 points

0 comments5 min readLW link

Alignment research for “meta” purposes

acylhalide16 Jun 2022 14:03 UTC

12 points

0 comments1 min readLW link

Specific problems with specific animal comparisons for AI policy

Trevor119 Jun 2022 1:27 UTC

3 points

1 comment2 min readLW link

[Question] AI misalignment risk from GPT-like systems?

fiso19 Jun 2022 17:35 UTC

10 points

8 comments1 min readLW link

Causal confusion as an argument against the scaling hypothesis

RobertKirk and David Krueger

20 Jun 2022 10:54 UTC

78 points

24 comments18 min readLW link

[Question] Is the study of AI an infohazard?

blackstampede20 Jun 2022 14:25 UTC

6 points

15 comments1 min readLW link

[LQ] Some Thoughts on Messaging Around AI Risk

𝕮𝖎𝖓𝖊𝖗𝖆25 Jun 2022 13:53 UTC

9 points

3 comments6 min readLW link

No comments.