TagLast edit: 3 May 2023 6:57 UTC by steven0461

A Superintelligence is a being with superhuman intelligence, and a focus of the Machine Intelligence Research Institute’s research. Specifically, Nick Bostrom (1997) defined it as

“An intellect that is much smarter than the best human brains in practically every field, including scientific creativity, general wisdom and social skills.”

The Machine Intelligence Research Institute is dedicated to ensuring humanity’s safety and prosperity by preparing for the development of an Artificial General Intelligence with superintelligence. Given its intelligence, it is likely to be incapable of being controlled by humanity. It is important to prepare early for the development of friendly artificial intelligence, as there may be an AI arms race. A strong superintelligence is a term describing a superintelligence which is not designed with the same architecture as the human brain.

An Artificial General Intelligence will have a number of advantages aiding it in becoming a superintelligence. It can improve the hardware it runs on and obtain better hardware. It will be capable of directly editing its own code. Depending on how easy its code is to modify, it might carry out software improvements that spark further improvements. Where a task can be accomplished in a repetitive way, a module preforming the task far more efficiently might be developed. Its motivations and preferences can be edited to be more consistent with each other. It will have an indefinite life span, be capable of reproducing, and transfer knowledge, skills, and code among its copies as well as cooperating and communicating with them better than humans do with each other.

The development of superintelligence from humans is another possibility, sometimes termed a weak superintelligence. It may come in the form of whole brain emulation, where a human brain is scanned and simulated on a computer. Many of the advantages a AGI has in developing superintelligence apply here as well. The development of Brain-computer interfaces may also lead to the creation of superintelligence. Biological enhancements such as genetic engineering and the use of nootropics could lead to superintelligence as well.

External Links

See Also

Su­per­in­tel­li­gence read­ing group

KatjaGrace31 Aug 2014 14:59 UTC
31 points
2 comments2 min readLW link

Su­per­in­tel­li­gent AGI in a box—a ques­tion.

Dmytry23 Feb 2012 18:48 UTC
16 points
77 comments2 min readLW link

“If we go ex­tinct due to mis­al­igned AI, at least na­ture will con­tinue, right? … right?”

plex18 May 2024 14:09 UTC
47 points
23 comments2 min readLW link

Su­per­in­tel­li­gence 7: De­ci­sive strate­gic advantage

KatjaGrace28 Oct 2014 1:01 UTC
19 points
60 comments6 min readLW link

In­ter­mit­tent Distil­la­tions #4: Semi­con­duc­tors, Eco­nomics, In­tel­li­gence, and Tech­nolog­i­cal Progress.

Mark Xu8 Jul 2021 22:14 UTC
81 points
9 comments10 min readLW link

Su­per­in­tel­li­gence via whole brain emulation

AlexMennen17 Aug 2016 4:11 UTC
15 points
33 comments3 min readLW link

En­tropic bound­ary con­di­tions to­wards safe ar­tifi­cial superintelligence

Santiago Nunez-Corrales20 Jul 2021 22:15 UTC
3 points
0 comments2 min readLW link

Su­per­in­tel­li­gence 5: Forms of Superintelligence

KatjaGrace14 Oct 2014 1:00 UTC
22 points
114 comments5 min readLW link

Pre­dic­tion: any un­con­trol­lable AI will turn earth into a gi­ant computer

Karl von Wendt17 Apr 2023 12:30 UTC
11 points
8 comments3 min readLW link

An Ap­peal to AI Su­per­in­tel­li­gence: Rea­sons to Pre­serve Humanity

James_Miller18 Mar 2023 16:22 UTC
30 points
72 comments12 min readLW link

Su­per­in­tel­li­gence Read­ing Group—Sec­tion 1: Past Devel­op­ments and Pre­sent Capabilities

KatjaGrace16 Sep 2014 1:00 UTC
43 points
233 comments7 min readLW link

Su­per­in­tel­li­gence FAQ

Scott Alexander20 Sep 2016 19:00 UTC
128 points
38 comments27 min readLW link

[Question] Op­ti­miz­ing for Agency?

Michael Soareverix14 Feb 2024 8:31 UTC
8 points
4 comments2 min readLW link

Thoughts for and against an ASI figur­ing out ethics for itself

sweenesm20 Feb 2024 23:40 UTC
6 points
10 comments3 min readLW link

After Over­mor­row: Scat­tered Mus­ings on the Im­me­di­ate Post-AGI World

Yuli_Ban24 Feb 2024 15:49 UTC
−3 points
0 comments26 min readLW link

We are headed into an ex­treme com­pute overhang

devrandom26 Apr 2024 21:38 UTC
45 points
26 comments2 min readLW link

Why is AGI/​ASI Inevitable?

DeathlessAmaranth2 May 2024 18:27 UTC
14 points
6 comments1 min readLW link

Open-ended ethics of phe­nom­ena (a desider­ata with uni­ver­sal moral­ity)

Ryo 8 Nov 2023 20:10 UTC
1 point
0 comments8 min readLW link

AGI: Hire Soft­ware Eng­ineers—All of Them, Right Now

MGow30 Mar 2023 18:40 UTC
−18 points
3 comments1 min readLW link

P(doom|su­per­in­tel­li­gence) or coin tosses and dice throws of hu­man val­ues (and other re­lated Ps).

Muyyd22 Apr 2023 10:06 UTC
−7 points
0 comments4 min readLW link

A con­cise sum-up of the ba­sic ar­gu­ment for AI doom

Mergimio H. Doefevmil24 Apr 2023 17:37 UTC
11 points
6 comments2 min readLW link

Free­dom Is All We Need

Leo Glisic27 Apr 2023 0:09 UTC
−1 points
8 comments10 min readLW link

​​ Open-ended/​Phenom­e­nal ​Ethics ​(TLDR)

Ryo 9 Nov 2023 16:58 UTC
3 points
0 comments1 min readLW link

Out of the Box

jesseduffield13 Nov 2023 23:43 UTC
5 points
1 comment7 min readLW link

LLMs May Find It Hard to FOOM

RogerDearnaley15 Nov 2023 2:52 UTC
11 points
30 comments12 min readLW link

A few Su­per­hu­man ex­am­ples of Su­per­al­igned Su­per­in­tel­li­gence from Google Bard (Thanks­giv­ing 2023)

23 Nov 2023 19:06 UTC
−9 points
1 comment17 min readLW link

Tak­ing Into Ac­count Sen­tient Non-Hu­mans in AI Am­bi­tious Value Learn­ing: Sen­tien­tist Co­her­ent Ex­trap­o­lated Volition

Adrià Moret2 Dec 2023 14:07 UTC
26 points
31 comments42 min readLW link

In­fer­ence Speed is Not Unbounded

OneManyNone8 May 2023 16:24 UTC
35 points
31 comments16 min readLW link

Rosko’s Wager

Wuksh16 May 2023 7:18 UTC
1 point
0 comments2 min readLW link

In­stru­men­tal Con­ver­gence to Com­plex­ity Preservation

Macro Flaneur13 Jul 2023 17:40 UTC
2 points
2 comments3 min readLW link

Ram­ble on STUFF: in­tel­li­gence, simu­la­tion, AI, doom, de­fault mode, the usual

Bill Benzon26 Aug 2023 15:49 UTC
5 points
0 comments4 min readLW link

Knowl­edge Base 2: The struc­ture and the method of building

iwis9 Oct 2023 11:53 UTC
2 points
4 comments8 min readLW link

Knowl­edge Base 4: Gen­eral applications

iwis16 Oct 2023 12:26 UTC
3 points
0 comments1 min readLW link

Knowl­edge Base 3: Shop­ping ad­vi­sor and other uses of knowl­edge base about products

iwis9 Oct 2023 11:53 UTC
0 points
0 comments4 min readLW link

Knowl­edge Base 8: The truth as an at­trac­tor in the in­for­ma­tion space

iwis25 Apr 2024 15:28 UTC
−10 points
0 comments2 min readLW link

Con­tain­ing the AI… In­side a Si­mu­lated Reality

HumaneAutomation31 Oct 2020 16:16 UTC
1 point
9 comments2 min readLW link

The Evolu­tion­ary Path­way from Biolog­i­cal to Digi­tal In­tel­li­gence: A Cos­mic Perspective

George3605 Sep 2023 17:47 UTC
−17 points
0 comments4 min readLW link

In­stru­men­tal Con­ver­gence and hu­man ex­tinc­tion.

Spiritus Dei2 Oct 2023 0:41 UTC
−10 points
3 comments7 min readLW link

Are we all mis­al­igned?

Mateusz Mazurkiewicz3 Jan 2021 2:42 UTC
11 points
0 comments5 min readLW link

Suffer­ing-Fo­cused Ethics in the In­finite Uni­verse. How can we re­deem our­selves if Mul­ti­verse Im­mor­tal­ity is real and sub­jec­tive death is im­pos­si­ble.

Szymon Kucharski24 Feb 2021 21:02 UTC
−2 points
4 comments70 min readLW link

Reflec­tion of Hier­ar­chi­cal Re­la­tion­ship via Nuanced Con­di­tion­ing of Game The­ory Ap­proach for AI Devel­op­ment and Utilization

Kyoung-cheol Kim4 Jun 2021 7:20 UTC
2 points
2 comments7 min readLW link

Is the ar­gu­ment that AI is an xrisk valid?

MACannon19 Jul 2021 13:20 UTC
5 points
61 comments1 min readLW link

[LINK] Wait But Why—The AI Revolu­tion Part 2

Adam Zerner4 Feb 2015 16:02 UTC
27 points
88 comments1 min readLW link

Ques­tion 1: Pre­dicted ar­chi­tec­ture of AGI learn­ing al­gorithm(s)

Cameron Berg10 Feb 2022 17:22 UTC
13 points
1 comment7 min readLW link

Why Copi­lot Ac­cel­er­ates Timelines

Michaël Trazzi26 Apr 2022 22:06 UTC
35 points
14 comments7 min readLW link

A Cri­tique of AI Align­ment Pessimism

ExCeph19 Jul 2022 2:28 UTC
9 points
1 comment9 min readLW link

Ar­tifi­cial in­tel­li­gence wireheading

Big Tony12 Aug 2022 3:06 UTC
5 points
2 comments1 min readLW link

[Question] Would a Misal­igned SSI Really Kill Us All?

DragonGod14 Sep 2022 12:15 UTC
6 points
7 comments6 min readLW link

[Question] Why Do Peo­ple Think Hu­mans Are Stupid?

DragonGod14 Sep 2022 13:55 UTC
22 points
41 comments3 min readLW link

[Question] Are Hu­man Brains Univer­sal?

DragonGod15 Sep 2022 15:15 UTC
16 points
28 comments5 min readLW link

Another prob­lem with AI con­fine­ment: or­di­nary CPUs can work as ra­dio transmitters

RomanS14 Oct 2022 8:28 UTC
35 points
1 comment1 min readLW link

AGI Im­pos­si­ble due to En­ergy Constrains

TheKlaus30 Nov 2022 18:48 UTC
−11 points
13 comments1 min readLW link

Limits of Superintelligence

Aleksei Petrenko13 Dec 2022 12:19 UTC
1 point
5 comments1 min readLW link

AGI doesn’t need un­der­stand­ing, in­ten­tion, or con­scious­ness in or­der to kill us, only intelligence

James Blaha20 Feb 2023 0:55 UTC
10 points
2 comments18 min readLW link

[Question] “Wide” vs “Tall” su­per­in­tel­li­gence

Templarrr19 Mar 2023 19:23 UTC
13 points
8 comments1 min readLW link

[Question] Are Speed Su­per­in­tel­li­gences Fea­si­ble for Modern ML Tech­niques?

DragonGod14 Sep 2022 12:59 UTC
9 points
7 comments1 min readLW link

Scal­able Over­sight and Weak-to-Strong Gen­er­al­iza­tion: Com­pat­i­ble ap­proaches to the same problem

16 Dec 2023 5:49 UTC
73 points
3 comments6 min readLW link

Nav­i­gat­ing the Nexus of AGI, Ethics, and Hu­man Sur­vival: A Math­e­mat­i­cal Inquiry

Kan Yuenyong29 Feb 2024 6:47 UTC
1 point
0 comments3 min readLW link

Why en­tropy means you might not have to worry as much about su­per­in­tel­li­gent AI

Ron J23 May 2024 3:52 UTC
−26 points
1 comment2 min readLW link

[Question] In­ves­ti­gat­ing Alter­na­tive Fu­tures: Hu­man and Su­per­in­tel­li­gence In­ter­ac­tion Scenarios

Hiroshi Yamakawa27 Dec 2023 18:19 UTC
−4 points
0 comments17 min readLW link

5. Mo­ral Value for Sen­tient An­i­mals? Alas, Not Yet

RogerDearnaley27 Dec 2023 6:42 UTC
35 points
41 comments23 min readLW link

In­ves­ti­gat­ing Alter­na­tive Fu­tures: Hu­man and Su­per­in­tel­li­gence In­ter­ac­tion Scenarios

Hiroshi Yamakawa3 Jan 2024 23:46 UTC
1 point
0 comments17 min readLW link

Si­mu­la­tions and Altruism

FateGrinder5 Apr 2024 13:38 UTC
1 point
0 comments21 min readLW link

Chap­ter 4: Know­ers Not Known

SashaWu6 Mar 2024 13:07 UTC
0 points
0 comments3 min readLW link

The ne­ces­sity of “Guardian AI” and two con­di­tions for its achievement

Proica26 May 2024 17:39 UTC
0 points
0 comments14 min readLW link

GPT as an “In­tel­li­gence Fork­lift.”

boazbarak19 May 2023 21:15 UTC
47 points
27 comments3 min readLW link

W2SG: Introduction

Maria Kapros10 Mar 2024 16:25 UTC
0 points
2 comments10 min readLW link

Up­date­less­ness doesn’t solve most problems

Martín Soto8 Feb 2024 17:30 UTC
125 points
43 comments12 min readLW link
No comments.