RSS

Re­cur­sive Self-Improvement

TagLast edit: 26 May 2023 0:53 UTC by Super AGI

Recursive self-improvement refers to the property of making improvements on one’s own ability of making self-improvements. It is an approach to Artificial General Intelligence that allows a system to make adjustments to its own functionality resulting in improved performance. The system could then feedback on itself with each cycle reaching ever higher levels of intelligence resulting in either a hard or soft AI takeoff.

An agent can self-improve and get a linear succession of improvements, however if it is able to improve its ability of making self-improvements, then each step will yield exponentially more improvements then the previous one.

Recursive self-improvement and AI takeoff

Recursively self-improving AI is considered to be the push behind the intelligence explosion. While any sufficiently intelligent AI will be able to improve itself, Seed AIs are specifically designed to use recursive self-improvement as their primary method of gaining intelligence. Architectures that had not been designed with this goal in mind, such as neural networks or large “hand-coded” projects like Cyc, would have a harder time self-improving.

Eliezer Yudkowsky argues that a recursively self-improvement AI seems likely to deliver a hard AI takeoff – a fast, abruptly, local increase in capability—since the exponential increase in intelligence would yield an exponential return in benefits and resources that would feed even more returns in the next step, and so on. In his view a soft takeoff scenario seems unlikely: “it should either flatline or blow up. You would need exactly the right law of diminishing returns to fly through the extremely narrow soft takeoff keyhole.”1.

Yudkowsky argues that there are several points which seem to support the hard takeoff scenario. Some of them are the fact that one improvement seems to lead the way to another, hardware overhang and the fact that sometimes- when navigating through problem space—one can find a succession of extremely easy to solve problems. These are all reasons for suddenly and abruptly increases in capability. On the other hand, Robin Hanson argues that there will be mostly a slow and gradual accumulation of improvements, without a sharp change.

Self-improvement in humans

The human species has made an enormous amount of progress since evolving around fifty thousand years ago. This is because we can pass on knowledge and infrastructure from previous generations. This is a type of self-improvement, but it is not recursive. If we never learned to modify our own brains, then we would eventually reach the point where making new discoveries required more knowledge than could be gained in a human lifetime. All human progress to date has been limited by the hardware we are born with, which is the same hardware Homo sapiens were born with fifty thousand years ago.

“True” recursive self-improvement will come when we discover how to drastically modify or augment our own brains in order to be more intelligent. This would lead us to more quickly being able to discover how to become even more intelligent.

Recursive self-improvement and Instrumental value

Nick Bostrom and Steve Omohundro have separately2 argued3 that despite the fact that values and intelligence are independent, any recursively self-improving intelligence would likely possess a common set of instrumental values which are useful for achieving any kind of goal. As a system’s intelligence continued modifying itself towards greater intelligence, it would be likely to adopt more of these behaviors.

Blog posts

See also

External links

Notes on notes on virtues

David Gross30 Dec 2020 17:47 UTC
71 points
11 comments11 min readLW link

Why all the fuss about re­cur­sive self-im­prove­ment?

So8res12 Jun 2022 20:53 UTC
158 points
62 comments7 min readLW link1 review

Towards a For­mal­i­sa­tion of Re­turns on Cog­ni­tive Rein­vest­ment (Part 1)

DragonGod4 Jun 2022 18:42 UTC
17 points
11 comments13 min readLW link

AGI sys­tems & hu­mans will both need to solve the al­ign­ment problem

Jeffrey Ladish24 Feb 2023 3:29 UTC
59 points
14 comments4 min readLW link

Notes on Gratitude

David Gross13 Jan 2021 20:37 UTC
11 points
0 comments19 min readLW link

AI Will Not Want to Self-Improve

petersalib16 May 2023 20:53 UTC
20 points
24 comments20 min readLW link

Nice in­tro video to RSI

Nathan Helm-Burger16 May 2023 18:48 UTC
12 points
0 comments1 min readLW link
(youtu.be)

AGI-Au­to­mated In­ter­pretabil­ity is Suicide

__RicG__10 May 2023 14:20 UTC
24 points
33 comments7 min readLW link

“text­books are all you need”

bhauth21 Jun 2023 17:06 UTC
66 points
18 comments2 min readLW link
(arxiv.org)

Con­tra An­ton 🏴‍☠️ on Kol­mogorov com­plex­ity and re­cur­sive self improvement

DaemonicSigil30 Jun 2023 5:15 UTC
25 points
12 comments2 min readLW link

Re­cur­sive Self-Improvement

Eliezer Yudkowsky1 Dec 2008 20:49 UTC
38 points
54 comments13 min readLW link

Re­cur­sively Self-Im­prov­ing Hu­man Intelligence

curiousepic17 Feb 2011 21:55 UTC
17 points
13 comments1 min readLW link

...Re­cur­sion, Magic

Eliezer Yudkowsky25 Nov 2008 9:10 UTC
27 points
28 comments5 min readLW link

Cas­cades, Cy­cles, In­sight...

Eliezer Yudkowsky24 Nov 2008 9:33 UTC
35 points
31 comments8 min readLW link

Self-im­prove­ment with­out self-modification

Stuart_Armstrong23 Jul 2015 9:59 UTC
7 points
5 comments1 min readLW link

Stable self-im­prove­ment as a re­search problem

paulfchristiano17 Nov 2014 17:51 UTC
8 points
7 comments7 min readLW link

Ngo and Yud­kowsky on AI ca­pa­bil­ity gains

18 Nov 2021 22:19 UTC
130 points
61 comments39 min readLW link1 review

Ex­am­ples of AI In­creas­ing AI Progress

TW12317 Jul 2022 20:06 UTC
107 points
14 comments1 min readLW link

[Question] Is “Re­cur­sive Self-Im­prove­ment” Rele­vant in the Deep Learn­ing Paradigm?

DragonGod6 Apr 2023 7:13 UTC
32 points
36 comments7 min readLW link

A Year of AI In­creas­ing AI Progress

TW12330 Dec 2022 2:09 UTC
148 points
3 comments2 min readLW link

Will Values and Com­pe­ti­tion De­cou­ple?

interstice28 Sep 2022 16:27 UTC
15 points
11 comments17 min readLW link

Squeez­ing foun­da­tions re­search as­sis­tance out of for­mal logic nar­row AI.

Donald Hobson8 Mar 2023 9:38 UTC
16 points
1 comment2 min readLW link

AGI will be made of het­ero­ge­neous com­po­nents, Trans­former and Selec­tive SSM blocks will be among them

Roman Leventov27 Dec 2023 14:51 UTC
33 points
9 comments4 min readLW link

The Hard In­tel­li­gence Hy­poth­e­sis and Its Bear­ing on Suc­ces­sion In­duced Foom

DragonGod31 May 2022 19:04 UTC
10 points
7 comments4 min readLW link

The al­ign­ment sta­bil­ity problem

Seth Herd26 Mar 2023 2:10 UTC
35 points
15 comments4 min readLW link

Why Re­cur­sive Self-Im­prove­ment Might Not Be the Ex­is­ten­tial Risk We Fear

Nassim_A24 Nov 2024 17:17 UTC
1 point
0 comments9 min readLW link

What pro­gram struc­tures en­able effi­cient in­duc­tion?

Daniel C5 Sep 2024 10:12 UTC
21 points
5 comments3 min readLW link

Align­ment Might Never Be Solved, By Hu­mans or AI

interstice7 Oct 2022 16:14 UTC
48 points
6 comments3 min readLW link

A Sim­ple The­ory Of Consciousness

SherlockHolmes8 Aug 2023 18:05 UTC
2 points
5 comments1 min readLW link
(peterholmes.medium.com)

Virtue ethics and why the ra­tio­nal­ist com­mu­nity might care about it.

David Gross22 Oct 2020 3:53 UTC
36 points
2 comments6 min readLW link

If Align­ment is Hard, then so is Self-Improvement

PavleMiha7 Apr 2023 0:08 UTC
21 points
20 comments1 min readLW link

Eric Sch­midt on re­cur­sive self-improvement

nikola5 Nov 2023 19:05 UTC
24 points
3 comments1 min readLW link
(www.youtube.com)

LLMs May Find It Hard to FOOM

RogerDearnaley15 Nov 2023 2:52 UTC
11 points
30 comments12 min readLW link

En­gelbart: In­suffi­ciently Recursive

Eliezer Yudkowsky26 Nov 2008 8:31 UTC
22 points
22 comments7 min readLW link

The AI Ex­plo­sion Might Never Happen

snewman19 Sep 2023 23:20 UTC
21 points
31 comments9 min readLW link

AI self-im­prove­ment is possible

bhauth23 May 2023 2:32 UTC
18 points
3 comments8 min readLW link

Con­crete vs Con­tex­tual values

whpearson2 Jun 2009 9:47 UTC
−1 points
32 comments3 min readLW link

[Question] What’s your view­point on the like­li­hood of GPT-5 be­ing able to au­tonomously cre­ate, train, and im­ple­ment an AI su­pe­rior to GPT-5?

Super AGI26 May 2023 1:43 UTC
7 points
15 comments1 min readLW link

Pro­posal: labs should pre­com­mit to paus­ing if an AI ar­gues for it­self to be improved

NickGabs2 Jun 2023 22:31 UTC
3 points
3 comments4 min readLW link

hu­man in­tel­li­gence may be al­ign­ment-limited

bhauth15 Jun 2023 22:32 UTC
16 points
3 comments2 min readLW link

[Question] Would you join the So­ciety of the Free & Easy?

David Gross10 Jul 2019 1:15 UTC
18 points
1 comment3 min readLW link

Do not miss the cut­off for im­mor­tal­ity! There is a prob­a­bil­ity that you will live for­ever as an im­mor­tal su­per­in­tel­li­gent be­ing and you can in­crease your odds by con­vinc­ing oth­ers to make achiev­ing the tech­nolog­i­cal sin­gu­lar­ity as quickly and safely as pos­si­ble the col­lec­tive goal/​pro­ject of all of hu­man­ity, Similar to “Fable of the Dragon-Tyrant.”

Oliver--Klozoff29 Jun 2023 3:45 UTC
1 point
0 comments28 min readLW link

notes on pri­ori­tiz­ing tasks & cog­ni­tion-threads

Emrik26 Nov 2024 0:28 UTC
3 points
1 comment4 min readLW link

ACI#4: Seed AI is the new Per­pet­ual Mo­tion Machine

Akira Pyinya8 Jul 2023 1:17 UTC
−7 points
0 comments6 min readLW link

Why Copi­lot Ac­cel­er­ates Timelines

Michaël Trazzi26 Apr 2022 22:06 UTC
35 points
14 comments7 min readLW link
No comments.