I think this may be proving Raemon’s point that there are a wide range of concepts. I consider the lower amount of alignment connotation of independence gaining a feature, not a bug, since we can say things like ethical independence gaining AGI or aligned independence gaining AGI without it sounding like an oxymoron. Also, I am not sure superintelligence is required to gain independence, since it may be possible to just think longer than a human to gain independence without thinking faster. That said, if out-of-control superintelligence is the right concept you are trying to get across, then use that.
jrincayc
Karma: 33
Overwhelming superintelligence sounds like a useful term. A term I started using is independence gaining artificial general intelligence as the threshold for when we need to start being concerned about the AGI’s alignment. An AI program that is sufficiently intelligent to be able to gain independence, such as by creating a self-replicating computer capable of obtaining energy and other things needed to achieve goals without any further assistance from humans.
For example, an independence gaining AGI connected to today’s internet might complete intellectual tasks for money and then use the money to mail order printed circuit boards and other hardware. An independence gaining AGI with access to 1800s level technology might mine coal and build a steam engine to power a Babbage-like computer and then bootstrap to faster computing elements. An independence gaining AGI on Earth’s moon might be able to produce solar panels and CPUs from the elements in the moon’s crust, and produce an electromagnetic rail to launch probes off the moon. Of course, how smart the AGI has to be to gain independence is a function of what kind of hardware the AGI can get access to. An overwhelming superintelligence might be able to take over the planet with just access to a hardware random number generator and a high precision timer, but a computer controlling a factory could probably be less intelligent and still be able to gain independence.
One of the reasons I started using the term is because human level AGI is vague, and we don’t know if we should be concerned by a human level AGI. Also, to determine if something is human level, we need to specify human level in what? 1950s computers were superhuman at arithmetic, but not chess, so is a 1950s computer human level or not? It may be hard to determine of a given computer + software is capable of gaining independence, but it is a more exact definition than just human level AGI.
Is it possible to prevent AGI?
Hm, I don’t think I want the Human-Descended Ideal Agent and the AI-Descended Ideal Agent to be in complete agreement. I want them to be compatible, as in able to live in the same universe. I want the AI to not make humans go extinct, and be ethical in a way that the AI can explain to me and (in a non-manipulative way) convince me is ethical. But in some sense, I hope that AI can come up with something better than just what humans would want in a CEV way. (And what about the opinion of the other vertebrates and cephalopods on this planet, and the small furry creatures from Alpha Centauri?)
I don’t think it is okay to do unethical things for music, music is not that important, but I hope that the AIs are doing some things that are as incomprehensible and pointless to us as music would be to evolution (or a being that was purely maximizing genetic fitness).
As a slightly different point, I think that the Ideal Agent is somewhat path dependent, and I think there are multiple different Ideal Agents that I would consider ethical and I would be happy to share the same galaxy with.
My P(doom) is 100% - Ɛ to Ɛ: 100% - Ɛ, Humanity has already destroyed the solar system (and the AGI is rapidly expanding to the rest of the galaxy) on at least some of the quantum branches due to an independence gaining AGI accident. I am fairly certain this was possible by 2010, and possibly as soon as around 1985. Ɛ, at least some of the quantum branches where independence gaining AGI happens will have a sufficiently benign AGI that we survive the experience or we will actually restrict computers enough to stop accidentally creating AGI. It is worth noting that if AGI has a very high probability of killing people, what it looks like in a world with quantum branching is periodically there will be AI winters when computers and techniques approach being capable of AGI, because many of the “successful” uses of AI result in deadly AGI, and so we just don’t live long enough to observe those.
And if I am just talking with an average person, I say > 1%, and make the side comment that if a passenger plane had a 1% probability of crashing in the next flight, it would not take off.
Edit: And to really explain this would take a lot longer post, so I apologize for that. That said, if I had to choose, to prevent accidental AGI, I would be willing to restrict myself and everyone else to computers on the order of 512 KiB of RAM, 2 MiB of disk space as part of a comprehensive program to prevent existential risk.
Use computers as powerful as in 1985 or AI controls humans or ?
Thank you for this, I still read it periodically.
First of all, I do agree with you that why haven’t other civilizations created AGIs that have then spread far enough to reach Earth is a really interesting question as well, and I would be happy to see a discussion on that question.
For that question, I think you are missing a fourth possibility, AGI is almost always deadly, so on quantum branches where it develops anywhere in the light cone, no one observes it (at least not for long). So we don’t see other civilization’s AGI because we just are not alive on those quantum branches.
AGI Fermi Paradox
My first attempt to turn this into a paper: http://jjc.freeshell.org/writings/hardware_limits_for_agi_d1.pdf
For what it is worth, I did write a programming language over the course of about two years of my life ( https://github.com/jrincayc/rust_pr7rs/ ). I do agree that there are better and worse ways to spend time, and it is probably worth thinking about this. I think that when you “recoiled from the thought of actually analyzing whether there was anything better I could have been doing” is a good hint maybe writing a programming language wasn’t the best thing for you. I wish you good skill and good luck in finding ways to spend your time.
Sounds interesting.
For somewhat related reasons, I made a 2 column version of Rationality from AI to Zombies https://github.com/jrincayc/rationality-ai-zombies which is easier to print than the original version, and have printed multiple copies in the hope that some survive a global catastrophe.
Thoughts on Hardware limits to Prevent AGI?
I agree that the human brain is roughly at a local optimum. But think about what could be done just with adding a fiber optic connection between two brains (I think there are some ethical issues here so this is a thought experiment, not something I recommend). The two brains could be a kilometer apart, and the signal between them on the fiber optic link takes less time than a signal takes to get from one side to the other of a regular brain. So these two brains could think together (probably with some (a lot?) neural rewiring) as fast as a regular brain thinks individually. Repeat with some more brains.
Or imagine if myelination was under conscious control. If you need to learn a new language, demyelinate the right parts of the brain, learn the language quickly, and then remyelinate it.So I think even without changing things much neurons could be used in ways that provide faster thinking and faster learning.
As for energy efficiency, there is no reason that a superintelligence has to be limited to the approximately 20 watts that a human brain has access to. Gaming computers can have 1000 W power supplies, which is 50 times more power. I think 50 brains thinking together really quickly (as in the interbrain connections are as fast as the intrabrain connections) could probably out-think a lot more than 50 humans.
And, today, there are supercomputers that use 20 or more megawatts of power, so if we have computing that is as energy efficient as the human brain, that is equivalent to 1 million human brains (20e6/20), and I think we might be able to agree that a million brains thinking together really well could probably out-think all of humanity.
Hm, neuron impulses travel at around 200 m/s, electric signals travel at around 2e8 m/s, so I think electronics have an advantage there. (I agree that you may have a point with “That Alien Mindspace”.)
Warren’s full speech is available at archive.org: “Unfortunately, however, many of our people and some of our authorities and, I am afraid, many of our people in other parts of the country are of the opinion that because we have had no sabotage and no fifth column activities in this State since the beginning of the war, that means that none have been planned for us. But I take the view that that is the most ominous sign in our whole situation. It convinces me more than perhaps any other factor that the sabotage that we are to get, the fifth column activities that we are to get, are timed just like Pearl Harbor was timed and just like the invasion of France, and of Denmark, and of Norway, and all those other countries.” Hon. Earl Warren, pg 11011-11012, San Francisco Hearings, February 21 and 23, 1942, part 29, National Defense Migration https://archive.org/details/nationaldefensem29unit
At least to me Claude 4.5 definitely reminds me of the description of Agent-1 in ai-2027 for Early 2026: “On the other hand, Agent-1 is bad at even simple long-horizon tasks, like beating video games it hasn’t played before. Still, the common workday is eight hours, and a day’s work can usually be separated into smaller chunks; you could think of Agent-1 as a scatterbrained employee who thrives under careful management. Savvy people find ways to automate routine parts of their jobs.”