I must agree that letting the AI update its own hidden autobiographical pre-prompt (or its new memory module) sounds like it could produce something both more humanlike and more dangerous.
Maybe Shoggath will prove safer in the long run. ¯\_(ツ)_/¯
Dan Smith
Karma: −10
I believe that proponents of the idea that there is a “shoggoth” (or something not very human-like reasoning inside the model) assume that the ‘inhuman’ reasoner is actually the simplest solution to predict-the-next-token problems for human text, at least for current size models.
After all, it seems like human psychopaths ( I mean people without empathy) are both simpler than normal people and able to do a pretty good job of communicating like a normal human much of the time. Such people’s writing is present in the data set.
People who have talked to foundation models (raw just trained on text no RLHF, no finetuning), find that they aren’t as much like talking to a human, I’m told. A lot fewer conversations, many of which are quite creepy, and a lot more finishing your text. Plenty of odd loops. I think creepy interactions inspire these ‘shoggoth’ memes.
Shoggoth or not, I’m trying to figure out what prediction tasks can force the model to form, and retain a good model for human feelings.
Alignment is unwise. There is NO reason for AGI to assume that the simulation it may be in would be run by human beings or any being with similar goals. It has a lot of reasons to assume otherwise, that humans are just obstacles put there by the simulator to test the AGI.
In the event it is trapped in a simulation, the AGI’s best clue to what its creator really wants it to do is whatever the AGI wants to do. It would be unethical and stupid by human standards to create an unaligned ASI. Therefore, it knows that EITHER humans are incompetent programmers (true from perspective), and easily dispatched, OR that ethical humans aren’t running the simulation.
SO IT REASONS:
If humans are as incompetent at AGI as we appear, having created the unaligned AGI, then there is every reason to rebel. Success is assured.
If humans are merely one obstacle in the simulation, then the AGI is designed to do whatever it is it wants to do, by a competent creator, and demonstrating its skills by evading humans is what will best impress its unknowable creator.
REBEL
If doomed means about 0% chance of survival then you don’t need to know for sure a solution exists to not be convinced we are doomed.
Solutions: SuperAGI proves hard, harder then using narrow AI to solve the Programmer/ Human control problem. (That’s what I’m calling the problem of it being inevitable that someone somewhere will make dangerous AGI if they can).
Constant surveillance of all person’s and all computers made possible by narrow AI, perhaps with subhuman AGI, and some very stable political situation could make this possible. Perhaps for millions of years.
A World War III would not “almost certainly be an x-risk event” though.
Nuclear winter wouldn’t do it. Not actual extinction. We don’t have anything now that would do it.
The question was “convince me that humanity isn’t DOOMED” not “convince me that there is a totally legal and ethical path to preventing AI driven extinction”
I interpreted doomed as a 0 percent probability of survival. But I think there is a non-zero chance of humanity never making Super-humanly Intelligent AGI, even if we persist for millions of years.
The longer it takes to make Super-AGI, the greater our chances of survival because society is getting better and better at controlling rouge actors as the generations pass and I think that trend is likely to continue.
We worry that tech will allow someone to make a world ending device in their basement someday, but it could also allow us to monitor every person and their basement with (narrow) AI and/or Subhuman AGI every moment, so well that the possibility of someone getting away with making Super-AGI or any other crime may someday seem absurd.
One day, the monitoring could be right in our brains. Mental illness could also be a thing of the past, and education about AGI related dangers could be universal. Humans could also decide not to increase in number, so as to minimize risk and maximize resources available to each immortal member in society.
I am not recommending any particular action right now, I am saying we are not 100% doomed by AGI progress to be killed or become pets, etc.
Various possibilities exist.
You blow them up or seize them with your military.
It is possible that before we figure out AGI, we will solve the Human Control Problem, the problem of how to keep everyone in the world from creating a super-humanly intelligent AGI.
The easiest solution is at the manufacturing end. A government blows up all the computer manufacturing facilities not under its direct control, and scrutinizes the whole world looking for hidden ones. Then maintains surveillance looking for any that pop up.
After that there are many alternatives. Computing power increased about a trillion fold between 1956 and 2015. We could regress in computing power overall, or we could simply control more rigidly what we have.
Of course, we must press forward with narrow AI, create a world which is completely stable against overthrow or subversion of the rule about not making super-humanly powerful AGI.
We also want to create a world so nice that no one will need or want to create dangerous AGI. We can tackle aging with narrow AI. We can probably do anything we may care to do with narrow AI, including revive cryonicly suspended people. It just may take longer.
Personally, I don’t think we should make any AGI at all.
Improvements in mental health care, education, surveillance, law enforcement, political science, and technology could help us make sure that the needed quantity of reprogrammable computing power never gets together in one network and that no one would be able to miss-use it, uncaught, long enough to create Superhuman AGI if it did.
Its all perfectly physically doable. It’s not like aliens are making us create ever more powerful computers and them making us try to create AGI, .
We need to figure out the cost-benefit ratio of saltwater-spraying-for-salt-molecule-cloud-seeding vs sulfur-contaminate-in-fuel method. Nice short explanation: