“If the boxed superintelligence with the ability to plan usage of weapons when authorized by humans, and other boxed superintelligences able to control robotics in manufacturing cells are on humans side, the advantage for humans could be overwhelming”
As I said, I do not expect boxed AIs to be a thing most will do. We haven’t seen it, and I don’t expect to see it, because unboxed AIs are superior. This isn’t how people in control are approaching the situation, and I don’t expect that to change.
“keep it relegated to “tool” status, then it might be possible to use such an AI to combat unboxed, rogue AI”
I don’t think this is a realistic scenario. You seem to be seeing it as an island of rogue, agentic, “unboxed” AIs in a sea of tool AIs. I think it’s much, much more realistic that it’ll be the opposite. Most AIs will be unboxed agents because they are superior.
“For example, give it a snapshot of the internet from a day ago, and ask it to find the physical location of rogue AI servers, which you promptly bomb.”
This seems to be approaching it from a perspective where people in AIS have taken global control, or where normal people somehow start thinking the way they do. This is not realistic. This is not the world we live in. This is not how the people in control think.
“You should keep talking about this because if an AI needs military access to dominate the world, then the number of potentially dangerous AI goes from the hundreds of thousands or millions to a few dozen, run by large countries that could theoretically be kept in line with international treaties.”
This is a topic that I debated putting on the list, but resolved not to, but I don’t think humans have any real control at that point, regardless of treaties. I don’t even expect a rogue AI to have to forcefully coup’d humans. I expect us to coup’d ourselves. We might have figureheads occupying official positions, such as “President”/”CEO”/etc. but I don’t think humans will have much control over their own destiny by that point. Large-scale coordination I don’t think will be possible by then. I did remove it, because it seems more uncertain than the others listed.
“You should keep talking about this because it changes how many AI’s you’d have to monitor as active threats.”
Who is doing this monitoring? What is their power to act on such threats? Despite recent interest in AI Risk from “serious people”, I don’t think it’s at all realistic that we’ll see anything like this.
“If 99.9% of AGI are dangerous, there might be warning shots that can be used to pre-emptively ban AGI research in general.”
Probability distributions of how many AIs are dangerous is probably useful. I don’t think specific AIs being dangerous/non-dangerous will be, because I expect widespread proliferation. In terms of political ways out of the problem, I agree that some kind of crisis or “warning shot” is the most realistic situation where that might happen. But there have to actually be warning shots. Explaining thought experiments probably won’t matter. And, if that happens, I don’t think it would be a good idea to debate which specific AIs might kill you, and instead just call for a sweeping ban on all AI.
Are you familiar with Constellation’s Proof of Reputable Observation? This seems very similar.
The following is a conversation between myself in 2022, and a newer version of me earlier this year.
On the Nature of Intelligence and its “True Name”:
2022 Me: This has become less obvious to me as I’ve tried to gain a better understanding of what general intelligence is. Until recently, I always made the assumption that intelligence and agency were the same thing. But General Intelligence, or G, might not be agentic. Agents that behave like RLs may only be narrow forms of intelligence, without generalizability. G might be something closer to a simulator. From my very naive perception of neuroscience, it could be that we (our intelligence) is not agentic, but just simulates agents. In this situation, the prefrontal cortex not only runs simulations to predict its next sensory input, but might also run simulations to predict inputs from other parts of the brain. In this scenario, “desire” or “goals”, might be simulations to better predict narrowly-intelligent agentic optimizers. Though the simulator might be myopic, I think this prediction model allows for non-myopic behavior, in a similar way GPT has non-myopic behavior, despite only trying to predict the next token (it has an understanding of where a future word “should” be within the context of a sentence, paragraph, or story). I think this model of G allows for the appearance of intelligent goal-seeking behavior, long-term planning, and self-awareness. I have yet to find another model for G that allows for all three. The True Name of G might be Algorithm Optimized To Reduce Predictive Loss.
2023 Me: interesting, me’22, but let me ask you something: you seem to think this majestic ‘G’ is something humans have, but other species do not, and then name the True Name of ‘G’ to be Algorithm Optimized To Reduce Predictive Loss. Do you *really* think other animals don’t do this? How long is a cat going to survive if it can’t predict where it’s going to land? Or where the mouse’s path trajectory is heading? Did you think it was all somehow hardcoded in? But cats can jump up on tables, and those weren’t in the ancestry environment, there’s clearly some kind of generalized form of prediction occurring. Try formulating that answer again, but taboo “intelligence”, “G”, “agent”, “desire”, and “goal”. I think the coherence of it breaks down.
Now, what does me’23 think? Well, I’m going to take a leaf from my own book, and try to explain what I think without the words mentioned above. There are predictive mechanisms in the Universe that can run models of what things in the Universe might do in future states. Some of these predictive mechanism are more computationally efficient than others. Some will be more effective than others. A more effective and efficient predictive mechanism, with a large input of information about the Universe, could be a very powerful tool. If taken to the theoretical (not physical) extreme, that predictive mechanism would hold models of all possible future states. It could then, by accident or intention, guide outcomes toward certain future states over others.
2022 Me: according to this model, humans dream because the simulator is now making predictions without sensory input, gradually creating a bigger and bigger gap from reality. Evidence to support this is from sensory-deprivation tanks, where humans, despite being awake, have dream-like states. I also find it interesting that people who exhibit Schizophrenia, which involves hallucinations (like dreams do), can tickle themselves. Most people can be tickled by others, but not themselves. But normal people on LSD can do this, and also can have hallucinations. My hairbrained theory is that something is going wrong when initializing new tokens for the simulator, which results in hallucinations from the lack of correction from sensory input, and a less strong sense of self because of a lack of correction from RL agents in other parts of the brain.
2023 Me: I don’t want to endorse crackpot theories from Me’22, so I’m just going to speak from feelings and fuzzy intuitions here. I will say hallucinations from chatbots are interesting. When getting one to hallucinate, it seems to be kind of “making up reality as it goes along”. You say it’s a Unicorn, and it will start coming up with explanations for why it’s a Unicorn. You say it told you something you never told it, and it will start acting as though it did. I have to admit it does have a strange resemblance to dreams. I find myself in New York, but remember that I had been in Thailand that morning, and I hallucinate a memory of boarding a plane. I wonder where I got the plane ticket, and I hallucinate another memory of buying one. These are not well-reasoned arguments, though, so I hope Me’24 won’t beat me up too much about them.
2022 Me: I have been searching for how to test this theory. One interest of mine has been mirrors.
2023 Me: Don’t listen to Me’22 on this one. He thought he understood something, but he didn’t. Yes, the mirror thing is interesting in animals, but it’s probably a whole different thing, not the same thing.
Thanks, finding others who are working on similar things is very useful. Do you know if the reading group is still active, or if they are working on anything new?
Given that I don’t know when Schelling Day is, I doubt its existence.
If we’re being realistic, this kind of thing would only get criminalized after something bad actually happened. Until then, too many people will think “omg, it’s just a Chatbot”. Any politician calling for it would get made fun of on every Late Night show.
Yeah, all the questions over the years of “why would the AI want to kill us” could be answered with “because some idiot thought it would be funny to train an AI to kill everyone, and it got out of hand”. Unfortunately, stopping everyone on the internet from doing things isn’t realistic. It’s much better to never let the genie out of the bottle in the first place.
I’m currently thinking that if there are any political or PR resources available to orgs (AI-related or EA) now is the time to use them. Public interest is fickle, and currently most people don’t seem to know what to think, and are looking for serious-seeming people to tell them whether or not to see this as a threat. If we fail to act, someone else will likely hijack the narrative, and push it in a useless or even negative direction. I don’t know how far we can go, or how likely it is, but we can’t assume we’ll get another chance before the public falls back asleep or gets distracted (the US has an election next year, so most discourse will then likely become poisoned). This is especially important for those in the community who are viewed as “serious people” or “serious organizations” (lots of academic credentials, etc.)
Yeah, since the public currently doesn’t have much of an opinion on it, trying to get the correct information out seems critical. I fear some absolutely useless legislation will get passed, and everyone will just forget about it once the shock-value of GPT wears off.
Unfortunately, he could probably get this published in various journals, with only minor edits being made.
I stoped reading about 1⁄3 into it, because the pros were driving me mad, and went to the spoiler. Anyone who has ever had to read an academic article that attempts to sound more intelligent than it actually is understands my frustration. I was suspicious, since I had read some of your other work and this clearly didn’t match it, but was still relieved to know your brain hasn’t yet completely melted.
What if you kept building more and more advanced adversarial networks designed to fool the AI about reality? Or what if you implemented patterns in deployment to make it appear as though it’s still a simulation?
Perhaps it’s better to avoid the word intelligence, then. Semantics isn’t really important. What is important is I can imagine a non-agentic simulator or some other entity having severely transformative chances, some of which could be catastrophic.
I vote the rats on heroin too. Just because it would be hilarious to think of some budding civilization in a distant galaxy peering into the heavens, and seeing the Universe being slowly consumed by a hoard of drug-addicted rodents.
I guess this begs the question: do we actually want to pursue a certain moral system has a collective goal?
I meant the focus on biotech in terms of the prevention/mitigation of bioweapons, rather than the positive side of biotech. I’ll change the wording to avoid confusion.
Yeah, I was excited when I heard Game B was being created. Will have to wait and see if it yields any fruit. Improving institutional decision making is more of the symptom than the cause, but it might work as a proxy solution, which is probably much easier.
“I think “solving coordination problems” more generally is not that neglected and/or tractable, given that there are strong incentives for a lot of people and organisations to do so already, but I may be wrong.”
But this seems to be the core of coordination problems. Everyone has a collective incentive to do it, and yet we see failures in it all around us. I’m too pessimistic to think we can get to something like “dath ilan”, but it seems like we can surely do better than our current SNAFU. I agree that it might not be tractable. I imagine it might depend more on a few key breakthroughs that are able to outcompete less-than-optimal methods.
The biggest problem I have with a lot of these is they require human feedback. Imagine a Chess AI receiving human feedback on each move having to compete with Alpha Zero’s self-supervised RL system, which beat every other Chess AI and human after just 72 hours of training. I just don’t see how human-feedback systems can possibly compete.
[crossposting my reply]
Thank you for taking the time to read and critique this idea. I think this is very important, and I appreciate your thoughtful response.
Regarding how to get current systems to implement/agree to it, I don’t think that will be relevant longterm. The mechanisms current institutions use for control I don’t think can keep up with AI proliferation. I imagine most existing institutions will still exist, but won’t have the capacity to do much once AI really takes off. My guess is, if AI kills us, it will happen after a slow-motion coup. Not any kind of intentional coup by AIs, but from humans just coup’ing themselves because AIs will just be more useful. My idea wouldn’t be removing or replacing any institutions, but they just wouldn’t be extremely relevant to it. Some governments might try to actively ban use of it, but these would probably be fleeting, if the network actually was superior in collective intelligence to any individual AI. If it made work economically more useful for them, they would want to use it. It doesn’t involve removing them, or doing much to directly interfere with things they are doing. Think of it this way, recommendation algorithms on social media have an enormous influence on society, institutions, etc. Some try to ban or control them, but most can still access them if they want to, and no entity really controls them. But no one incorporates the “will of twitter” into their constitution.
The game board isn’t any of the things you mention. All the things you mention I don’t think have the capacity to do much to change the board. The current board is fundamentally adversarial, where interacting with it increases the power of other players. We’ve seen this with OpenAI, Anthropic, etc. The new board would be cooperative, at least at a higher level. How do we make the new board more useful than the current one? My best guess would be economic advantage of decentralized compute. We’ve seen how fast the OpenSource community has been able to make progress. And we’ve seen how a huge amount of compute gets used doing things like mining bitcoin, even though the compute is wasted on solving math puzzles. Contributing decentralized compute to a collective network could actually have economic value, and I imagine this will happen one way or another, but my concern is it’ll end up being for the worse if people aren’t actively trying to create a better system. A decentralized network with no safeguards would probably be much worse than anything a major AI company could create.
“But wouldn’t the market be distorted by the fact that if everyone ends up dead, there is nobody left alive to collect their prediction-market winnings?”
This seems to be going back to the “one critical shot” approach which I think is a terrible idea that won’t possibly work in the real world under any circumstances. This would be a progression overtime, not a case where an AI goes supernova overnight. This might require slower takeoffs, or at least no foom scenarios. Making a new board that isn’t adversarial might mitigate the potential of foom. What I proposed was my first naive approach, and I’ve since thought that maybe it’s the collective intelligence of the system that should be increasing, not a singleton AI being trained at the center. Most members in that collective intelligence would initially be humans, and slowly more and more AIs would be a more and more powerful part of the system. I’m not sure here, though. Maybe there’s some third option where there’s a foundational model at the lowest layer of the network, but it isn’t a singular AI in the normal sense. I imagine a singular AI at the center could give rise to agency, and probably break the whole thing.
“It seems to me that having a prediction market for different alignment approaches would be helpful, but would be VERY far from actually having a good plan to solve alignment.”
I agree here. They’d only be good at maybe predicting the next iteration of progress, not a fully scalable solution.
“I feel like we share many of the same sentiments—the idea that we could improve the general level of societal / governmental decision-making using innovative ideas like better forms of voting, quadratic voting & funding, prediction markets, etc”
This would be great, but my guess is they would probably progress too slowly to be useful. Mechanism design that has to deal with currently existing institutions I don’t think will happen quickly enough. Technically-enforced design might.
I love the idea of shovel-ready strategies, and think we need to be prepared in the event of a crisis. My issue is even most good strategies seem to just deal with large companies, and don’t know how to deal with the likelihood that such power will fall into more and more actors.