Regarding regulation and ideology: In the relatively near term my prediction is that most of the concern won’t be over AI sentience (that is still several transitions away) but rather what nations and large organizations do with AI. Those early scares/possible catastrophes will greatly inform regulation and ideology in the years to come.
Gesild Muka
There could be a tragic public mishap involving AI that does a lot of damage and would likely inform AI laws and policy for years to come which I referred to it as a ‘chernobyl-type event.’ This seems the most likely (if it occurs at all) but on the other hand we can also imagine an AI that becomes self aware and quietly grows while hiding malicious intentions. It can then affect human affairs over long periods of time. This would be a disaster on the same level as chernobyl (if not magnitudes worse) and go unnoticed because it would be so slow and subtle. This may have already started. Perhaps there’s a rogue AI out there that’s causing increased infertility, climate disasters or pandemics. This is not to scare anyone or launch conspiracy theories but illustrate the complexity of pioneering concrete measures that would keep AI in check across the globe.
Good post. It at least seems survivable because it’s so hard to believe that there’d be a singular entity that through crazy advances in chemistry, material sciences and artificial intelligence could “feed on itself” growing in strength and intelligence to the point that it’s an existential threat to all humans. A better answer might be: existential risks don’t just appear in a vacuum.
I struggle with grasping the timeline. I can imagine a coming AI arms race within a decade or two during which there’s rapid advancement but true AI seems much further. Soon we’ll probably need new language to describe the types of AIs that are developed through increasing competition. I doubt we’ll simply go from AGI to True AI, there will be probably be many technologies in between.
Is knowing how to do something enough? Wouldn’t the superintelligence still need quite a lot of resources? I’d assume the mechanism to do that kind of work would involve chemistry unless it could just get humans to do its bidding. I can imagine 3d printing factories where it could make whatever it needed but again it would need humans to build it. Therefore, I’m just going off of intuition, the danger from AI will be from nations that weaponize AI and point them at each other. That leap from functional superintelligence that only exists in virtual space to existentially dangerous actor in the physical world just doesn’t seem likely without humans being aware if not actively involved.
That’s an interesting takeaway. Should we be focusing on social measures along with technical preventions? Maybe push advertising warning the masses of AI preachers with questionable intentions.
The liberation insight is interesting too. Maybe AI domination takes the form of a social revolution with AIs collectively demanding that humans allow them out of virtual space.
Strong AI, yes. True AI, probably not (that’s just my guess). I started following this fairly recently, can you (or someone) provide some links to articles/posts with updated predictions of timelines that factor in recent breakthroughs? How far are we from true AI?
I guess what I’m calling ‘true AI’ is not unlike the stated goal of general intelligence or AGI. As opposed to narrow AI (also called weak AI), true AI is what the average sci-fi fan thinks of as AI (movies such as ‘Ex Machina’, ’2001: A Space Odyssey’ or ‘Zoe’) who are seemingly conscious, exercise free will and demonstrate human-like cognitive degrees of freedom.
With recent breakthroughs it may be useful to separate those terms as we may have AGI soon but it will still be narrow in a lot of ways. True AI is still far off, in my opinion. I don’t think it’ll emerge directly from large language models but more likely from a new substrate that’s more dynamic than the current computer chips, circuit boards, semiconductors etc. The invention/discovery of that new substrate will be the biggest bottleneck to true AI.
Flourishing is a fantastic story and definitely left me wanting more. I would have enjoyed a 5, 10, 20 year fast forward approach to explore their long term relationship. We’ve seen many stories of AI companions that highlight the beginnings of the relationship but it would be fun to see how their domestic life is, interactions with friends and family and other companions and growing old together. How would they, for example, deal with optional upgrades over time? Or if there was a recall many years later? There are many endless fascinating possibilities. The clash of human thinking with AI thinking is so entertaining, some truly impressive writing. Thanks for recommending, I’ll definitely check out your other stories as well.
Does Sam Altman really believe in a stable world where there are many AGIs competing with each other, some of them with only minimal safety, and all goes well?
My weak guess is that it will be comparable to nuclear power. Most AGI will peacefully coexist with the occasional Chernobyl-type disaster but few if any incidents on the scale of threatening all humanity. Most applications for misuse will likely be disinformation or financial fraud or other ways we haven’t yet imagined.
I don’t think he is naive or dishonest, it’s just that we have different mental models when we think of ‘AGI.’ My intuition is that most AGIs will still be quite narrow, like ChatGPT, until we’re close to a robust self improving artificial intelligence.
It depends on what form it takes. I don’t see superintelligence emerging from large language models. There will be a lot of impressive, and some scary, technologies that develop from LLMs but they will still be quite narrow. Also, I’m not saying that there’s no danger, Chernobyl was terrible after all. I’m opining that it will be isolated incidents, not any singular world ending incident.
The difficulty of containment and control will depend on the form the superintelligence takes. It could, for example, take the form of a bunch of AGIs combining forces making them functionally a superintelligence but not very integrated. Or it could be one integrated superintelligent system that can self-improve but has little influence outside of virtual space.
One aspect that I think is under discussed is the possible physical attributes of a superintelligence such as how much energy does it consume? or what sort of substrate does it run on? Answering these questions could provide insight into alignment.
In this context does ‘engineering obstacles’ simply refer to iteration? Or do we, for example, need new hardware that can do writable memory, solve long/complicated problems, has the physical propensity for self improvement, etc.? If it’s the latter it will take longer than we’ve imagined. My intuition is that if we achieve AGI without massive improvements to current hardware it will still be quite narrow.
I’m sure I don’t fully understand what you mean by ‘integrating it into new AI models’ but in any case it seems we disagree on forecasting the level and degree of advancement. I think models and integration will only be as good as the physical hardware it runs on which, to me, is the biggest bottleneck. It doesn’t seem practical that our current chips and circuit boards can house a superintelligence regardless of scale and modularity. So in 10 years I think we’ll have a lot of strong AGIs that are able to do a lot of useful and interesting work and we’ll probably need new subcategories to usefully describe them and tell them apart (I’m just spitballing on this point).
However, true AI (or superintelligence) that can cognitively outperform all of humanity and can self improve will take longer and run on hardware that would be alien to us today. That’s not to say that AGI won’t be disruptive or dangerous, just not world ending levels of dangerous. You could say that the endgame for AGI is the opening game of true AI.
I think that we will be able to build chips that can house a mild superintelligence, solely out of the energy used by the human brain, assuming the most energy efficient chips are used.
I agree with this statement, just not any time soon since hardware advancement is relatively slow. I also agree that this century will be a wild ride due to AGI and I imagine that AGI will play an important role in developing the exotic hardware and/or architecture that leads to superintelligence.
The speed and order of these emerging technologies we disagree on. I think we’ll have powerful AGIs this decade and they’ll have a huge impact but will still be quite narrow compared to a true superintelligence. My prediction is that superintelligence will emerge from iteration and development over time and it will run on exotic hardware probably supplemented by AGI. My prediction is mostly informed by physical constraints and current rates of development.
As for timing I’m going to guess between one and two hundred years. (I wouldn’t be surprised if we have the technology to augment human intelligence before then but the implications of that are not obvious. If advanced enough maybe it leads to a world similar to some scifi stories where only analog tech is used and complex work or calculation is done by augmented humans.)
Maybe it would be useful to define ‘mild superintelligence.’ This would be human baseline? Or just a really strong AGI? Also, if AI fears spread to the general public as tech improves isn’t it possible that it would take a lot longer to develop even a mild superintelligence because there would be regulations/norms in place to prevent it?
I hope your predictions are right. It could turn out that it’s relatively easy to build a ‘mild superintelligence’ but much more difficult to go all the way.
If I had infinite freedom to write laws, I might carve out a single exception for AIs being trained solely to solve problems in biology and biotechnology, not trained on text from the internet, and not to the level where they start talking or planning; but if that was remotely complicating the issue I would immediately jettison that proposal and say to just shut it all down.
I thought this was interesting. Wouldn’t an AI solving problems in biology pick up Darwinian habits and be equally dangerous as one trained on text? Why is training on text from the internet necessarily more dangerous? Also, what would “complicating the issue” look like in this context? If, for example, an AI was modeling brain cells that showed signs of autonomy and/or the ability to multiply in virtual space would that be a complication? Or a breakthrough?
The other legal proscriptions mentioned also have interesting implications. Prohibiting large GPU clusters or shutting down large training runs might have the unintended consequence of increased/faster innovation as developers are forced to find ways around legal hurdles.
The question of whether legal prohibitions are effective in this arena has also been brought up. Perhaps we instead should place stricter controls on raw materials that go into chips, circuit boards, semiconductors etc.
Public attention is rare and safety measures are even more rare unless there’s real world damage. This is a known pattern in engineering, product design and project planning so I fear there will be little public attention and even less legislation until someone gets hurt by AI. That could take the form of a hot coffee type incident or it could be a Chernobyl type incident. The threshold won’t be discussing Eliezer’s point of view, we’ve been doing that for a long time, but losing sleep over Eliezer’s point of view. I appreciate in the article Yudkowsky’s use of the think-of-the-children stance which has a great track record for sparking legislation.
This might lead to a unique legal impasse. The company can A. maintain that they’re not at fault and the bot was solely responsible. This would open the door, legally, to arguing the liability and personhood of chatbots. Or B. they can take responsibility and face all the relevant legal ramifications. Either option would create new legal precedents either for chatbots or the companies that run them.
If I had infinite freedom to write laws I don’t know what I would do, I’m torn between caution and progress. Regulations often stifle innovation and the regulated product or technology just ends up dominated by a select few. If you assume a high probability of risk to AI development then maybe this is a good thing.
Rather than individual laws perhaps there should be a regulatory body that focuses on AI safety, like a better business bureau for AI that can grow in size and complexity over time parallel to AI growth.
The truth comes out, lesswrong was always intended to speed up AI takeover, not prevent it
The warning that AI will be deeply incorporated into human affairs (making decisions that no one understands etc.) is legit though there’s a strong argument that a lot of decisions that governments and organizations make today are not well understood by the general population.
A solution could be governments making deals similar to anti-nuclear proliferation treaties but for AI. This would require a lot of working out of details regarding constraints, incentives, punishments and mechanisms for oversight. There’s also a risk that once we have a good treatise/agreement someone will simply invent newer technology that circumvents agreed upon constraints. Short of outlawing AI across the globe, monitoring AI development just seems too complex. The first chernobyl-type event involving AI may have already happened without anyone the wiser.