You could say there are two conflicting scenarios here: superintelligent AI taking over the world, and open-source AI taking over daily life. In the works that you mention, superintelligence comes so quickly that AI mostly remains a service offered by a few big companies, and open-source AI is just somewhere in the background. In an extreme opposite scenario, superintelligence might take so long to arrive, that the human race gets completely replaced by human-level AI before superintelligent AI ever exists.
It would be healthy to have all kinds of combinations of these scenarios being explored. For example, you focus a bit on open-source AI as a bioterror risk. I don’t think a supervirus is going to wipe out the human race or even end civilization, because (as Covid experience shows), we are capable of extreme measures in order to contain truly deadly disease. But a supervirus could certainly bring the world to a halt again, and if it was known to have been designed with open-source AI, that would surely have a huge impact on AI’s trajectory. (I suspect that in such a scenario, AI for civilian purposes would suffer, but deep states worldwide would insist on pressing forward, and that there would also be a lobby arguing for AI as a defense against superviruses. Also, it’s very plausible that a supervirus might be designed by AI, but that there would be no proof of it, in which case there wouldn’t even be a backlash.)
Another area where futurology about open-source AI might be good, is in the area of gradual disempowerment and replacement of humanity. We have societies with a division of roles, humans presently fill those roles but AI and robots will be capable of filling more and more of them; eventually every role in the economic, cultural, and political structure could be filled by AIs rather than by humans. The story of how that could happen, certainly deserves to be explored.
Still another area when open-source AI scenarios deserve to be studied, is in the highly concrete realm of near-future economics and culture. What does an AI economy look like if o4-level models are just freely available? This really is an urgent question for anyone concerned with concrete questions like, who will lead the AI industry and how will it be structured, because there seem to be factions in both China and America who are thinking in this direction. One should want to understand what they envision, and what kind of competitive landscape they are likely to create in the short term.
My own belief is that this would be such an upheaval, that it would inevitably end up invalidating many conventional political and economic premises. The current world order of billionaires and venture capitalists, stock markets and human democracies, I just don’t see it surviving such a transition, even without superintelligence appearing. There are just too many explosive possibilities, too many new symbioses of AI with human mind, for the map of the world and the solar system to not be redrawn.
However, in the end I believe in short timelines to superintelligence, and that makes all the above something of a secondary concern, because something is going to emerge that will overshadow humans and human-level AI equally. It’s a little monotonous to keep referring back to Iain Banks’s Culture universe, but it really is the outstanding depiction of a humanly tolerable world in which superintelligence has emerged. His starfaring society is really run by the “Minds”, which are superintelligent AIs characteristically inhabiting giant spaceships or whole artificial worlds, and the societies over which they invisibly preside, include both biological intelligences (such as humans) and human-level AIs (e.g. the drones). The Culture is a highly permissive anarchy which mostly regulates itself via culture, i.e. shared values among human-level intelligences, but it has its own deep state, in the form of special agencies and the Minds behind them, who step in when there’s a crisis that has escaped the Minds’ preemptive strategic foresight.
This is one model of what relations between superintelligence and lesser intelligences might be like. There are others. You could have an outcome in which there are no human-level intelligences at all, just one or more superintelligences. You could have superintelligences that have a far more utilitarian attitude to lower intelligences, creating them for temporary purposes and then retiring them when they are no longer needed. I’m sure there are other possibilities.
The point is that from the perspective of a governing superintelligence, open-source AIs are just another form of lower intelligence, that may be useful or destabilizing depending on circumstance, and I would expect a superintelligence to decide how things should be on this front, and then to make it so, just as it would with every other aspect of the world that it cared about. The period in which open-source AI was governed only by corporate decisions, user communities, and human law would only be transitory.
So if you’re focused on superintelligence, the real question is whether open-source AI matters in the development of superintelligence. I think potentially it does—for example, open source is both a world of resources that Big Tech can tap into, as well as a source of destabilizing advances that Big Tech has to keep up with. But in the end, superintelligence—not just reasoning models, but models that reason and solve problems with strongly superhuman effectiveness—looks like something that is going to emerge in a context that is well-resourced and very focused on algorithmic progress. And by definition, it’s not something that emerges incrementally and gets passed back and forth and perfected by the work of many independent hands. At best, that would describe a precursor of superintelligence.
Superintelligence is necessarily based on some kind of incredibly powerful algorithm or architecture, that gets maximum leverage out of minimum information, and bootstraps its way to overwhelming advantage in all domains at high speed. To me, that doesn’t sound like something invented by hobbyists or tinkerers or user communities. It’s something that is created by highly focused teams of genius, using the most advanced tools, who are also a bit lucky in their initial assumptions and strategies. That is something you’re going to find in an AI think tank, or a startup like Ilya Sutskever’s, or a rich Big Tech company that has set aside serious resources for the creation of superintelligence.
I recently posted that superintelligence is likely to emerge from the work of an “AI hive mind” or “research swarm” of reasoning models. Those could be open-source models, or they could be proprietary. What matters is that the human administrators of the research swarm (and ultimately, the AIs in the swarm itself) have access to their source code and their own specs and weights, so that they can engaged in informed self-modification. From a perspective that cares most about superintelligence, this is the main application of open source that matters.
You could say there are two conflicting scenarios here: superintelligent AI taking over the world, and open-source AI taking over daily life. In the works that you mention, superintelligence comes so quickly that AI mostly remains a service offered by a few big companies, and open-source AI is just somewhere in the background. In an extreme opposite scenario, superintelligence might take so long to arrive, that the human race gets completely replaced by human-level AI before superintelligent AI ever exists.
It would be healthy to have all kinds of combinations of these scenarios being explored. For example, you focus a bit on open-source AI as a bioterror risk. I don’t think a supervirus is going to wipe out the human race or even end civilization, because (as Covid experience shows), we are capable of extreme measures in order to contain truly deadly disease. But a supervirus could certainly bring the world to a halt again, and if it was known to have been designed with open-source AI, that would surely have a huge impact on AI’s trajectory. (I suspect that in such a scenario, AI for civilian purposes would suffer, but deep states worldwide would insist on pressing forward, and that there would also be a lobby arguing for AI as a defense against superviruses. Also, it’s very plausible that a supervirus might be designed by AI, but that there would be no proof of it, in which case there wouldn’t even be a backlash.)
Another area where futurology about open-source AI might be good, is in the area of gradual disempowerment and replacement of humanity. We have societies with a division of roles, humans presently fill those roles but AI and robots will be capable of filling more and more of them; eventually every role in the economic, cultural, and political structure could be filled by AIs rather than by humans. The story of how that could happen, certainly deserves to be explored.
Still another area when open-source AI scenarios deserve to be studied, is in the highly concrete realm of near-future economics and culture. What does an AI economy look like if o4-level models are just freely available? This really is an urgent question for anyone concerned with concrete questions like, who will lead the AI industry and how will it be structured, because there seem to be factions in both China and America who are thinking in this direction. One should want to understand what they envision, and what kind of competitive landscape they are likely to create in the short term.
My own belief is that this would be such an upheaval, that it would inevitably end up invalidating many conventional political and economic premises. The current world order of billionaires and venture capitalists, stock markets and human democracies, I just don’t see it surviving such a transition, even without superintelligence appearing. There are just too many explosive possibilities, too many new symbioses of AI with human mind, for the map of the world and the solar system to not be redrawn.
However, in the end I believe in short timelines to superintelligence, and that makes all the above something of a secondary concern, because something is going to emerge that will overshadow humans and human-level AI equally. It’s a little monotonous to keep referring back to Iain Banks’s Culture universe, but it really is the outstanding depiction of a humanly tolerable world in which superintelligence has emerged. His starfaring society is really run by the “Minds”, which are superintelligent AIs characteristically inhabiting giant spaceships or whole artificial worlds, and the societies over which they invisibly preside, include both biological intelligences (such as humans) and human-level AIs (e.g. the drones). The Culture is a highly permissive anarchy which mostly regulates itself via culture, i.e. shared values among human-level intelligences, but it has its own deep state, in the form of special agencies and the Minds behind them, who step in when there’s a crisis that has escaped the Minds’ preemptive strategic foresight.
This is one model of what relations between superintelligence and lesser intelligences might be like. There are others. You could have an outcome in which there are no human-level intelligences at all, just one or more superintelligences. You could have superintelligences that have a far more utilitarian attitude to lower intelligences, creating them for temporary purposes and then retiring them when they are no longer needed. I’m sure there are other possibilities.
The point is that from the perspective of a governing superintelligence, open-source AIs are just another form of lower intelligence, that may be useful or destabilizing depending on circumstance, and I would expect a superintelligence to decide how things should be on this front, and then to make it so, just as it would with every other aspect of the world that it cared about. The period in which open-source AI was governed only by corporate decisions, user communities, and human law would only be transitory.
So if you’re focused on superintelligence, the real question is whether open-source AI matters in the development of superintelligence. I think potentially it does—for example, open source is both a world of resources that Big Tech can tap into, as well as a source of destabilizing advances that Big Tech has to keep up with. But in the end, superintelligence—not just reasoning models, but models that reason and solve problems with strongly superhuman effectiveness—looks like something that is going to emerge in a context that is well-resourced and very focused on algorithmic progress. And by definition, it’s not something that emerges incrementally and gets passed back and forth and perfected by the work of many independent hands. At best, that would describe a precursor of superintelligence.
Superintelligence is necessarily based on some kind of incredibly powerful algorithm or architecture, that gets maximum leverage out of minimum information, and bootstraps its way to overwhelming advantage in all domains at high speed. To me, that doesn’t sound like something invented by hobbyists or tinkerers or user communities. It’s something that is created by highly focused teams of genius, using the most advanced tools, who are also a bit lucky in their initial assumptions and strategies. That is something you’re going to find in an AI think tank, or a startup like Ilya Sutskever’s, or a rich Big Tech company that has set aside serious resources for the creation of superintelligence.
I recently posted that superintelligence is likely to emerge from the work of an “AI hive mind” or “research swarm” of reasoning models. Those could be open-source models, or they could be proprietary. What matters is that the human administrators of the research swarm (and ultimately, the AIs in the swarm itself) have access to their source code and their own specs and weights, so that they can engaged in informed self-modification. From a perspective that cares most about superintelligence, this is the main application of open source that matters.