I think those “lines in the sand” are very artificial. That’s especially true about AGI, because the road to superintelligence goes not via human equivalence, but around it.
So at any point in time we have AI systems which are somewhat deficient compared to humans along some dimensions to be called “true AGI”, but also strongly superhuman along larger and larger number of dimensions. At the point in time when all important deficiencies compared to humans are gone and we can call a system “AGI” without reservations, it’s already wildly superhuman along many dimensions (including many capabilities related to biomedical research).
But also we expect continuous progress, we don’t expect saturation, so at any given point in time any given task remains easier to accomplish in the future. But that’s not a good reason to postpone, because we usually need the solution ASAP. People are dying now, more than a million each week, and the sooner we can start to meaningfully decrease this number, the better.
In any case, AIs need to get better at biomedical research in order to be helpful with this, and it takes time. I doubt there is a generic intelligence capability from which everything follows automatically and super rapidly. The direction is towards artificial research assistants, then to artificial researchers, then to very superhuman artificial researchers, but one still needs to push it for any given application field. (Of course, people prioritize AI research first, for obvious reasons, and that’s also where the most formidable existential safety challenges come from, because artificial AI researchers do mean straightforward non-saturating recursive self-improvement, so safety-wise we should talk about that aspect first. But it’s good that they are pushing towards research help in more applied areas too, when those applied areas are urgent. It grounds the whole thing in the right values and the right priorities to some extent. If it slows down the rush to superintelligence a bit, it might be a positive thing too. Although I don’t really expect a slowdown from that, I think AI practioners and AIs themselves will learn a lot from those “biomedical exercises”.)
I agree with a lot of what you’re saying, and it made me realize I left out some of my reasoning that’s maybe more central than I realized.
Namely, what is the rate-limiting step in getting improved outcomes for people, health-wise? I would say the limiter is regulatory, in ways I don’t see current or near term AI significantly altering. In other words, under OpenAI’s own claimed timelines, I wouldn’t expect AI-assisted health innovation to generate real world results before close-enough-to-AGI-to-be-really-dangerous gets developed. Of course we should be using AI to advance medicine faster as soon as we can do so. But I don’t see why we need a non-profit to fund that, when it will also be very profitable to the companies that will use it. Conversely, an additional $25B invested in making future AI safer doesn’t have a whole lot of other funders lining up to make it happen.
Yes, in this sense you are right. In many countries, regulatory barriers are all-important. Although, a good chunk of the world can start adopting fast (and medical tourism does exist).
I think the main body of OpenAI will be dealing with the key safety issues, not even the whole main body, but the “core group”. They have to, the key safety problems are of such nature they can’t be dealt from the outside, and the non-profit is “the outside” in this sense, they can only direct/advise/assent/review the plans, but they can’t do more than that, they just don’t know how to do it. We’ve got a glimpse of OpenAI current thinking on “core safety” from Jakub Pachocki during the latest livestream (that’s whom they now have instead of Ilya), it has sounded good modulo the main difficulty, and we don’t know if they are well prepared to address the main difficulty (maintaining invariant properties through accelerating recursive self-improvement provided by artificial AI researchers, so not letting those properties diverge and tightening the delta between what those properties ideally should be and what they are at the moment, making sure that not only the probability of big disaster per unit of time does not grow, but that it diminishes fast enough, so that the accumulated probability of big disaster remains moderate in the infinite limit).
The other big project led by the non-profit, the cybersecurity improvements, shows that the non-profit is ready to lead on externalities, on systemic safety problems downstream of AI development. They are better equipped to do that, they have connections across the industry, this requires a systemic action, a lot of coordination.
(I presume their biomedical project will also try to quietly (or not so quietly) include prevention of artificial pandemics, which is another big downstream safety externality of AI development. The non-profit is capable of driving that.)
But with the core safety of self-modifying, self-improving systems, one can’t split safety and capability, it has to be the same group of people, a group of leading AI researchers who need to be strongly mindful of existential safety, to have a correct approach of collaborating on that set of issues with AI systems, and to drive a take-off jointly with collaborating AI systems (I don’t know if OpenAI has a right group of people in this sense these days).
I think those “lines in the sand” are very artificial. That’s especially true about AGI, because the road to superintelligence goes not via human equivalence, but around it.
So at any point in time we have AI systems which are somewhat deficient compared to humans along some dimensions to be called “true AGI”, but also strongly superhuman along larger and larger number of dimensions. At the point in time when all important deficiencies compared to humans are gone and we can call a system “AGI” without reservations, it’s already wildly superhuman along many dimensions (including many capabilities related to biomedical research).
But also we expect continuous progress, we don’t expect saturation, so at any given point in time any given task remains easier to accomplish in the future. But that’s not a good reason to postpone, because we usually need the solution ASAP. People are dying now, more than a million each week, and the sooner we can start to meaningfully decrease this number, the better.
In any case, AIs need to get better at biomedical research in order to be helpful with this, and it takes time. I doubt there is a generic intelligence capability from which everything follows automatically and super rapidly. The direction is towards artificial research assistants, then to artificial researchers, then to very superhuman artificial researchers, but one still needs to push it for any given application field. (Of course, people prioritize AI research first, for obvious reasons, and that’s also where the most formidable existential safety challenges come from, because artificial AI researchers do mean straightforward non-saturating recursive self-improvement, so safety-wise we should talk about that aspect first. But it’s good that they are pushing towards research help in more applied areas too, when those applied areas are urgent. It grounds the whole thing in the right values and the right priorities to some extent. If it slows down the rush to superintelligence a bit, it might be a positive thing too. Although I don’t really expect a slowdown from that, I think AI practioners and AIs themselves will learn a lot from those “biomedical exercises”.)
I agree with a lot of what you’re saying, and it made me realize I left out some of my reasoning that’s maybe more central than I realized.
Namely, what is the rate-limiting step in getting improved outcomes for people, health-wise? I would say the limiter is regulatory, in ways I don’t see current or near term AI significantly altering. In other words, under OpenAI’s own claimed timelines, I wouldn’t expect AI-assisted health innovation to generate real world results before close-enough-to-AGI-to-be-really-dangerous gets developed. Of course we should be using AI to advance medicine faster as soon as we can do so. But I don’t see why we need a non-profit to fund that, when it will also be very profitable to the companies that will use it. Conversely, an additional $25B invested in making future AI safer doesn’t have a whole lot of other funders lining up to make it happen.
Yes, in this sense you are right. In many countries, regulatory barriers are all-important. Although, a good chunk of the world can start adopting fast (and medical tourism does exist).
I think the main body of OpenAI will be dealing with the key safety issues, not even the whole main body, but the “core group”. They have to, the key safety problems are of such nature they can’t be dealt from the outside, and the non-profit is “the outside” in this sense, they can only direct/advise/assent/review the plans, but they can’t do more than that, they just don’t know how to do it. We’ve got a glimpse of OpenAI current thinking on “core safety” from Jakub Pachocki during the latest livestream (that’s whom they now have instead of Ilya), it has sounded good modulo the main difficulty, and we don’t know if they are well prepared to address the main difficulty (maintaining invariant properties through accelerating recursive self-improvement provided by artificial AI researchers, so not letting those properties diverge and tightening the delta between what those properties ideally should be and what they are at the moment, making sure that not only the probability of big disaster per unit of time does not grow, but that it diminishes fast enough, so that the accumulated probability of big disaster remains moderate in the infinite limit).
The other big project led by the non-profit, the cybersecurity improvements, shows that the non-profit is ready to lead on externalities, on systemic safety problems downstream of AI development. They are better equipped to do that, they have connections across the industry, this requires a systemic action, a lot of coordination.
(I presume their biomedical project will also try to quietly (or not so quietly) include prevention of artificial pandemics, which is another big downstream safety externality of AI development. The non-profit is capable of driving that.)
But with the core safety of self-modifying, self-improving systems, one can’t split safety and capability, it has to be the same group of people, a group of leading AI researchers who need to be strongly mindful of existential safety, to have a correct approach of collaborating on that set of issues with AI systems, and to drive a take-off jointly with collaborating AI systems (I don’t know if OpenAI has a right group of people in this sense these days).