The issue isn’t that you wouldn’t be aware of the possibility of a catastrophically dangerous AI emerging training, the issue is that you don’t know how to detect or stop it. Do you think that the measures you currently implement or will be able to feasibly implement in the future are likely to prevent this? If so, would you be willing to have a more extended debate on that point with an expert who believes the opposite? (I don’t have a specific one in mind, but would try to find one if you’d be interested.) If you’re not willing, why not?
Risks of internal deployment are something we are tracking and (as this blog shows) are actively working on. I don’t think it will be useful to debate an external expert that doesn’t know the details of our internal setup. However we are continuously working on mitigations, reporting (system cards, blogs) as well as collaborating with third parties
When we have something to publish we will do so. Generally our system cards and other publications contain evaluations of different aspects of safety and alignment by us and third parties. I expect that as capabilities grow and stakes of internal and external deployment are higher, we will continue and expand both our own evaluations and such collaborations.
(Not sure if you agree about this one?) You agree that if someone made an unaligned ASI while there is not also a comparably powerful aligned ASI, that would likely spell doom for humanity.
You agree that an unaligned ASI could emerge during research in general and in particular during large frontier training runs involving lots of compute.
You state that OpenAI is doing some activity related to checking whether or not that’s happening in their research.
(Not sure if you’re meaning to imply this or not?) You imply that those activities are somehow adequate to the task of preventing the creation of an unaligned ASI before the creation of an aligned ASI.
Do you agree with these bullet points? I would add this bullet point:
There’s no good public reason for anyone to believe that implication, whether in the form of a technical plan or analysis of why it’s feasible, or review from independent experts, or any discussion or debate on this point from you or anyone else at OpenAI.
I don’t really agree with any of these bulletpoints. I am not even sure we are on the same page of the definition of ASI and I don’t view “emerging” as a good way to describe training. I feel like we are getting into more fundamental disagreements which I covered to some extent here.
The issue isn’t that you wouldn’t be aware of the possibility of a catastrophically dangerous AI emerging training, the issue is that you don’t know how to detect or stop it. Do you think that the measures you currently implement or will be able to feasibly implement in the future are likely to prevent this? If so, would you be willing to have a more extended debate on that point with an expert who believes the opposite? (I don’t have a specific one in mind, but would try to find one if you’d be interested.) If you’re not willing, why not?
Risks of internal deployment are something we are tracking and (as this blog shows) are actively working on. I don’t think it will be useful to debate an external expert that doesn’t know the details of our internal setup. However we are continuously working on mitigations, reporting (system cards, blogs) as well as collaborating with third parties
Does that include review from independent experts on the risk of a catastrophic AI emerging? Who would that be?
When we have something to publish we will do so. Generally our system cards and other publications contain evaluations of different aspects of safety and alignment by us and third parties. I expect that as capabilities grow and stakes of internal and external deployment are higher, we will continue and expand both our own evaluations and such collaborations.
So to recap,
(Not sure if you agree about this one?) You agree that if someone made an unaligned ASI while there is not also a comparably powerful aligned ASI, that would likely spell doom for humanity.
You agree that an unaligned ASI could emerge during research in general and in particular during large frontier training runs involving lots of compute.
You state that OpenAI is doing some activity related to checking whether or not that’s happening in their research.
(Not sure if you’re meaning to imply this or not?) You imply that those activities are somehow adequate to the task of preventing the creation of an unaligned ASI before the creation of an aligned ASI.
Do you agree with these bullet points? I would add this bullet point:
There’s no good public reason for anyone to believe that implication, whether in the form of a technical plan or analysis of why it’s feasible, or review from independent experts, or any discussion or debate on this point from you or anyone else at OpenAI.
And I wonder what you think of it.
I don’t really agree with any of these bulletpoints. I am not even sure we are on the same page of the definition of ASI and I don’t view “emerging” as a good way to describe training. I feel like we are getting into more fundamental disagreements which I covered to some extent here.