[First of all, many thanks for writing the post; it seems both useful and the kind of thing that’ll predictably attract criticism]
I’m not quite sure what you mean to imply here (please correct me if my impression is inaccurate—I’m describing how-it-looks-to-me, and I may well be wrong):
I would expect OpenAI leadership to put more weight on experimental evidence than you...
Specifically, John’s model (and mine) has: X = [Class of high-stakes problems on which we’ll get experimental evidence before it’s too late] Y = [Class of high-stakes problems on which we’ll get no experimental evidence before it’s too late]
Unless we expect Y to be empty, when we’re talking about Y-problems the weighting is irrelevant: we get no experimental evidence.
Weighting of evidence is an issue when dealing with a fixed problem. It seems here as if it’s being used to select the problem: we’re going to focus on X-problems because we put a lot of weight on experimental evidence. (obviously silly, so I don’t imagine anyone consciously thinks like this—but out-of-distribution intuitions may be at work)
What kind of evidence do you imagine would lead OpenAI leadership to change their minds/approach? Do you / your-model-of-leadership believe that there exist Y-problems?
I don’t think I understand your question about Y-problems, since it seems to depend entirely on how specific something can be and still count as a “problem”. Obviously there is already experimental evidence that informs predictions about existential risk from AI in general, but we will get no experimental evidence of any exact situation that occurs beforehand. My claim was more of a vague impression about how OpenAI leadership and John tend to respond to different kinds of evidence in general, and I do not hold it strongly.
To rephrase, it seems to me that in some sense all evidence is experimental. What changes is the degree of generalisation/abstraction required to apply it to a particular problem.
Once we make the distinction between experimental and non-experimental evidence, then we allow for problems on which we only get the “non-experimental” kind—i.e. the kind requiring sufficient generalisation/abstraction that we’d no longer tend to think of it as experimental.
So the question on Y-problems becomes something like:
Given some characterisation of [experimental evidence] (e.g. whatever you meant that OpenAI leadership would tend to put more weight on than John)...
...do you believe there are high-stakes problems for which we’ll get no decision-relevant [experimental evidence] before it’s too late?
[First of all, many thanks for writing the post; it seems both useful and the kind of thing that’ll predictably attract criticism]
I’m not quite sure what you mean to imply here (please correct me if my impression is inaccurate—I’m describing how-it-looks-to-me, and I may well be wrong):
Specifically, John’s model (and mine) has:
X = [Class of high-stakes problems on which we’ll get experimental evidence before it’s too late]
Y = [Class of high-stakes problems on which we’ll get no experimental evidence before it’s too late]
Unless we expect Y to be empty, when we’re talking about Y-problems the weighting is irrelevant: we get no experimental evidence.
Weighting of evidence is an issue when dealing with a fixed problem.
It seems here as if it’s being used to select the problem: we’re going to focus on X-problems because we put a lot of weight on experimental evidence. (obviously silly, so I don’t imagine anyone consciously thinks like this—but out-of-distribution intuitions may be at work)
What kind of evidence do you imagine would lead OpenAI leadership to change their minds/approach?
Do you / your-model-of-leadership believe that there exist Y-problems?
I don’t think I understand your question about Y-problems, since it seems to depend entirely on how specific something can be and still count as a “problem”. Obviously there is already experimental evidence that informs predictions about existential risk from AI in general, but we will get no experimental evidence of any exact situation that occurs beforehand. My claim was more of a vague impression about how OpenAI leadership and John tend to respond to different kinds of evidence in general, and I do not hold it strongly.
To rephrase, it seems to me that in some sense all evidence is experimental. What changes is the degree of generalisation/abstraction required to apply it to a particular problem.
Once we make the distinction between experimental and non-experimental evidence, then we allow for problems on which we only get the “non-experimental” kind—i.e. the kind requiring sufficient generalisation/abstraction that we’d no longer tend to think of it as experimental.
So the question on Y-problems becomes something like:
Given some characterisation of [experimental evidence] (e.g. whatever you meant that OpenAI leadership would tend to put more weight on than John)...
...do you believe there are high-stakes problems for which we’ll get no decision-relevant [experimental evidence] before it’s too late?