I can’t speak for Eliezer, but I can make some short comments about how I am suspicious of thinking in terms of numbers too quickly. I warn you beforehand my thoughts on the subject aren’t very crisp (else, of course, I could put a number on them!)
Mostly I feel like emphasizing the numbers too much fails to respect the process by which we generate them in the first place. When I go as far as putting a number on it, the point is to clarify my beliefs on the subject; it is a summary statistic about my thoughts, not the output of a computation (I mean it technically is, but not a legible computation process we can inspect and/or maybe reverse). The goal of putting a number on it, whatever it may be, is not to manipulate the number with numerical calculations any more than the goal of writing an essay to is grammatically manipulate the concluding sentence, in my view.
Through the summary statistic analogy, I think that I basically disagree with the idea of numbers providing a strong upside in clarity. While I agree that numbers as a format are generally clear, they are only clear as far as that number goes—they communicate very little about the process by which they were reached, which I claim is the key information we want to share.
Consider the arithmetic mean. This number is perfectly clear, insofar as it means there are some numbers which got added together and then divided by how many numbers were summed. Yet this tells us nothing about how many numbers there were, or what the values of the numbers themselves were, or how wide the range of numbers was, or what the possible values were; there are infinitely many variations behind just the mean. It is also true going from no number at all to a mean screens out infinitely many possibilities, and I expect that infinity is substantially larger than the number of possibilities behind any given average. I feel like the crux of my disagreement with the idea of emphasizing numbers is people who endorse them strongly look at the number of possibilities eliminated in the step of going from nothing to an average and think “Look at how much clarity we have gained!” whereas I look at the number of possibilities remaining and think “This is not clear enough to be useful.”
The problem gets worse when numbers are used to communicate. Supposing two people meet in a Bay Area House Party and tell each other their averages. If they both say “seven,” they’ll probably assume they agree, even though it is perfectly possible for the average of what to have literally zero overlap. This is the point at which numbers turn actively misleading, in the literal sense that before they exchanged averages they at least knew they knew nothing, and after exchanging averages they wrongly conclude they agree.
Contrast this with a more practical and realistic case where we might get two different answers on something like probabilities from a data science question. Because it’s a data science question we are already primed to ask questions about the underlying models and the data to see why the numbers are different. We can of course do the same with the example about averages, but in the context of the average even giving the number in the first place is a wasted step because we gain basically nothing until we have the data information (where the sum-of-all-n-divided-by-n is the model). By contrast, in the data science question we can reasonably infer that the models will be broadly similar, and that if they aren’t that information by itself likely points to the cruxes between them. As a consequence, getting the direct numbers is still useful; if two data science sources give very similar answers, they likely do agree very closely.
In sum, we collectively have gigantic uncertainty about the qualitative questions of models and data for whether AI can/will cause human extinction. I claim the true value of quantifying our beliefs, the put-a-number-on-it mental maneuver, is clarifying the qualitative questions. This is also what we really want to be talking about with other people. The trouble is the number we have put on all of this internally is what we communicate but does not contain the process for generating the number, and then the conversation invariably becomes about the numbers, and in my experience this actively obscures the key information we want to exchange.
Thanks, I think I’m sympathetic to a good chunk of this (though I think I still put somewhat greater value on subjective credences than you do). In particular, I agree that there are lots of ways people can mess up when putting subjective credences on things, including “assuming they agree more than they do”.
I think the best solution to this is mostly to teach people about the ways that numbers can mislead, and how to avoid that, so that they can get the benefits of assigning numbers without getting the downside. (E.g.: complementing numerical forecasts with scenario forecasts. I’m a big fan of scenario forecasts.)
My impression is that Eliezer holds a much stronger position than yours. In the bit I quoted above, I think Eliezer isn’t only objecting to putting too much emphasis on subjective credences, but is objecting to putting subjective credences on things at all.
I also think he objects to putting numbers on things, and I also avoid doing it. A concrete example: I explicitly avoid putting numbers on things in LessWrong posts. The reason is straightforward—if a number appears anywhere in the post, about half of the conversation in the comments will be on that number to the exclusion of the point of the post (or the lack of one, etc). So unless numbers are indeed the thing you want to be talking about, in the sense of detailed results of specific computations, they are positively distracting from the rest of the post for the audience.
I focused on the communication aspect in my response, but I should probably also say that I don’t really track what the number is when I actually go to the trouble of computing a prior, personally. The point of generating the number is clarifying the qualitative information, and then the point remains the qualitative information after I got the number; I only really start paying attention to what the number is if it stays consistent enough after doing the generate-a-number move that I recognize it as being basically the same as the last few times. Even then, I am spending most of my effort on the qualitative level directly.
I make an analogy to computer programs: the sheer fact of successfully producing an output without errors weighs much more than whatever the value of the output is. The program remains our central concern, and continuing to improve it using known patterns and good practices for writing code is usually the most effective method. Taking the programming analogy one layer further, there’s a significant chunk of time where you can be extremely confident the output is meaningless; suppose you haven’t even completed what you already know to be minimum requirements, and compile the program anyway, just to test for errors so far. There’s no point in running the program all the way to an output, because you know it would be meaningless. In the programming analogy, a focus on the value of the output is a kind of “premature optimization is the root of all evil” problem.
I do think this probably reflects the fact that Eliezer’s time is mostly spent on poorly understood problems like AI, rather than on stable well-understood domains where working with numbers is a much more reasonable prospect. But it still feels like even in the case where I am trying to learn something that is well-understood, just not by me, trying for a number feels opposite the idea of hugging the query, somehow. Or in virtue language: how does the number cut the enemy?
I can’t speak for Eliezer, but I can make some short comments about how I am suspicious of thinking in terms of numbers too quickly. I warn you beforehand my thoughts on the subject aren’t very crisp (else, of course, I could put a number on them!)
Mostly I feel like emphasizing the numbers too much fails to respect the process by which we generate them in the first place. When I go as far as putting a number on it, the point is to clarify my beliefs on the subject; it is a summary statistic about my thoughts, not the output of a computation (I mean it technically is, but not a legible computation process we can inspect and/or maybe reverse). The goal of putting a number on it, whatever it may be, is not to manipulate the number with numerical calculations any more than the goal of writing an essay to is grammatically manipulate the concluding sentence, in my view.
Through the summary statistic analogy, I think that I basically disagree with the idea of numbers providing a strong upside in clarity. While I agree that numbers as a format are generally clear, they are only clear as far as that number goes—they communicate very little about the process by which they were reached, which I claim is the key information we want to share.
Consider the arithmetic mean. This number is perfectly clear, insofar as it means there are some numbers which got added together and then divided by how many numbers were summed. Yet this tells us nothing about how many numbers there were, or what the values of the numbers themselves were, or how wide the range of numbers was, or what the possible values were; there are infinitely many variations behind just the mean. It is also true going from no number at all to a mean screens out infinitely many possibilities, and I expect that infinity is substantially larger than the number of possibilities behind any given average. I feel like the crux of my disagreement with the idea of emphasizing numbers is people who endorse them strongly look at the number of possibilities eliminated in the step of going from nothing to an average and think “Look at how much clarity we have gained!” whereas I look at the number of possibilities remaining and think “This is not clear enough to be useful.”
The problem gets worse when numbers are used to communicate. Supposing two people meet in a Bay Area House Party and tell each other their averages. If they both say “seven,” they’ll probably assume they agree, even though it is perfectly possible for the average of what to have literally zero overlap. This is the point at which numbers turn actively misleading, in the literal sense that before they exchanged averages they at least knew they knew nothing, and after exchanging averages they wrongly conclude they agree.
Contrast this with a more practical and realistic case where we might get two different answers on something like probabilities from a data science question. Because it’s a data science question we are already primed to ask questions about the underlying models and the data to see why the numbers are different. We can of course do the same with the example about averages, but in the context of the average even giving the number in the first place is a wasted step because we gain basically nothing until we have the data information (where the sum-of-all-n-divided-by-n is the model). By contrast, in the data science question we can reasonably infer that the models will be broadly similar, and that if they aren’t that information by itself likely points to the cruxes between them. As a consequence, getting the direct numbers is still useful; if two data science sources give very similar answers, they likely do agree very closely.
In sum, we collectively have gigantic uncertainty about the qualitative questions of models and data for whether AI can/will cause human extinction. I claim the true value of quantifying our beliefs, the put-a-number-on-it mental maneuver, is clarifying the qualitative questions. This is also what we really want to be talking about with other people. The trouble is the number we have put on all of this internally is what we communicate but does not contain the process for generating the number, and then the conversation invariably becomes about the numbers, and in my experience this actively obscures the key information we want to exchange.
Thanks, I think I’m sympathetic to a good chunk of this (though I think I still put somewhat greater value on subjective credences than you do). In particular, I agree that there are lots of ways people can mess up when putting subjective credences on things, including “assuming they agree more than they do”.
I think the best solution to this is mostly to teach people about the ways that numbers can mislead, and how to avoid that, so that they can get the benefits of assigning numbers without getting the downside. (E.g.: complementing numerical forecasts with scenario forecasts. I’m a big fan of scenario forecasts.)
My impression is that Eliezer holds a much stronger position than yours. In the bit I quoted above, I think Eliezer isn’t only objecting to putting too much emphasis on subjective credences, but is objecting to putting subjective credences on things at all.
I also think he objects to putting numbers on things, and I also avoid doing it. A concrete example: I explicitly avoid putting numbers on things in LessWrong posts. The reason is straightforward—if a number appears anywhere in the post, about half of the conversation in the comments will be on that number to the exclusion of the point of the post (or the lack of one, etc). So unless numbers are indeed the thing you want to be talking about, in the sense of detailed results of specific computations, they are positively distracting from the rest of the post for the audience.
I focused on the communication aspect in my response, but I should probably also say that I don’t really track what the number is when I actually go to the trouble of computing a prior, personally. The point of generating the number is clarifying the qualitative information, and then the point remains the qualitative information after I got the number; I only really start paying attention to what the number is if it stays consistent enough after doing the generate-a-number move that I recognize it as being basically the same as the last few times. Even then, I am spending most of my effort on the qualitative level directly.
I make an analogy to computer programs: the sheer fact of successfully producing an output without errors weighs much more than whatever the value of the output is. The program remains our central concern, and continuing to improve it using known patterns and good practices for writing code is usually the most effective method. Taking the programming analogy one layer further, there’s a significant chunk of time where you can be extremely confident the output is meaningless; suppose you haven’t even completed what you already know to be minimum requirements, and compile the program anyway, just to test for errors so far. There’s no point in running the program all the way to an output, because you know it would be meaningless. In the programming analogy, a focus on the value of the output is a kind of “premature optimization is the root of all evil” problem.
I do think this probably reflects the fact that Eliezer’s time is mostly spent on poorly understood problems like AI, rather than on stable well-understood domains where working with numbers is a much more reasonable prospect. But it still feels like even in the case where I am trying to learn something that is well-understood, just not by me, trying for a number feels opposite the idea of hugging the query, somehow. Or in virtue language: how does the number cut the enemy?