There are several instances in the comments of the following schema: Someone finds lsusr’s (self-admittedly not very fleshed out) arguments unsatisfactory and explains why; lsusr responds with just “Would you like to register a counter-prediction?”.
I do not think this is a productive mode of discussion, and would like to see less of it. I think the fault is lsusr’s.
These responses give me the very strong impression—and I would bet fairly heavily that I am not alone in this—that lsusr considers those comments inappropriate, and is trying to convey something along the lines of “put up or shut up; if you disagree with me then you should be able to make a concrete prediction that differs from mine; otherwise, no one should care about your arguments”.
I see three possibilities.
First, maybe lsusr considers that (especially given the last paragraph in the OP) any discussion of the arguments at all is inappropriate. In that case, I think either (1) the arguments should simply not be in the OP at all or else (2) there should be an explicit statement along the lines of “I am not willing to discuss my reasoning and will ignore comments attempting to do so”.
Second, maybe lsusr is happy to discuss the arguments, but only with people who demonstrate their seriousness by making a concrete prediction that differs from lsusr’s. I think this would be a mistake, because it is perfectly possible to find something wrong in an argument without having a definite opinion on the proposition the argument is purporting to support. If someone says “2+2=5, therefore the COVID-19 pandemic was the result of a lab leak”, one should not have to have formed a definite opinion on the origin of the pandemic in order to notice that 2+2 isn’t 5 or that (aside from ex falso quodlibet) there is no particular connection between basic arithmetic and the origins of the pandemic.
(Also: some of those arguments may be of independent interest. lsusr’s claim that poorly-aligned world-optimizing agents converge on “seizing competing pieces of their own mental infrastructure” sounds rather interesting if true—and, for what it’s worth, implausible to me prima facie—regardless of how long the human race is going to survive.)
Third, maybe the impression I have is wrong, and lsusr doesn’t in fact disapprove of, or dislike, comments addressing the arguments without a concrete disagreement with the conclusions. In that case, I think lsusr is giving a misleading impression by not engaging with any such comments in any way other than by posting the same “Would you like to register a counter-prediction?” response to all of them.
Your comment is contingent on several binary possibilities about my intentions. I appreciate your attempt to address all leaves of the decision tree. Here I will help limit the work you have to do by pinning things down.
To clarify,
My post serves one purpose: to register a public prediction. I am betting reputation. But it makes no sense to bet reputation on something everyone agrees on. It only makes sense to bet on things people disagree on. I’m hoping people will make counter-predictions because that can help verify, in the future, that the claims I made were disputed at the time.
These responses give me the very strong impression—and I would bet fairly heavily that I am not alone in this—that lsusr considers those comments [comments without specific predictions] inappropriate, and is trying to convey something along the lines of “put up or shut up; if you disagree with me then you should be able to make a concrete prediction that differs from mine; otherwise, no one should care about your arguments”.
Not exactly. The comments are prefectly appropriate. I don’t plan on engaging them in either direction because the purpose of this post isn’t (for me) to debate. It’s to register a public prediction.
So why ask if people want to make a counter-prediction? Because arguing against me without making a concrete prediction after I just made a public prediction puts comments in a frustratingly ambiguous state where it’s not obvious whether they constitute a counter-prediction. I want to avoid ambiguity in future evaluation of these threads.
To put it another way, someone could claim “I knew lsusr was wrong—see this comment I made” if I turn out to be wrong. That same person could also claim “I don’t lose reputation because I didn’t make an explicit counter-prediction”. I want to avoid this potential for strategic ambiguity.
First, maybe lsusr considers that (especially given the last paragraph in the OP) any discussion of the arguments at all is inappropriate. In that case, I think either (1) the arguments should simply not be in the OP at all or else (2) there should be an explicit statement along the lines of “I am not willing to discuss my reasoning and will ignore comments attempting to do so”.
They’re not arguments intended to convince anyone else of anything. They’re personal reasons for my conclusion. I think it’s better to have them than not to have them, because my post is part of a collaborative effort to find the truth, and more transparency is better toward achieving this end.
As for an explicit statement, here’s something I could try the next time I make a similar post:
This post primarily serves as a public prediction. Please begin all comments with either a counter-prediction or [no prediction]. You are welcome to debate the logic of my reasoning, but do not expect me to engage with you. Right now I am putting skin in the game, not grandstanding.
Second, maybe lsusr is happy to discuss the arguments, but only with people who demonstrate their seriousness by making a concrete prediction that differs from lsusr’s.
Nope, but I appreciate you considering the possibility. I am happy to consider the arguments elsewhere, but the arguments presented here are too thin to defend. For each tiny point I’d have to write a whole post, and if/when I do that I’d rather make an actual top-level post.
Third, maybe the impression I have is wrong, and lsusr doesn’t in fact disapprove of, or dislike, comments addressing the arguments without a concrete disagreement with the conclusions.
I “[don’t] in fact disapprove of, or dislike, comments addressing the arguments without a concrete disagreement with the conclusions.” I enjoy them, actually. I just want to clarify whether the comments are counter-predictions or not.
I appreciate the feedback. I will consider it in the future to as not to give a misleading impression.
I’m not sure I quite agree with you about strategic ambiguity, though. Again, imagine that you’d said “I am 80% confident that the human race will still be here in 100 years, because 2+2=5”. If someone says “I don’t know anything about existential risk, but I know that 2+2 isn’t 5 and that aside from ex falso quodlibet basic arithmetic like this obviously can’t tell us anything about it”, then I am perfectly happy for them to claim that they knew you were wrong even though they didn’t stand to lose anything if your overall prediction turns out right.
(My own position, not that anyone should care: my gut agrees with lsusr’s overall position “but I try not to think with my gut”; I don’t think I understand all the possible ways AI progress could go well enough for any prediction I’d make by explicitly reasoning it out to be worth much; accordingly I decline to make a concrete prediction; I mostly agree that making such predictions is a virtuous activity because it disincentivizes overconfident-looking bullshitting, but I think admitting one’s ignorance is about equally virtuous; the arguments mentioned in the OP seem to me unlikely to be correct but I could well be missing important insights that would make them more plausible. And I do agree that the comments lsusr replied to in the way I’m gently objecting to would have been improved by adding “and therefore I think our chance of survival is below 20%” or “but I do agree that we will probably still be here in 100 years” or “and I have no idea about the actual prediction lsusr is making” or whatever.)
I think the most virtuous solution to your hypothetical is to say “I don’t know anything about existential risk, but I’d bet at 75% confidence that a mathematician will prove that 2+2≠5” (or something along those lines).
While I agree with you that the approach lsusr’s taking isn’t great, and I disagree with the focus on prediction right now, at the same time I do sympathize with lsusr’s approach, and I think this is related to a problem LW has on AI safety epistemics, which is related to general problems of epistemics on LW.
1a3orn nailed it perfectly in that LW has theories that are not predictive enough, in that you could justify too many outcomes of AI safety with the theories we have, and from my meta/outside viewpoint, a lot of LW is neither using empirical evidence like science or trying to formalize things as mathematics has done, but rather philosophizing, which is terrible at getting anywhere close to the epistemic truth of the matter often, and the response is some variation of LW is special at something, but why is this assumed rather than deferring to the outside view.
Don’t get me wrong, I think LW is right on that the science establishment we have is deeply inadequate, and is stuck in an inadequate equilibrium, but one thing we have learned at great cost is that we need to be empirical and touch reality, rather than appeal to cultural traditions or out of touch grand theories.
What lsusr is trying to do is to get people to have predictions on AI safety that are falsifiable and testable, to prevent the problem of being out of touch with reality.
A good model here is Ajeya Cotra and the Technical interpretability community. While Stephen Casper criticized them, they’re probably the only group that is actually trying to touch reality. The fact that that most LWers are either hostile to empirical evidence is probably a product of assumed specialness, without any reason for that specialness.
There are several instances in the comments of the following schema: Someone finds lsusr’s (self-admittedly not very fleshed out) arguments unsatisfactory and explains why; lsusr responds with just “Would you like to register a counter-prediction?”.
I do not think this is a productive mode of discussion, and would like to see less of it. I think the fault is lsusr’s.
These responses give me the very strong impression—and I would bet fairly heavily that I am not alone in this—that lsusr considers those comments inappropriate, and is trying to convey something along the lines of “put up or shut up; if you disagree with me then you should be able to make a concrete prediction that differs from mine; otherwise, no one should care about your arguments”.
I see three possibilities.
First, maybe lsusr considers that (especially given the last paragraph in the OP) any discussion of the arguments at all is inappropriate. In that case, I think either (1) the arguments should simply not be in the OP at all or else (2) there should be an explicit statement along the lines of “I am not willing to discuss my reasoning and will ignore comments attempting to do so”.
Second, maybe lsusr is happy to discuss the arguments, but only with people who demonstrate their seriousness by making a concrete prediction that differs from lsusr’s. I think this would be a mistake, because it is perfectly possible to find something wrong in an argument without having a definite opinion on the proposition the argument is purporting to support. If someone says “2+2=5, therefore the COVID-19 pandemic was the result of a lab leak”, one should not have to have formed a definite opinion on the origin of the pandemic in order to notice that 2+2 isn’t 5 or that (aside from ex falso quodlibet) there is no particular connection between basic arithmetic and the origins of the pandemic.
(Also: some of those arguments may be of independent interest. lsusr’s claim that poorly-aligned world-optimizing agents converge on “seizing competing pieces of their own mental infrastructure” sounds rather interesting if true—and, for what it’s worth, implausible to me prima facie—regardless of how long the human race is going to survive.)
Third, maybe the impression I have is wrong, and lsusr doesn’t in fact disapprove of, or dislike, comments addressing the arguments without a concrete disagreement with the conclusions. In that case, I think lsusr is giving a misleading impression by not engaging with any such comments in any way other than by posting the same “Would you like to register a counter-prediction?” response to all of them.
Your comment is contingent on several binary possibilities about my intentions. I appreciate your attempt to address all leaves of the decision tree. Here I will help limit the work you have to do by pinning things down.
To clarify,
My post serves one purpose: to register a public prediction. I am betting reputation. But it makes no sense to bet reputation on something everyone agrees on. It only makes sense to bet on things people disagree on. I’m hoping people will make counter-predictions because that can help verify, in the future, that the claims I made were disputed at the time.
Not exactly. The comments are prefectly appropriate. I don’t plan on engaging them in either direction because the purpose of this post isn’t (for me) to debate. It’s to register a public prediction.
So why ask if people want to make a counter-prediction? Because arguing against me without making a concrete prediction after I just made a public prediction puts comments in a frustratingly ambiguous state where it’s not obvious whether they constitute a counter-prediction. I want to avoid ambiguity in future evaluation of these threads.
To put it another way, someone could claim “I knew lsusr was wrong—see this comment I made” if I turn out to be wrong. That same person could also claim “I don’t lose reputation because I didn’t make an explicit counter-prediction”. I want to avoid this potential for strategic ambiguity.
They’re not arguments intended to convince anyone else of anything. They’re personal reasons for my conclusion. I think it’s better to have them than not to have them, because my post is part of a collaborative effort to find the truth, and more transparency is better toward achieving this end.
As for an explicit statement, here’s something I could try the next time I make a similar post:
This post primarily serves as a public prediction. Please begin all comments with either a counter-prediction or [no prediction]. You are welcome to debate the logic of my reasoning, but do not expect me to engage with you. Right now I am putting skin in the game, not grandstanding.
Nope, but I appreciate you considering the possibility. I am happy to consider the arguments elsewhere, but the arguments presented here are too thin to defend. For each tiny point I’d have to write a whole post, and if/when I do that I’d rather make an actual top-level post.
I “[don’t] in fact disapprove of, or dislike, comments addressing the arguments without a concrete disagreement with the conclusions.” I enjoy them, actually. I just want to clarify whether the comments are counter-predictions or not.
I appreciate the feedback. I will consider it in the future to as not to give a misleading impression.
Thanks for the clarification.
I’m not sure I quite agree with you about strategic ambiguity, though. Again, imagine that you’d said “I am 80% confident that the human race will still be here in 100 years, because 2+2=5”. If someone says “I don’t know anything about existential risk, but I know that 2+2 isn’t 5 and that aside from ex falso quodlibet basic arithmetic like this obviously can’t tell us anything about it”, then I am perfectly happy for them to claim that they knew you were wrong even though they didn’t stand to lose anything if your overall prediction turns out right.
(My own position, not that anyone should care: my gut agrees with lsusr’s overall position “but I try not to think with my gut”; I don’t think I understand all the possible ways AI progress could go well enough for any prediction I’d make by explicitly reasoning it out to be worth much; accordingly I decline to make a concrete prediction; I mostly agree that making such predictions is a virtuous activity because it disincentivizes overconfident-looking bullshitting, but I think admitting one’s ignorance is about equally virtuous; the arguments mentioned in the OP seem to me unlikely to be correct but I could well be missing important insights that would make them more plausible. And I do agree that the comments lsusr replied to in the way I’m gently objecting to would have been improved by adding “and therefore I think our chance of survival is below 20%” or “but I do agree that we will probably still be here in 100 years” or “and I have no idea about the actual prediction lsusr is making” or whatever.)
I think the most virtuous solution to your hypothetical is to say “I don’t know anything about existential risk, but I’d bet at 75% confidence that a mathematician will prove that 2+2≠5” (or something along those lines).
While I agree with you that the approach lsusr’s taking isn’t great, and I disagree with the focus on prediction right now, at the same time I do sympathize with lsusr’s approach, and I think this is related to a problem LW has on AI safety epistemics, which is related to general problems of epistemics on LW.
1a3orn nailed it perfectly in that LW has theories that are not predictive enough, in that you could justify too many outcomes of AI safety with the theories we have, and from my meta/outside viewpoint, a lot of LW is neither using empirical evidence like science or trying to formalize things as mathematics has done, but rather philosophizing, which is terrible at getting anywhere close to the epistemic truth of the matter often, and the response is some variation of LW is special at something, but why is this assumed rather than deferring to the outside view.
Don’t get me wrong, I think LW is right on that the science establishment we have is deeply inadequate, and is stuck in an inadequate equilibrium, but one thing we have learned at great cost is that we need to be empirical and touch reality, rather than appeal to cultural traditions or out of touch grand theories.
What lsusr is trying to do is to get people to have predictions on AI safety that are falsifiable and testable, to prevent the problem of being out of touch with reality.
A good model here is Ajeya Cotra and the Technical interpretability community. While Stephen Casper criticized them, they’re probably the only group that is actually trying to touch reality. The fact that that most LWers are either hostile to empirical evidence is probably a product of assumed specialness, without any reason for that specialness.
I am very much on board with encouraging more specific predictions and more contact with reality.