In the below article I give an honest list of considerations about my thoughts about AI. Currently it sits on −1 karma (without my own).
This is sort of fine. I don’t think it’s a great article and I am not sure that it’s highly worthy of people’s attention, but I think that a community that wants to encourage thinking about AI might not want to penalise those who do so.
When I was a Christian, questions were always “acceptable”. But if I asked “so why is the bible true” that would have received a sort of stern look and then a single paragraph answer. While the question was explictly accepted, it was implictly a thing I should stop doing.
I think it’s probable you are doing the same thing. “oh yes, lets think about AI” but if I write something about my thoughts on AI that disagrees with you, it isn’t worth reading or engaging with.
And my article is relatively mild pushback. What happens if a genuine critic comes on here and writes something. I agree that some criticism is bad, but what if it is in the form that you ask for (lists of considerations, transparently written)?
Is the only criticism worth reading that which is actually convincing to you? And won’t, due to some bias, that likely leave this place an echo chamber?
To be fair, going to check the article in question, my main problem with it is that I find it hard to read and sort of disconnected, in a way that makes it difficult for me to even keep reading it and forming a full notion of what its thesis is to engage with it. Lots of “perhaps” that feel like branching points (am I supposed to assume you think this is true or not? Am I supposed to keep it in mind as a case of “we’ll go over both options”?), lots of disconnected facts presented in bullet point style before we start seeing a thread of synthesis of all these things into a coherent argument. I’d say it’s well possible that the downvoting is more about the problems with the article as an example of writing than with the specific views it expresses. To get at the views you need to make it through the article in the first place.
Nathan Young is a very strong trader on manifold markets. This is the ramblings of a mind that gets things right a lot. This isn’t the first time he’s been downvoted for high quality work before. I suspect editing is the problem here. I’ve noticed I often want to independently downvote readability vs semantics.
Also, I suspect that doing really well on a prediction market screws up your “learned decision theory”, for the usual self-referential-market reasons, which I tend to think are more pervasive than consensus seems to. relatedly, my impression is that prediction markets consistently get things wrong where those things are dangerous for people to make non-anonymous predictions about or non-anonymously make markets about.[1]
Those things make me suspect that someone who is very good at prediction markets is likely not using evidence about adversarial action very well, since winning at a market where there’s a lopsidedness in what questions get asked seems like it wouldn’t train some relevant things. I left some links in comments on nathan’s post but, well, my comments didn’t get upvoted,
anyway I reversed my small downvote to a large upvote of his post, and will likely not downvote similar posts in the future, thanks to this complaining. habryka, if you want more downvoting, can you explain why more downvoting is good even in the face of pushback like this?
hey nathan can u fix that pls using ur prediction market superpowers, I want to see prediction markets have a much higher ratio of “this terrible thing will happen by this amount during …” [answers: list of months] questions.
What happens if a genuine critic comes on here and writes something. I agree that some criticism is bad, but what if it is in the form that you ask for (lists of considerations, transparently written)?
Is the only criticism worth reading that which is actually convincing to you? And won’t, due to some bias, that likely leave this place an echo chamber?
More than one (imo, excellent) critic of AI doom pessimism has cut back on the amount they contribute to LW because it’s grueling / unrewarding. So there’s already a bit of a spiral going on here.
My view is that LW probably continues to slide more the AI-doom-o-sphere than towards rationality due to dynamics, including but not limited to:
lower standards for AI doom, higher standards for not-doom
lower standards of politeness required of prominent doomers
network dynamics
continued prominence given to doom content, i.e., treatment of MIRI’s book
I know this is contrary to what people leading LW would like. But in absence of sustained contrary action this looks to me like a trend that’s already going on, rather than a trend that’s inchoate.
Yeah, I think the default direction is towards evaporative cooling here, and my guess is we probably want to do something more intentional to avoid that. We do stuff like applying a bias to curating ideas from new authors and contributors that bring in more new ideas and perspectives, but we don’t do a huge amount.
Curious if you have suggestions for things to do. I also am curious about who the critics are who cut back on the amount of contributions, if you could DM me their names, since that kind of data would be helpful (both for me judging how big their loss is, and understanding the specific dynamics around them cutting back).
Have you ever piloted disagree votes on posts? I think people are predictably downvoting posts they disagree with rather than disagree-voting them. it’s been requested many times, and seems like an obvious fix.
I also think that anonymous voting on substrings (like reacts but anonymous) might be good, to allow the feedback to be more targeted.
If I ran a website I would have required that a downvote [edit: or an upvote!] come with a substring and made it easy to provide a downvote reason seen only by the post creator.
as might be prompts to add reacts if you vote.
I understand that, for reasons unknown to me, your explore rate is low on these topics, which seems strange to me. My best guess is it’s just that you’re busy and this is a somewhat heavy codebase.
I think you’re throwing away a lot more value than you expect by not giving a non-downvote outlet for disagreement signals on posts.
I think downvote-only reasoning-requirements would be bad, people already don’t downvote enough. If you make them symmetrical you are basically just removing voting as a signal, because you just increased the associated friction by >10x.
I don’t feel super confident about not enabling disagree-voting on posts, but I currently think it would be pretty bad. Agree/disagree voting is already playing a kind of dangerous game by encouraging rounding the epistemic content of a comment into a single statement that it makes sense to assign a single truth-value to. This usually works fine for comments which are short, though sometimes fails. I think it basically never works for posts. I agree we could do an anonymous version of inline-reacts to get something more similar, but I think that would add too much complexity to the already very complicated react system.
Agree/disagree voting is already playing a kind of dangerous game by encouraging rounding the epistemic content of a comment into a single statement that it makes sense to assign a single truth-value to.
(I obviously haven’t thought about this as much as you. Very low confidence.)
I’m inclined to say that a strong part of human nature is to round the vibes that one feels towards a post into a single axis of “yay” versus “boo”, and then to feel a very strong urge to proclaim those vibes publicly.
And I think that people are doing that right now, via the karma vote.
I think an agree/disagree dial on posts would be an outlet (“tank”) to absorb those vibe expressions, and that this would shelter karma, allowing karma to retain its different role (“it’s good / bad that this exists”).
I agree that this whole thing (with people rounding everything into a 1D boo/yay vibes axis and then feeling an urge to publicly proclaim where they sit) is dumb, and if only we could all be autistic decouplers etc. But in the real world, I think the benefits of agree-voting (in helping prevent the dynamic where people with minority opinions get driven to negative karma and off the site) probably outweigh the cost (in having an agree / disagree tally on posts which is kinda dumb and meaningless).
This usually works fine for comments which are short, though sometimes fails. I think it basically never works for posts.
Right, but it’s already never working for posts, you just have an even lower dimensional view of it. Agree votes function more as a sink that isn’t karma-loaded when something is validly argued but seems wrong.
As someone who likely is not one of the people you’re criticizing for not downvoting enough, I’m not entirely sure I understand why others don’t, but my guess is that it feels rude. I know that downvoting someone new feels rude to me to do, so I usually leave a comment, but leaving a named comment is pretty high overhead.
If I felt it was welcome, I might have left a bunch of critical span-specific reacts on Nathan’s post, but I have the impression that that’s not considered acceptable by the mods, since I once did that and got scolded. If you want more downvoting, perhaps explicitly requesting something like this? either in the ui or as a pinned post perhaps
I try pretty hard (and I think most of the team does) to at least moderate AI x-risk criticism more leniently. But of course, it’s tricky to know if you’re doing a good job. Am I undercorrecting or overcorrecting for my bias? If you ever notice some examples that seem like moderation bias, please lmk!
Of course, moderation is only a small part of what drives the site culture/reward dynamics
Yeah to be clear, although I would act differently, I do think the LW team both tries hard to do well here, and tries more effectually than most other teams would.
It’s just that once LW has become much more of a Schelling point for doom more than for rationality, there’s a pretty steep natural slope.
I thought the post was fine and was surprised it was so downvoted. Even if people don’t agree with the considerations, or think all the most important considerations are missing, why should a post saying, “Here’s what I think and why I think it, feel free to push back in the comments,” be so poorly received? Commenters can just say what they think is missing.
Seems likely that it wouldn’t have been so downvoted if its bottom line was that AI risk is very high. Increases my P(LW groupthink is a problem) a bit.
I agree site design wise. Right now, upvotes and downvotes on posts get used partially as agree disagree, and despite that an LW dev apparently thinks one can’t disagree with a post rationally, I think it’s happening already and so might as well give it an outlet.
this is a “negative” post with hundreds of upvotes and meaningful discussion in the comments. The different between your post and this one is not the “level of criticism”, but the quality and logical basis coming from the argument. I agree with Seth Herds argument from the comments of your post re the difference here, can’t figure out how to link it. There are many fair criticisms of lesswrong culture, but “biased” and “echochamber” are not among them in my experience. I don’t mean to attack your character, writing skills, or general opinions, as I’m sure you are capable of writing something of higher quality that better expresses your thoughts and opinions.
You’ll note that the negative post you linked is negative about AI timelines (“AI timelines are longer than many think”), while OP’s is negative about AI doom being an issue (“I’m probably going to move from ~5% doom to ~1% doom.”)
I explained in a comment on that post what’s wrong with it by LW standards.
The general question is worth investigating, but making that post the example for consideration isn’t a good idea. It was neither that good nor that rationalist/rational in focus, the two things that qualify posts for LW standards and objectives.
Making a more general argument for the bias of LW would be a great way to earn some upvotes. We’re suckers for analyzing our own biases.
The problem is it’s hard to distinguish between being biased, and just being more rational and more right. Most arguments for AI doom suck. Most arguments against AI doom also suck. Most of the arguments that don’t suck land you at way, way above the 1-5% doom you quoted. BUT of course we could just be mistaking massive amounts of thought and analysis for bias. Or not. It’s an open question.
It’s tricky to approach the meta-question without approaching the object level.
It could just be that LW doesn’t like most arguments against AGI x-risk because people only make those arguments before they’ve considered the whole question, so they tend to not be very rational.
I’ve tried to steelman arguments against, and I can’t get anywhere near “oh yeah this should be fine” without leaving out huge chunks of the question and likely futures. In particular, if I ONLY think of AI as LLMs, I get those low doom probabilities—but we’re just obviously not going to stop there without some remarkable changes.
The best I can get is something like “people tend to worry a lot, and people tend to solve problems pretty well once they’re close and so seem important”. That might get me down to like 10% if I’m feeling super optimistic. That’s if I’m considering the whole problem: we’re making a new alien species that will be way smarter than us.
In the below article I give an honest list of considerations about my thoughts about AI. Currently it sits on −1 karma (without my own).
This is sort of fine. I don’t think it’s a great article and I am not sure that it’s highly worthy of people’s attention, but I think that a community that wants to encourage thinking about AI might not want to penalise those who do so.
When I was a Christian, questions were always “acceptable”. But if I asked “so why is the bible true” that would have received a sort of stern look and then a single paragraph answer. While the question was explictly accepted, it was implictly a thing I should stop doing.
I think it’s probable you are doing the same thing. “oh yes, lets think about AI” but if I write something about my thoughts on AI that disagrees with you, it isn’t worth reading or engaging with.
And my article is relatively mild pushback. What happens if a genuine critic comes on here and writes something. I agree that some criticism is bad, but what if it is in the form that you ask for (lists of considerations, transparently written)?
Is the only criticism worth reading that which is actually convincing to you? And won’t, due to some bias, that likely leave this place an echo chamber?
https://www.lesswrong.com/posts/HKHqFWT7qiac2tvtF/my-ai-vibes-are-shifting?commentId=RNFEEi7tm6eFcwkgP
To be fair, going to check the article in question, my main problem with it is that I find it hard to read and sort of disconnected, in a way that makes it difficult for me to even keep reading it and forming a full notion of what its thesis is to engage with it. Lots of “perhaps” that feel like branching points (am I supposed to assume you think this is true or not? Am I supposed to keep it in mind as a case of “we’ll go over both options”?), lots of disconnected facts presented in bullet point style before we start seeing a thread of synthesis of all these things into a coherent argument. I’d say it’s well possible that the downvoting is more about the problems with the article as an example of writing than with the specific views it expresses. To get at the views you need to make it through the article in the first place.
Nathan Young is a very strong trader on manifold markets. This is the ramblings of a mind that gets things right a lot. This isn’t the first time he’s been downvoted for high quality work before. I suspect editing is the problem here. I’ve noticed I often want to independently downvote readability vs semantics.
Also, I suspect that doing really well on a prediction market screws up your “learned decision theory”, for the usual self-referential-market reasons, which I tend to think are more pervasive than consensus seems to. relatedly, my impression is that prediction markets consistently get things wrong where those things are dangerous for people to make non-anonymous predictions about or non-anonymously make markets about.[1]
Those things make me suspect that someone who is very good at prediction markets is likely not using evidence about adversarial action very well, since winning at a market where there’s a lopsidedness in what questions get asked seems like it wouldn’t train some relevant things. I left some links in comments on nathan’s post but, well, my comments didn’t get upvoted,
anyway I reversed my small downvote to a large upvote of his post, and will likely not downvote similar posts in the future, thanks to this complaining. habryka, if you want more downvoting, can you explain why more downvoting is good even in the face of pushback like this?
hey nathan can u fix that pls using ur prediction market superpowers, I want to see prediction markets have a much higher ratio of “this terrible thing will happen by this amount during …” [answers: list of months] questions.
More than one (imo, excellent) critic of AI doom pessimism has cut back on the amount they contribute to LW because it’s grueling / unrewarding. So there’s already a bit of a spiral going on here.
My view is that LW probably continues to slide more the AI-doom-o-sphere than towards rationality due to dynamics, including but not limited to:
lower standards for AI doom, higher standards for not-doom
lower standards of politeness required of prominent doomers
network dynamics
continued prominence given to doom content, i.e., treatment of MIRI’s book
I know this is contrary to what people leading LW would like. But in absence of sustained contrary action this looks to me like a trend that’s already going on, rather than a trend that’s inchoate.
Yeah, I think the default direction is towards evaporative cooling here, and my guess is we probably want to do something more intentional to avoid that. We do stuff like applying a bias to curating ideas from new authors and contributors that bring in more new ideas and perspectives, but we don’t do a huge amount.
Curious if you have suggestions for things to do. I also am curious about who the critics are who cut back on the amount of contributions, if you could DM me their names, since that kind of data would be helpful (both for me judging how big their loss is, and understanding the specific dynamics around them cutting back).
Have you ever piloted disagree votes on posts? I think people are predictably downvoting posts they disagree with rather than disagree-voting them. it’s been requested many times, and seems like an obvious fix.
I also think that anonymous voting on substrings (like reacts but anonymous) might be good, to allow the feedback to be more targeted.
If I ran a website I would have required that a downvote [edit: or an upvote!] come with a substring and made it easy to provide a downvote reason seen only by the post creator.
as might be prompts to add reacts if you vote.
I understand that, for reasons unknown to me, your explore rate is low on these topics, which seems strange to me. My best guess is it’s just that you’re busy and this is a somewhat heavy codebase.
I think you’re throwing away a lot more value than you expect by not giving a non-downvote outlet for disagreement signals on posts.
I think downvote-only reasoning-requirements would be bad, people already don’t downvote enough. If you make them symmetrical you are basically just removing voting as a signal, because you just increased the associated friction by >10x.
I don’t feel super confident about not enabling disagree-voting on posts, but I currently think it would be pretty bad. Agree/disagree voting is already playing a kind of dangerous game by encouraging rounding the epistemic content of a comment into a single statement that it makes sense to assign a single truth-value to. This usually works fine for comments which are short, though sometimes fails. I think it basically never works for posts. I agree we could do an anonymous version of inline-reacts to get something more similar, but I think that would add too much complexity to the already very complicated react system.
(I obviously haven’t thought about this as much as you. Very low confidence.)
I’m inclined to say that a strong part of human nature is to round the vibes that one feels towards a post into a single axis of “yay” versus “boo”, and then to feel a very strong urge to proclaim those vibes publicly.
And I think that people are doing that right now, via the karma vote.
I think an agree/disagree dial on posts would be an outlet (“tank”) to absorb those vibe expressions, and that this would shelter karma, allowing karma to retain its different role (“it’s good / bad that this exists”).
I agree that this whole thing (with people rounding everything into a 1D boo/yay vibes axis and then feeling an urge to publicly proclaim where they sit) is dumb, and if only we could all be autistic decouplers etc. But in the real world, I think the benefits of agree-voting (in helping prevent the dynamic where people with minority opinions get driven to negative karma and off the site) probably outweigh the cost (in having an agree / disagree tally on posts which is kinda dumb and meaningless).
Right, but it’s already never working for posts, you just have an even lower dimensional view of it. Agree votes function more as a sink that isn’t karma-loaded when something is validly argued but seems wrong.
As someone who likely is not one of the people you’re criticizing for not downvoting enough, I’m not entirely sure I understand why others don’t, but my guess is that it feels rude. I know that downvoting someone new feels rude to me to do, so I usually leave a comment, but leaving a named comment is pretty high overhead.
If I felt it was welcome, I might have left a bunch of critical span-specific reacts on Nathan’s post, but I have the impression that that’s not considered acceptable by the mods, since I once did that and got scolded. If you want more downvoting, perhaps explicitly requesting something like this? either in the ui or as a pinned post perhaps
EA forum has agree / disagree on posts, but I don’t spend enough time there to have an opinion about its effects.
I try pretty hard (and I think most of the team does) to at least moderate AI x-risk criticism more leniently. But of course, it’s tricky to know if you’re doing a good job. Am I undercorrecting or overcorrecting for my bias? If you ever notice some examples that seem like moderation bias, please lmk!
Of course, moderation is only a small part of what drives the site culture/reward dynamics
Yeah to be clear, although I would act differently, I do think the LW team both tries hard to do well here, and tries more effectually than most other teams would.
It’s just that once LW has become much more of a Schelling point for doom more than for rationality, there’s a pretty steep natural slope.
I thought the post was fine and was surprised it was so downvoted. Even if people don’t agree with the considerations, or think all the most important considerations are missing, why should a post saying, “Here’s what I think and why I think it, feel free to push back in the comments,” be so poorly received? Commenters can just say what they think is missing.
Seems likely that it wouldn’t have been so downvoted if its bottom line was that AI risk is very high. Increases my P(LW groupthink is a problem) a bit.
I agree site design wise. Right now, upvotes and downvotes on posts get used partially as agree disagree, and despite that an LW dev apparently thinks one can’t disagree with a post rationally, I think it’s happening already and so might as well give it an outlet.
strong disagree, see https://www.lesswrong.com/posts/oKAFFvaouKKEhbBPm/a-bear-case-my-predictions-regarding-ai-progress
this is a “negative” post with hundreds of upvotes and meaningful discussion in the comments. The different between your post and this one is not the “level of criticism”, but the quality and logical basis coming from the argument. I agree with Seth Herds argument from the comments of your post re the difference here, can’t figure out how to link it. There are many fair criticisms of lesswrong culture, but “biased” and “echochamber” are not among them in my experience. I don’t mean to attack your character, writing skills, or general opinions, as I’m sure you are capable of writing something of higher quality that better expresses your thoughts and opinions.
You’ll note that the negative post you linked is negative about AI timelines (“AI timelines are longer than many think”), while OP’s is negative about AI doom being an issue (“I’m probably going to move from ~5% doom to ~1% doom.”)
Agree the above post is a weak-ish example. This post feels like a better example: https://www.lesswrong.com/posts/LDRQ5Zfqwi8GjzPYG/counterarguments-to-the-basic-ai-x-risk-case
This feels like weak evidence against my point, though I think “timelines” and “overall AI risk” are different levels of safe to argue about.
I explained in a comment on that post what’s wrong with it by LW standards.
The general question is worth investigating, but making that post the example for consideration isn’t a good idea. It was neither that good nor that rationalist/rational in focus, the two things that qualify posts for LW standards and objectives.
Making a more general argument for the bias of LW would be a great way to earn some upvotes. We’re suckers for analyzing our own biases.
The problem is it’s hard to distinguish between being biased, and just being more rational and more right. Most arguments for AI doom suck. Most arguments against AI doom also suck. Most of the arguments that don’t suck land you at way, way above the 1-5% doom you quoted. BUT of course we could just be mistaking massive amounts of thought and analysis for bias. Or not. It’s an open question.
It’s tricky to approach the meta-question without approaching the object level.
It could just be that LW doesn’t like most arguments against AGI x-risk because people only make those arguments before they’ve considered the whole question, so they tend to not be very rational.
I’ve tried to steelman arguments against, and I can’t get anywhere near “oh yeah this should be fine” without leaving out huge chunks of the question and likely futures. In particular, if I ONLY think of AI as LLMs, I get those low doom probabilities—but we’re just obviously not going to stop there without some remarkable changes.
The best I can get is something like “people tend to worry a lot, and people tend to solve problems pretty well once they’re close and so seem important”. That might get me down to like 10% if I’m feeling super optimistic. That’s if I’m considering the whole problem: we’re making a new alien species that will be way smarter than us.