First, to answer your question: the big thing you’re missing are the technical arguments for the difficulty of alignment. That’s where most of the variance lies. All of the factors you list are small potatoes if it just turns out that alignment is quite hard.
The other big factor is the overall societal dynamics. You casually mention”then we shouldn’t build it”. The big question is if we SHOULD stop, COULD we stop? I think there’s only maybe a 10% chance we could if we should. The incentives are just too strong and the coordination problem is unsolved. And the people in power on this are looking quite incompetent to deal with it. They’ll probably start taking it seriously at some point, but whether that’s soon enough to help is a big question. You could look at my Whether governments will control AGI is important and neglected if you wanted a little more of my logic on that.
It’s a problem for logic based on careful observations and data. Vibes-based arguments are actively unhelpful. It’s been fine to just guess at most other stuff in history, but not this.
I’m being kind of judgmental because I think having good estimates of AGI risk is quite important for our collective odds of survival and/or flourishing And LW is, among other things, the best place on earth to get good guesses on that.. I apologize for being a little harsh.
If you titled this “some factors maybe in AI risk” or “some changes that have shifted my p(doom)” or something and left out the p(doom) I’d have upvoted because you have some interesting observations.
As it is, I did very much think this is anti-worth reading in the context of LW. I couldn’t decide between normal and big downvotes.
I think this is polluting the community epistemics by making predictions based on vibes. Then you deny they’re predictions in the comments. P(doom) is very much a prediction and you shouldn’t make it publicly if you aren’t really trying. Or maybe that’s what twitter or reddit are for?
It would be pretty rare that 30 minutes of work would be worth reading on LW compared to the many excellent high effort posts here. I do appreciate you putting that caveat. The exception would be an expert or unique insight or unique idea. Just voicing non-expert quick takes isn’t really what LW is for IMO. At most it would be a quick take.
Or to put it this way: LW is for the opposite of vibes-based opinions.
I think considerations are in important input into decision making and if you downvote anyone who writes clear considerations without conforming to your extremely high standards then you will tax disagreement.
Perhaps you are very confident that you are only taxing bad takes and not just contrary ones, but I am not as confident as you are.
Overall, I think this is poor behaviour from a truth-seeking community. I don’t expect every critic to be complimented to high heaven (as sometimes happens on the EA forum) but I think that this seems like a bad equilibrium for a post that is (in my view) fine and presented in the way this community requests (transparent and with a list of considerations).
As for the title:
If you titled this “some factors maybe in AI risk” or “some factors changes that have shifted my p(doom)” or something and left out the p(doom) I’d have upvoted because you have some interesting observations.
This is particular seems like a dodge. The actual title “My AI Vibes are Shifting” is hardly confident or declarative. Are you sure you would actully upvote if I had titled as you suggest?
I went back and reread it. Because you did mark that p(doom) as vibes based and Said you weren’t making strong predictions near the top, I removed my small downvote.
I said I’d have upvoted if you removed the prediction. The prediction is the problem here because it is appears to be based on bad logic—vibes instead of gears.
I have never downvoted something that disagrees with my stance if it tries to come to grips with the central problem of how difficult alignment is.
Nor have I downvoted pieces that scope to address only part of the pmquestion and don’t make a P(doom) prediction.
I have frequently complained on new authors’ behalf that the LW community has downvoted unfairly.
Or to put it this way: LW is for the opposite of vibes-based opinions.
I think vibes which are actually gestalts of seeing a lot of mechanisms are potentially okay, but then I expect to see the vibes be responsive to evidence. Predictors who consistently get things right are likely to be pretty good at getting an accurate vibe, but then in order to export their view to others, I want to see them disassemble their vibe into parts. A major flaw in prediction markets in that they don’t demand you share the reasoning, or even have any particular reasoning. They allow being right for arbitrarily wrong reasons, which generalizes poorly.
Since you imply you want feedback, I’ll give it.
First, to answer your question: the big thing you’re missing are the technical arguments for the difficulty of alignment. That’s where most of the variance lies. All of the factors you list are small potatoes if it just turns out that alignment is quite hard.
The other big factor is the overall societal dynamics. You casually mention”then we shouldn’t build it”. The big question is if we SHOULD stop, COULD we stop? I think there’s only maybe a 10% chance we could if we should. The incentives are just too strong and the coordination problem is unsolved. And the people in power on this are looking quite incompetent to deal with it. They’ll probably start taking it seriously at some point, but whether that’s soon enough to help is a big question. You could look at my Whether governments will control AGI is important and neglected if you wanted a little more of my logic on that.
It’s a problem for logic based on careful observations and data. Vibes-based arguments are actively unhelpful. It’s been fine to just guess at most other stuff in history, but not this.
I’m being kind of judgmental because I think having good estimates of AGI risk is quite important for our collective odds of survival and/or flourishing And LW is, among other things, the best place on earth to get good guesses on that.. I apologize for being a little harsh.
If you titled this “some factors maybe in AI risk” or “some changes that have shifted my p(doom)” or something and left out the p(doom) I’d have upvoted because you have some interesting observations.
As it is, I did very much think this is anti-worth reading in the context of LW. I couldn’t decide between normal and big downvotes.
I think this is polluting the community epistemics by making predictions based on vibes. Then you deny they’re predictions in the comments. P(doom) is very much a prediction and you shouldn’t make it publicly if you aren’t really trying. Or maybe that’s what twitter or reddit are for?
It would be pretty rare that 30 minutes of work would be worth reading on LW compared to the many excellent high effort posts here. I do appreciate you putting that caveat. The exception would be an expert or unique insight or unique idea. Just voicing non-expert quick takes isn’t really what LW is for IMO. At most it would be a quick take.
Or to put it this way: LW is for the opposite of vibes-based opinions.
I think considerations are in important input into decision making and if you downvote anyone who writes clear considerations without conforming to your extremely high standards then you will tax disagreement.
Perhaps you are very confident that you are only taxing bad takes and not just contrary ones, but I am not as confident as you are.
Overall, I think this is poor behaviour from a truth-seeking community. I don’t expect every critic to be complimented to high heaven (as sometimes happens on the EA forum) but I think that this seems like a bad equilibrium for a post that is (in my view) fine and presented in the way this community requests (transparent and with a list of considerations).
As for the title:
This is particular seems like a dodge. The actual title “My AI Vibes are Shifting” is hardly confident or declarative. Are you sure you would actully upvote if I had titled as you suggest?
I went back and reread it. Because you did mark that p(doom) as vibes based and Said you weren’t making strong predictions near the top, I removed my small downvote.
I said I’d have upvoted if you removed the prediction. The prediction is the problem here because it is appears to be based on bad logic—vibes instead of gears.
I have never downvoted something that disagrees with my stance if it tries to come to grips with the central problem of how difficult alignment is.
Nor have I downvoted pieces that scope to address only part of the pmquestion and don’t make a P(doom) prediction.
I have frequently complained on new authors’ behalf that the LW community has downvoted unfairly.
I think vibes which are actually gestalts of seeing a lot of mechanisms are potentially okay, but then I expect to see the vibes be responsive to evidence. Predictors who consistently get things right are likely to be pretty good at getting an accurate vibe, but then in order to export their view to others, I want to see them disassemble their vibe into parts. A major flaw in prediction markets in that they don’t demand you share the reasoning, or even have any particular reasoning. They allow being right for arbitrarily wrong reasons, which generalizes poorly.