This seems like it’s engaging with the question of “what do critics think?” in a sort of model-free, uninformed, “who to defer to” sort of way.
For awhile, I didn’t fully update on arguments for AI Risk being a Big Deal because the arguments were kinda complex and I could imagine clever arguers convincing me of it without it being true. One of the things that updated me over the course of 4 years was actually reading the replies (including by people like Hanson) and thinking “man, they didn’t seem to even understand or address the main points.”
i.e. it’s not that they didn’t engage with the arguments, it’s that they engaged with the arguments badly which lowered my credence on taking their opinion seriously.
(I think nowadays I have seen some critics who do seem to me to have engaged with most of the real points. None of their counterarguments seem like they’ve added up to “AI is not a huge fucking deal that is extremely risky” in a way that makes any sense to me, but, some of them add up to alternate frames of looking at the problem that might shift what is the best thing(s) to do about it)
I agree that critics engaging with arguments badly is an update towards the arguments being real, but I am essentially claiming that the fact that this selection effect exists and is very difficult to eliminate/reduce to a useful level implicitly means that you can only get a very limited amount of evidence from arguments.
One particular part of my model here is that selection effects are usually very strong and difficult to eliminate by default, unfortunately, and thus one of the central problems of science in general is how to deal with this sort of effect.
But it’s nice to hear from you on how you’ve come to believe in AI risk being a big deal.
Edit: I wrote a linkpost on the main way people turn ideologically crazy that explains why you can only get a very limited amount of evidence from arguments, and retracted the statement that “I agree that critics engaging with arguments badly is an update towards the arguments being real”.
This seems like it’s engaging with the question of “what do critics think?” in a sort of model-free, uninformed, “who to defer to” sort of way.
For awhile, I didn’t fully update on arguments for AI Risk being a Big Deal because the arguments were kinda complex and I could imagine clever arguers convincing me of it without it being true. One of the things that updated me over the course of 4 years was actually reading the replies (including by people like Hanson) and thinking “man, they didn’t seem to even understand or address the main points.”
i.e. it’s not that they didn’t engage with the arguments, it’s that they engaged with the arguments badly which lowered my credence on taking their opinion seriously.
(I think nowadays I have seen some critics who do seem to me to have engaged with most of the real points. None of their counterarguments seem like they’ve added up to “AI is not a huge fucking deal that is extremely risky” in a way that makes any sense to me, but, some of them add up to alternate frames of looking at the problem that might shift what is the best thing(s) to do about it)
I agree that critics engaging with arguments badly is an update towards the arguments being real, but I am essentially claiming that the fact that this selection effect exists and is very difficult to eliminate/reduce to a useful level implicitly means that you can only get a very limited amount of evidence from arguments.One particular part of my model here is that selection effects are usually very strong and difficult to eliminate by default, unfortunately, and thus one of the central problems of science in general is how to deal with this sort of effect.
But it’s nice to hear from you on how you’ve come to believe in AI risk being a big deal.
Edit: I wrote a linkpost on the main way people turn ideologically crazy that explains why you can only get a very limited amount of evidence from arguments, and retracted the statement that “I agree that critics engaging with arguments badly is an update towards the arguments being real”.