I think the research was presented fine; it didn’t blame people for talking about the possibility of misaligned AIs. In fact they specifically talk about how they are using this info to better align AIs, which seems like the opposite of using hyperstition as a lazy way to blame doomers for bad alignment.
On the other hand, in The Adolescence of Technology, Dario says this about doomerism:
Here,I mean “doomerism” not just in the sense of believing doom is inevitable (which is both a false and self-fulfilling belief), but more generally, thinking about AI risks in a quasi-religious way.
I agree this in particular and the piece in general is too dismissive of hard AI risk concerns. I don’t really know if it’s accusing them of hyperstitioning; the self-fulfilling belief could also refer to, e.g., attempts by doomers to get everyone who cares about AI risk to quit working on AI (thereby ceding the field to the least risk-aware people) which I think is a real problem.
Either way, I agree that people concerned about future misaligned AIs should be free to voice those concerns without being accused of hyperstitioning them.
Yeah, to be clear—as I concede—they didn’t blame people for writing about misaligned AI. But this was seemingly what a lot of people took from it. I don’t know why they had to word this tweet this way, I don’t feel like it’s substantiated by the post and the research.
the self-fulfilling belief could also refer to, e.g., attempts by doomers to get everyone who cares about AI risk to quit working on AI
I hadn’t thought of that, though calling an idea a self-fulfilling believe is almost the same as calling it a hyperstition per definition. But through the article I treat hyperstition more narrowly as meaning: Writing about misalignment risk causing misalignment risk.
Yeah you are correct that the straightforward interpretation of self-fulfilling is hyperstition, which I agree isn’t a fair accusation.
Although if we do get self-fulfilling doom through hyperstition I do think it means that the misaglinment people were wrong in an important sense (AI psychology differed from their view enough that misalignment happened through such a weird method rather than straightforward instrumental convergence + orthogonality).
I think the research was presented fine; it didn’t blame people for talking about the possibility of misaligned AIs. In fact they specifically talk about how they are using this info to better align AIs, which seems like the opposite of using hyperstition as a lazy way to blame doomers for bad alignment.
On the other hand, in The Adolescence of Technology, Dario says this about doomerism:
I agree this in particular and the piece in general is too dismissive of hard AI risk concerns. I don’t really know if it’s accusing them of hyperstitioning; the self-fulfilling belief could also refer to, e.g., attempts by doomers to get everyone who cares about AI risk to quit working on AI (thereby ceding the field to the least risk-aware people) which I think is a real problem.
Either way, I agree that people concerned about future misaligned AIs should be free to voice those concerns without being accused of hyperstitioning them.
Yeah, to be clear—as I concede—they didn’t blame people for writing about misaligned AI. But this was seemingly what a lot of people took from it. I don’t know why they had to word this tweet this way, I don’t feel like it’s substantiated by the post and the research.
I hadn’t thought of that, though calling an idea a self-fulfilling believe is almost the same as calling it a hyperstition per definition. But through the article I treat hyperstition more narrowly as meaning: Writing about misalignment risk causing misalignment risk.
Yeah you are correct that the straightforward interpretation of self-fulfilling is hyperstition, which I agree isn’t a fair accusation.
Although if we do get self-fulfilling doom through hyperstition I do think it means that the misaglinment people were wrong in an important sense (AI psychology differed from their view enough that misalignment happened through such a weird method rather than straightforward instrumental convergence + orthogonality).