How did we get here? How did Rationality go from being about winning at life to a Cthullu RPG? I’m pretty interested in AI, and the fact that stuff like this is real (it still doesn’t feel real, it feels like LARP) has really turned me off of alignment research. On a personal level, the literally-having-coworkers-get-driven-insane of alignment research seems a lot worse than merely-slightly-increasing-existential-risk of capabilities research.
I think this is maybe implied by Aiyen’s comment, but to highlight an element here:
This way of thinking doesn’t have you trade sanity for slight xrisk decrease.
It has you trading sanity for perceived slight xrisk decreases.
If you start making those trades, over time your ability to perceive what’s real decays.
If there’s anything that has anything whatsoever like agency and also benefits from you feeding your sanity into a system like this, it’ll exploit you right into a Hell of your own making.
Any time you’re caught in a situation that asks for you to make a trade like this, it’s worth asking if you’ve somehow slipped into the thrall of an influence like this.
It’s way more common than you might think at first.
On the one hand, that’s literally true. On the other, I feel like the connotations are dangerous. Existential risk is one of the worst possible things, and nearly anything is better than slightly increasing it. However, we should be careful that that mindset doesn’t lead us into Pascal’s Muggings and/or burnout. We certainly aren’t likely to be able to fight existential risk if it drives us insane!
I strongly suspect that it’s not self-sacrificing researchers who will solve alignment and bring us safely through the current crisis, but ones who are able to address the situation calmly and without freaking out, even though freaking out seems potentially justified.
Eliezer created LW in order to sort of spell out his understanding of the Art.
Part of the point of that was to act as a kind of combo summon spell and filter for potential AI risk researchers for MIRI.
The reason was in the wake of Eliezer noticing he had been ludicrously wrong in his thinking about AI, and noticing how hard it is to think about AI clearly, and how AI researchers in particular couldn’t think clearly about the topic clearly.
So the “This is literally unfathomably important, we have to actually solve this thinking clearly thing or literally everything anyone could ever value is going to get permanently destroyed!” tone was woven in from the very beginning.
This is just one example.
So, yeah. It’s always been like this. Just varying degrees of cloaked vs. transparent, with corresponding peripheral folk sometimes floating in who aren’t as much about AI risk.
But the thrust as a whole has always come from this Lovecraftian core.
It was always a Cthulhu LARP. Remember that one thing?
Groups polarize over time. One of the ways to signal group membership is to react explosively to the things you’re supposed to react explosively to. This is why as politics in the US have polarized, everyone has grown more breathless and everything is ${OUTGROUP}ist. You gain ${KARMA} by being more breathless and emotional. You can only stop this with explicit effort, but if you do that, you look dissociated, disembodied, autistic, and the social pressure against that is stronger than the social pressure against letting (totally normal, status-quo) group polarization get to the point of literally mindkilling people.
The military also has techniques to avoid this, but there are similar social pressures against them from both sides of the equation, because the military is low-status both to the postrationalist types (like OP) and to actual rationalists (like the people setting social norms at CFAR). So you don’t see those being used.
It’s completely possible to have the alignment research you want, you’ll just have to set the group norms, which is a completely orthogonal problem that involves a lot of things besides alignment research. Personally I think this would be a very good thing to do and I encourage you to try for the experience of it if nothing else.
I completely agree and I think that levying the charges “disembodied” against anything on the opposite side of the mental dichotomy of “woo” is a weasel-word for the ableist slur of “autistic.” I’m sorry this wasn’t more clear, but I thought that sentence was fairly dripping as is. I’ve written about this before as it applies to this topic, which is not to excuse the harm I’ve done if I’ve triggered you, but to show that I’ve precommitted to this stance on this issue.
I feel slightly nervous I am overtly spending internet ink on this but here I go.
I was unsure how obvilously wrong it was ment to appear. To me saying that “someone seems cold” is not problematic. “dissociated” and “disembodied” read to me to be part of natural feeling (expression), somebody could mean technical things with them and not have a attitude loaded into them. Those parts did not constitute drippingness for me. For autistic there was no technical meaning that could make sense.
I was not triggered but it did cross my mind that not everyone thinks that is unbased take to me (and kind of categorising this knowledge as common only in the small subcommunity). Having those fly without anybody batting an eyelid would be normalising the hateful conduct. I was unsure whether the eyelid was batted already so I batted a separate eyelid. And tried to include indicators that it is a mild reaction (a kind of messaging I have reason to believe I frequently screw up).
I do reflect that if I was in the context where this could be assumed common knowledge I would not probably be making this move (bat the eyelid). So I am wondering whether it connects to the object level phenomena (of groups polarising) where people harden their signals so that outsiders with more noise and less decoding ability do not get unintended messages.
Lots of ink, but lots to think about. I’m thankful for this post fwiw.
The “no technical meaning” could maybe be an indicator of sarcasm. But you’re right that there was no way for you to know I wasn’t just misapplying the term in the same way as the OP.
I don’t think this relates to group polarization per se but I take your point.
I didn’t mean “triggered” to mean extremely so, someone can be mildly triggered and again, I apologize for (in my perception, based on your comment) doing that. I think you did the right thing.
With no resonable way of knowing without context I am using “technical” here in a very idiosyncratic way. If two speech acts that have very different connotations and then strip them of the connotations if they are the same then the technical meaning is the same.
If someone is being hateful I often proceed to “fix the message from them” mentally in my receiving end. So while I starkly reject parts of it, rejecting everything of it punishes also the non-hateful parts. Thus I have the cognitive task of “what they should have said”. If there is no innocent message left after removing the hate, it is pure hate. This is a kind of “could a reasonable opiner opine this?” standard. It is easy to read “disembodied” in a ablist way but it might just be a clumsy way to refer to low charisma (is is “repairable”). So after phrasing incompetence is exhausted an assumption of malice starts.
To have the statistical mean human deduce “That guy gets passionate in an unnatural way → that guy is autistiic” has low plausibility. Backtracing where this logic would be natural, worrying about upholding a mask about a behaviour that has lots of details and has high fluency from the mimic target making it highly likely to be a statistical outlier that a masking strategy does not cover well (this is not meant to be a mask review). Confusion, “stiffness” or “odd feeling” would represent what happens in situations like these. Zero to 100% autistic label is irrealistic. The average hater is not that informed.
How did we get here? How did Rationality go from being about winning at life to a Cthullu RPG? I’m pretty interested in AI, and the fact that stuff like this is real (it still doesn’t feel real, it feels like LARP) has really turned me off of alignment research. On a personal level, the literally-having-coworkers-get-driven-insane of alignment research seems a lot worse than merely-slightly-increasing-existential-risk of capabilities research.
I’d rather go insane than “slightly increase” existential risk.
I think this is maybe implied by Aiyen’s comment, but to highlight an element here:
This way of thinking doesn’t have you trade sanity for slight xrisk decrease.
It has you trading sanity for perceived slight xrisk decreases.
If you start making those trades, over time your ability to perceive what’s real decays.
If there’s anything that has anything whatsoever like agency and also benefits from you feeding your sanity into a system like this, it’ll exploit you right into a Hell of your own making.
Any time you’re caught in a situation that asks for you to make a trade like this, it’s worth asking if you’ve somehow slipped into the thrall of an influence like this.
It’s way more common than you might think at first.
On the one hand, that’s literally true. On the other, I feel like the connotations are dangerous. Existential risk is one of the worst possible things, and nearly anything is better than slightly increasing it. However, we should be careful that that mindset doesn’t lead us into Pascal’s Muggings and/or burnout. We certainly aren’t likely to be able to fight existential risk if it drives us insane!
I strongly suspect that it’s not self-sacrificing researchers who will solve alignment and bring us safely through the current crisis, but ones who are able to address the situation calmly and without freaking out, even though freaking out seems potentially justified.
It’s woven deep in LW’s memetic history.
Eliezer created LW in order to sort of spell out his understanding of the Art.
Part of the point of that was to act as a kind of combo summon spell and filter for potential AI risk researchers for MIRI.
The reason was in the wake of Eliezer noticing he had been ludicrously wrong in his thinking about AI, and noticing how hard it is to think about AI clearly, and how AI researchers in particular couldn’t think clearly about the topic clearly.
So the “This is literally unfathomably important, we have to actually solve this thinking clearly thing or literally everything anyone could ever value is going to get permanently destroyed!” tone was woven in from the very beginning.
This is just one example.
So, yeah. It’s always been like this. Just varying degrees of cloaked vs. transparent, with corresponding peripheral folk sometimes floating in who aren’t as much about AI risk.
But the thrust as a whole has always come from this Lovecraftian core.
turns out life is a Cthullu RPG, so we gotta win at that
It was always a Cthulhu LARP. Remember that one thing?
Groups polarize over time. One of the ways to signal group membership is to react explosively to the things you’re supposed to react explosively to. This is why as politics in the US have polarized, everyone has grown more breathless and everything is ${OUTGROUP}ist. You gain ${KARMA} by being more breathless and emotional. You can only stop this with explicit effort, but if you do that, you look dissociated, disembodied, autistic, and the social pressure against that is stronger than the social pressure against letting (totally normal, status-quo) group polarization get to the point of literally mindkilling people.
The military also has techniques to avoid this, but there are similar social pressures against them from both sides of the equation, because the military is low-status both to the postrationalist types (like OP) and to actual rationalists (like the people setting social norms at CFAR). So you don’t see those being used.
It’s completely possible to have the alignment research you want, you’ll just have to set the group norms, which is a completely orthogonal problem that involves a lot of things besides alignment research. Personally I think this would be a very good thing to do and I encourage you to try for the experience of it if nothing else.
This is a real stigma consideration. Reminder that water is water. Autism as something inherently negative is ablism and improper.
I completely agree and I think that levying the charges “disembodied” against anything on the opposite side of the mental dichotomy of “woo” is a weasel-word for the ableist slur of “autistic.” I’m sorry this wasn’t more clear, but I thought that sentence was fairly dripping as is. I’ve written about this before as it applies to this topic, which is not to excuse the harm I’ve done if I’ve triggered you, but to show that I’ve precommitted to this stance on this issue.
I feel slightly nervous I am overtly spending internet ink on this but here I go.
I was unsure how obvilously wrong it was ment to appear. To me saying that “someone seems cold” is not problematic. “dissociated” and “disembodied” read to me to be part of natural feeling (expression), somebody could mean technical things with them and not have a attitude loaded into them. Those parts did not constitute drippingness for me. For autistic there was no technical meaning that could make sense.
I was not triggered but it did cross my mind that not everyone thinks that is unbased take to me (and kind of categorising this knowledge as common only in the small subcommunity). Having those fly without anybody batting an eyelid would be normalising the hateful conduct. I was unsure whether the eyelid was batted already so I batted a separate eyelid. And tried to include indicators that it is a mild reaction (a kind of messaging I have reason to believe I frequently screw up).
I do reflect that if I was in the context where this could be assumed common knowledge I would not probably be making this move (bat the eyelid). So I am wondering whether it connects to the object level phenomena (of groups polarising) where people harden their signals so that outsiders with more noise and less decoding ability do not get unintended messages.
Lots of ink, but lots to think about. I’m thankful for this post fwiw.
The “no technical meaning” could maybe be an indicator of sarcasm. But you’re right that there was no way for you to know I wasn’t just misapplying the term in the same way as the OP.
I don’t think this relates to group polarization per se but I take your point.
I didn’t mean “triggered” to mean extremely so, someone can be mildly triggered and again, I apologize for (in my perception, based on your comment) doing that. I think you did the right thing.
With no resonable way of knowing without context I am using “technical” here in a very idiosyncratic way. If two speech acts that have very different connotations and then strip them of the connotations if they are the same then the technical meaning is the same.
If someone is being hateful I often proceed to “fix the message from them” mentally in my receiving end. So while I starkly reject parts of it, rejecting everything of it punishes also the non-hateful parts. Thus I have the cognitive task of “what they should have said”. If there is no innocent message left after removing the hate, it is pure hate. This is a kind of “could a reasonable opiner opine this?” standard. It is easy to read “disembodied” in a ablist way but it might just be a clumsy way to refer to low charisma (is is “repairable”). So after phrasing incompetence is exhausted an assumption of malice starts.
To have the statistical mean human deduce “That guy gets passionate in an unnatural way → that guy is autistiic” has low plausibility. Backtracing where this logic would be natural, worrying about upholding a mask about a behaviour that has lots of details and has high fluency from the mimic target making it highly likely to be a statistical outlier that a masking strategy does not cover well (this is not meant to be a mask review). Confusion, “stiffness” or “odd feeling” would represent what happens in situations like these. Zero to 100% autistic label is irrealistic. The average hater is not that informed.