I have the feeling that this post is getting close to a critical point, but then doesn’t quite express it.
(At the moment, I can’t quite express it myself, so I can’t complain much.)
I think I want an essay that is really making the case for this:
Here’s another danger, which I think may be worse. If you insist on working somewhere x-risk-themed, you’re asking for someone to make you a sucker.
Musing a bit on it...
There’s something like, “the desire to help” / “the desire to be important” is an attack surface.
A lot of people want to help, but figuring out how to help is actually really hard to do. Many things to try are worthless, many more are actively harmful. Strategic thinking is hard and often annoying, and is just the kind of thing that even pretty smart people aren’t cut out for.
A lot of people (including myself) try to slot themselves into roles where they can be a force-multiplier for some strategic vision, that makes sense to them, but which they couldn’t really defend from incisive critique.
I think a lot of EA-ish people treat this as a kind of neutral operation, where they’re straightforwardly glad to have an opportunity to have impact, rather than a kind of fraught transaction where one party is offering their agency and another is offering their strategic orientation / impact opportunity.
When this works well, both parties get to work together to do more good in the world than either party could alone. Which is great!
But this transaction is fundamentally one that presupposes an information asymmetry. In order for this transaction to make any sense, the party offering their agency has to have less strategic discernment than the party offering strategic orientation.
So this setup is ripe ground for scams.
I suppose the general pattern is “if someone really wants something, X, but they have a weak ability to discern if they are achieving it or not, there is an incentive gradient towards scams that make them feel like they’re getting X.”
My ideal version of your hypothetical post probably also talks about illegible problems—I think there’s a kind of cursed dynamic where it’s easy for people to gather around more legible directions and then there’s more social momentum behind them, even if they’re not that helpful. Conversely, the dynamics described in the above post pull towards being not just x-risk-themed but visibly x-risk-themed. You don’t get to go to constellation just because you think your work is important.
I have the feeling that this post is getting close to a critical point, but then doesn’t quite express it.
(At the moment, I can’t quite express it myself, so I can’t complain much.)
I think I want an essay that is really making the case for this:
Musing a bit on it...
There’s something like, “the desire to help” / “the desire to be important” is an attack surface.
A lot of people want to help, but figuring out how to help is actually really hard to do. Many things to try are worthless, many more are actively harmful. Strategic thinking is hard and often annoying, and is just the kind of thing that even pretty smart people aren’t cut out for.
A lot of people (including myself) try to slot themselves into roles where they can be a force-multiplier for some strategic vision, that makes sense to them, but which they couldn’t really defend from incisive critique.
I think a lot of EA-ish people treat this as a kind of neutral operation, where they’re straightforwardly glad to have an opportunity to have impact, rather than a kind of fraught transaction where one party is offering their agency and another is offering their strategic orientation / impact opportunity.
When this works well, both parties get to work together to do more good in the world than either party could alone. Which is great!
But this transaction is fundamentally one that presupposes an information asymmetry. In order for this transaction to make any sense, the party offering their agency has to have less strategic discernment than the party offering strategic orientation.
So this setup is ripe ground for scams.
I suppose the general pattern is “if someone really wants something, X, but they have a weak ability to discern if they are achieving it or not, there is an incentive gradient towards scams that make them feel like they’re getting X.”
As an additional point, there are also potentially quite significant negative externalities to slotting in—because you validate the strategic vision, which makes it able to pick up other slotters. (Cf. https://tsvibt.blogspot.com/2024/09/the-moral-obligation-not-to-be-eaten.html )
I wrote a little bit about suckerhood dynamics here. I want to expand it and then crosspost to LessWrong.
Your framing is very reminiscent of this post by John Wentworth.
My ideal version of your hypothetical post probably also talks about illegible problems—I think there’s a kind of cursed dynamic where it’s easy for people to gather around more legible directions and then there’s more social momentum behind them, even if they’re not that helpful. Conversely, the dynamics described in the above post pull towards being not just x-risk-themed but visibly x-risk-themed. You don’t get to go to constellation just because you think your work is important.