This thought was inspired by reading this Ben Hoffman’s recent post, and more proximally by this Wei Dai shortfrom, which claims that approximately no one exhibits long horizon agency.
You might think that means “almost no one except these EAs and rationalists”. But, since it’s easy to use “optimizing for the long term future” as a cope to hide from accountability, I suggest that many (most?) of the people who appear to be exerting long-horizon agency are using that as a cover and less actually steering the future.
You might think that means “almost no one except these EAs and rationalists”.
I guess I didn’t express this in that shortform, but I meant to include EAs and rationalists in “approximately does not exist”. Examples: “philosophers don’t bother to think about long term implications of AI on philosophy production (positive or negative)” includes philosophers in EA who never talked about this until very recently (with founding of Forethought) and even now talk about this to a much lesser extent than I expect or think they should, and Eliezer making what I think are serious strategic mistakes earlier in his career.
About the “missing mood”, what do you think about this post of mine? I feel like the solution to the problem that we’re talking about isn’t to do less long-horizon agency/strategy, but to do more of it but more tentatively, slowly building up traditions/methodologies/foundations, kind of like how philosophy has progressed over time (but hopefully avoiding the destructive overconfidence of past philosophers). If we simply do it less, then how does civilization ever get to a state where we’re navigating the future with strategic competence?
I’ve been thinking lately about the attractors that develop when a person or community has spent some time thinking on something (enough to grow attached to their thoughts). It takes real effort to just waveat concepts you’ve seen before and then keep driving, whenever something you’re thinking about might roundto something more familiar.
Especially if the conceit of one’s inquiry is ‘I suspect we are very wrong about very fundamental things’, there are many alternatives and objections that lie along the path, which themselves dead-end in unsatisfying places. Walking down each once is fine, but I find it wearying to trod down the same handful of knee jerk off-ramps over and over again, every time I share an idea with a new interlocutor.
The slower feedback loops of academic life seem to help here; once you have some cache, you can more or less hole up and work on your maybe-insane idea for the rest of your life, with very limited accountability. This, of course, creates other problems, the enumeration of which is its own well-worn off-ramp.
I think that the rapid advance of AI is correctly prompting a taking of inventory, a moment of circum/introspection, to get a sense of whether or not we’re ‘ready’, and most people who’ve genuinely engaged with pre-existing literature that attempted to grapple with these possibilities answer that question with a resounding ‘NO’, but we’ve not yet seen a shift to a more butterfly-nurturing temperament, and we’ve not yet reached consensus on quite how far we ought to back up, which makes a lot of people who consider themselves to have backed up wince a bit when someone later says ‘no, further’.
It seems like you’re the person who’s over and over again saying ‘no, further’, and many people just hear the local ‘no’ to their particular idea, struggling to think that there’s any further to back up.
[fwiw, I currently think that we need to back up/zoom out/pass off-ramps, and then uh… keep doing that for a long while]
Humans are not automagically strategic. Failing at long timehorizon strategy is the default for humans because it requires thinking correctly which people fail at by default without feedback loops. No cope required. You can do real scarry things with feedback loops as your full-time job and do the long-term thinking as your hobby and side-hustle. Humans have timehorizons just like AI, but humans are trainable just like AI.
You can actually have a large impact over time by focusing on strategic moves, and also be running from tactical day to day operational accountability, and using the former to cover for the latter.
Should we stop attempting because it’s hard? Or notice cope and cover and correct it?
Not necessarily, but we should maybe be less gung ho about “trying to do the most good”.
It feels like there’s maybe a missing mood to a lot of EA and x-risk related efforts. Probably more people should be more paranoid about the impacts of their actions, and more willing to do ostensibly less ambitious things that are less fake to them?
This thought was inspired by reading this Ben Hoffman’s recent post, and more proximally by this Wei Dai shortfrom, which claims that approximately no one exhibits long horizon agency.
You might think that means “almost no one except these EAs and rationalists”. But, since it’s easy to use “optimizing for the long term future” as a cope to hide from accountability, I suggest that many (most?) of the people who appear to be exerting long-horizon agency are using that as a cover and less actually steering the future.
I guess I didn’t express this in that shortform, but I meant to include EAs and rationalists in “approximately does not exist”. Examples: “philosophers don’t bother to think about long term implications of AI on philosophy production (positive or negative)” includes philosophers in EA who never talked about this until very recently (with founding of Forethought) and even now talk about this to a much lesser extent than I expect or think they should, and Eliezer making what I think are serious strategic mistakes earlier in his career.
About the “missing mood”, what do you think about this post of mine? I feel like the solution to the problem that we’re talking about isn’t to do less long-horizon agency/strategy, but to do more of it but more tentatively, slowly building up traditions/methodologies/foundations, kind of like how philosophy has progressed over time (but hopefully avoiding the destructive overconfidence of past philosophers). If we simply do it less, then how does civilization ever get to a state where we’re navigating the future with strategic competence?
I’ve been thinking lately about the attractors that develop when a person or community has spent some time thinking on something (enough to grow attached to their thoughts). It takes real effort to just wave at concepts you’ve seen before and then keep driving, whenever something you’re thinking about might round to something more familiar.
Especially if the conceit of one’s inquiry is ‘I suspect we are very wrong about very fundamental things’, there are many alternatives and objections that lie along the path, which themselves dead-end in unsatisfying places. Walking down each once is fine, but I find it wearying to trod down the same handful of knee jerk off-ramps over and over again, every time I share an idea with a new interlocutor.
The slower feedback loops of academic life seem to help here; once you have some cache, you can more or less hole up and work on your maybe-insane idea for the rest of your life, with very limited accountability. This, of course, creates other problems, the enumeration of which is its own well-worn off-ramp.
I think that the rapid advance of AI is correctly prompting a taking of inventory, a moment of circum/introspection, to get a sense of whether or not we’re ‘ready’, and most people who’ve genuinely engaged with pre-existing literature that attempted to grapple with these possibilities answer that question with a resounding ‘NO’, but we’ve not yet seen a shift to a more butterfly-nurturing temperament, and we’ve not yet reached consensus on quite how far we ought to back up, which makes a lot of people who consider themselves to have backed up wince a bit when someone later says ‘no, further’.
It seems like you’re the person who’s over and over again saying ‘no, further’, and many people just hear the local ‘no’ to their particular idea, struggling to think that there’s any further to back up.
[fwiw, I currently think that we need to back up/zoom out/pass off-ramps, and then uh… keep doing that for a long while]
Humans are not automagically strategic. Failing at long timehorizon strategy is the default for humans because it requires thinking correctly which people fail at by default without feedback loops. No cope required. You can do real scarry things with feedback loops as your full-time job and do the long-term thinking as your hobby and side-hustle. Humans have timehorizons just like AI, but humans are trainable just like AI.
Two things can be true.
You can actually have a large impact over time by focusing on strategic moves, and also be running from tactical day to day operational accountability, and using the former to cover for the latter.
Interesting. This I agree with. But it seems to apply to those doing more easily evaluate technical work at least as much.
Should we stop attempting because it’s hard? Or notice cope and cover and correct it?
Not necessarily, but we should maybe be less gung ho about “trying to do the most good”.
It feels like there’s maybe a missing mood to a lot of EA and x-risk related efforts. Probably more people should be more paranoid about the impacts of their actions, and more willing to do ostensibly less ambitious things that are less fake to them?
I guess I’m not understanding what mood you think is missing.
I’m not super clear on what the ideal response to this state of affairs is. I’m both confused and probably wrong.
But does “paranoia” not evoke the difference?