As the funder that you are very likely referring to, I do want to highlight that I don’t feel like this summarizes my views particularly well. In particular this section:
EA really does seem to be missing a management layer. People are thinking about their careers, starting organisations, doing direct work and research. Not many people are drawing up plans for coordination on a higher level and telling people what to do. Someone ought to be dividing up the big picture into roles for people to fill. You can see the demand for this by how seriously we take 80k. They’re the only ones doing this beyond the organisational level.
Much the same in the cause area we call AI Safety Education. Most AIS organisations are necessarily thinking about hiring and training, but no one is specializing in it. In the coming year, our aim is to fill this niche, building expertise and doing management consulting. We will aim to smarten up the coordination there. Concrete outputs might be:
+ Advice for grantmakers that want to invest in the AI Safety researcher pipeline
+ Advice for students that want to get up to speed and test themselves quickly
+ Suggesting interventions for entrepreneurs that want to fill up gaps in the ecosystem
+ Publishing thinkpieces that advance the discussion of the community, like this one
+ Creating and keeping wiki pages about subjects that are relevant to us
+ Helping AIS research orgs with their recruitment process
I think in general people should be very hesitant to work on social coordination problems because they can’t find a way to make progress on the object-level problems. My recommendation was very concretely “try to build an internal model of what really needs to happen for AI-risk to go well” and very much not “try to tell other people what really needs to happen for AI-risk”, which is almost the exact opposite.
I actually think going explicitly in this direction is possibly worse than RAISE’s previous plans. One of my biggest concerns with RAISE was precisely that it was trying far too early to tell people what exactly to learn and what to do, without understanding the relevant problems themselves first. This seems like it exacerbates that problem by trying to make your job explicitly about telling other people what to do.
This works for versions of “do something” that mainly interact with objective reality, but there’s a pretty awful value-misalignment problem if the way you figure out what works is through feedback from social reality.
So, for instance, learning to go camping or cook or move your body better or paint a mural on your wall might count, but starting a socially legible project may be actively harmful if you don’t have a specific need that’s meeting that you’re explicitly tracking. And unfortunately too much of people’s idea of what “go do something” ends up pointing to trying to collect credit for doing things.
Sitting somewhere doing nothing (which is basically what much meditation is) is at least unlikely to be harmful, and while of limited use in some circumstances, often an important intermediate stage in between trying to look like you’re doing things, and authentically acting in the world.
Ray:
It’s been said before for sure, but worth saying periodically.
Something I’d add, which particularly seems like the failure mode I see in EA-spheres (less in rationalist spheres but they blur together)
Try to do somethingother thansolve coordination problems.
Or, try to do something that providesimmediate value to whoever uses it, regardless of whether other people are also using it.
A failure mode I see (and have often fallen to) is looking around and thinking “hmm, I don’t know how to do something technical, and/or I don’t have the specialist skills necessary to do something specialist. But, I can clearly see problems that stem from people being uncoordinated. I think I roughly know how people work, and I think I can understand this problem, so I will work on that.”
But:
+ It actually requires just as much complex specialist knowledge to solve coordination problems as it does to do [whatever other thing you were considering].
+ Every time someone attempts to rally people around a new solution, and fails, they make it harder for the next person who tries to rally people around a new solution. This makes the coordination system overall worse.
This is a fairly different framing than Benquo’s (and Eliezer’s) advice, although I think it amounts to something similar.
My recommendation was very concretely “try to build an internal model of what really needs to happen for AI-risk to go well”
I’m not sure anyone knows what really needs to happen for AI-risk to go well, including people who have been thinking about this question for many years. Do you really mean for RAISE to solve this problem, or just to think about this question for more than they already have, or to try to learn the best available model from someone else (if so who)?
Mostly think more about this question than they already have, which likely includes learning the best available models from others.
The critique here was more one of intention than one of epistemic state. It seems to me like there is a mental motion of being curious about how to make progress on something, even if one is still confused, which I contrast with a mental motion of “trying to look like you are working on the problem”.
Ah ok. Given that, it seems like you need to explain your critique more, or try to figure out the root cause of the wrong intention and address that, otherwise wouldn’t they just switch to “trying to look like you’re trying to build models of what needs to be done to solve AI risk”?
Another problem is that it seems even harder to distinguish between people who are really trying to build such models, and people who are just trying to look like they’re doing that, because there’s no short-term feedback from reality to tell you whether someone’s model is any good. It seems like suggesting people to do that when you’re not sure of their intention is really dangerous, as it could mess up the epistemic situation with AI risk models (even more than it already is). Maybe it would be better to just suggest some concrete short-term projects for them to do instead?
Maybe there is a possible project in this direction. I’ll assume that this is general advice you’d give to many people who want to work in this space. If it is important for people to build a model of what is required for AI to go well then people may as well work on this together. And sure there’s websites like Less Wrong, but people can exchange information much faster by chatting either in person or over Skype. (Of course there are worries that this might lead to overly correlated answers)
Looks like I didn’t entirely succeed in explaining our plan.
My recommendation was very concretely “try to build an internal model of what really needs to happen for AI-risk to go well” and very much not “try to tell other people what really needs to happen for AI-risk”, which is almost the exact opposite.
And that’s also what we meant. The goal isn’t to just give advice. The goal is to give useful and true advice, and this necessarily requires a model of what really needs to happen for AI risk.
We’re not just going to spin up some interesting ideas. That’s not the mindset. The mindset is to generate a robust model and take it from there, if we ever get that far.
We might be talking to people in the process, but as long as we are in the dark the emphasis will be on asking questions.
EDIT: this wasn’t a thoughtful reply. I take it back. See Ruby’s comments below
The goal is to give useful and true advice, and this necessarily requires a model of what really needs to happen for AI risk.
My understanding of habryka, and the position I’d agree with, is that the goal should be solely to be forming a model at this time. As currently stated, even before you’ve formed the model, there is a presumption that once you have a model the right thing to do is then instruct others. That is premature.
The mindset is to generate a robust model and take it from there, if we ever get that far.
That kind of seems good, but that’s not what your plan says. The plan as written states that a management layer is required—that seems like possibly a conclusion you reach after you’ve done your model building stage, yet this plan states up front. That seems like a good recipe for confirmation bias to me.
I think there’s an alternative version of this point, and a good underlying procedure, which is that along the way of forming a model of things you generate various hypotheses. That the EA community needs a management layer or someone to connect the dots, etc., is maybe a reasonable hypothesis / proto-model of the situation. It’s something you could possibly post about and solicit feedback and ideas from others and talk to about. That seems good and it seems good if you can generate that hypothesis plus other hypotheses (caveats being, as habryka stated, you should likely start with non-coordination work).
An alternative version of your plan which might receive better reception from many is purely research. “We are going to research questions X and Y.” You might have the knack for the that, and I could see it getting funded if you had a good agenda/good plan and seemed your questioning wouldn’t disrupt or displace other work + others weren’t worried you were going to pivot it into something counterproductive.
Sorry, I have to add one more piece that really seems worth calling out.
Looks like I didn’t entirely succeed in explaining our plan.
>”My recommendation was very concretely “try to build an internal model of what really needs to happen for AI-risk to go well” and very much not “try to tell other people what really needs to happen for AI-risk”, which is almost the exact opposite.”
And that’s also what we meant. The goal isn’t to just give advice. The goal is to give useful and true advice, and this necessarily requires a model of what really needs to happen for AI risk.
If someone’s position is “very much don’t try to tell other people what really needs to happen for AI-risk”, you cannot claim that you meant the same thing when your position includes “the goal is to give useful and true advice”. You don’t also mean what Habryka means at all (you’re ignoring it) and it strikes me as disingenuous to claim that you mean the same thing thing as him at all.
More formally:
Person A: “X, definitely not Y”
Person B: “X and Y”
Person B is absolutely not saying the same thing as A.
Additional thought, just as an example, something in the post that seemed off:
Instead, someone should be thinking about the bigger picture first.
Researchers and people working at the various organizations are clever, capable people. I would venture that they’re necessarily thinking about the big picture as part of their work and how it fits in. Given their connection and involvement (possibly over the course of years), their models are probably difficult to surpass if you’re starting from the outside. I also don’t think it’s as simple as talking to lots of people to get all their models and combining them (communicating and synthesizing models is really hard + you’d need to build a lot of credibility before others trusted you could, assuming anyone could really do this well).
The line quoted above, as written, almost makes seems like there’s this low hanging fruit because everyone currently working was heads-down on their problems and no one thought to work on the big picture. That strikes me as a very bad assumption and makes me worry about the kind of reasoning you would use to advise others. Possibly you meant something different from that and more defensible . . . but then unambiguous and clear communication is going to be very key in any coordination/advisory role.
Anyhow, I dislike being purely critical. Not many people are dedicating themselves to trying to solve important problems, so I want to say that I do approve of efforts trying. I think it’s good that you sought feedback on your first project and then formed a new plan. I’ve written these comments because I hope they help nudge you in the direction of really good plans. If they’re biased critical, it’s because I’m trying to explain more of factors I think might be leading to a negative reception of this plan. Because we need all the good plans and good people working on them we can get.
As the funder that you are very likely referring to, I do want to highlight that I don’t feel like this summarizes my views particularly well. In particular this section:
I think in general people should be very hesitant to work on social coordination problems because they can’t find a way to make progress on the object-level problems. My recommendation was very concretely “try to build an internal model of what really needs to happen for AI-risk to go well” and very much not “try to tell other people what really needs to happen for AI-risk”, which is almost the exact opposite.
I actually think going explicitly in this direction is possibly worse than RAISE’s previous plans. One of my biggest concerns with RAISE was precisely that it was trying far too early to tell people what exactly to learn and what to do, without understanding the relevant problems themselves first. This seems like it exacerbates that problem by trying to make your job explicitly about telling other people what to do.
A lot of my thoughts in this space are summarized by the discussion around Davis’ recent post “Go Do Something”, in particular Ray’s and Ben Hoffman’s comments about working on social coordination technology:
Benquo:
Ray:
I’m not sure anyone knows what really needs to happen for AI-risk to go well, including people who have been thinking about this question for many years. Do you really mean for RAISE to solve this problem, or just to think about this question for more than they already have, or to try to learn the best available model from someone else (if so who)?
Mostly think more about this question than they already have, which likely includes learning the best available models from others.
The critique here was more one of intention than one of epistemic state. It seems to me like there is a mental motion of being curious about how to make progress on something, even if one is still confused, which I contrast with a mental motion of “trying to look like you are working on the problem”.
Ah ok. Given that, it seems like you need to explain your critique more, or try to figure out the root cause of the wrong intention and address that, otherwise wouldn’t they just switch to “trying to look like you’re trying to build models of what needs to be done to solve AI risk”?
Another problem is that it seems even harder to distinguish between people who are really trying to build such models, and people who are just trying to look like they’re doing that, because there’s no short-term feedback from reality to tell you whether someone’s model is any good. It seems like suggesting people to do that when you’re not sure of their intention is really dangerous, as it could mess up the epistemic situation with AI risk models (even more than it already is). Maybe it would be better to just suggest some concrete short-term projects for them to do instead?
Maybe there is a possible project in this direction. I’ll assume that this is general advice you’d give to many people who want to work in this space. If it is important for people to build a model of what is required for AI to go well then people may as well work on this together. And sure there’s websites like Less Wrong, but people can exchange information much faster by chatting either in person or over Skype. (Of course there are worries that this might lead to overly correlated answers)
Looks like I didn’t entirely succeed in explaining our plan.
And that’s also what we meant. The goal isn’t to just give advice. The goal is to give useful and true advice, and this necessarily requires a model of what really needs to happen for AI risk.
We’re not just going to spin up some interesting ideas. That’s not the mindset. The mindset is to generate a robust model and take it from there, if we ever get that far.
We might be talking to people in the process, but as long as we are in the dark the emphasis will be on asking questions.
EDIT: this wasn’t a thoughtful reply. I take it back. See Ruby’s comments below
My understanding of habryka, and the position I’d agree with, is that the goal should be solely to be forming a model at this time. As currently stated, even before you’ve formed the model, there is a presumption that once you have a model the right thing to do is then instruct others. That is premature.
That kind of seems good, but that’s not what your plan says. The plan as written states that a management layer is required—that seems like possibly a conclusion you reach after you’ve done your model building stage, yet this plan states up front. That seems like a good recipe for confirmation bias to me.
I think there’s an alternative version of this point, and a good underlying procedure, which is that along the way of forming a model of things you generate various hypotheses. That the EA community needs a management layer or someone to connect the dots, etc., is maybe a reasonable hypothesis / proto-model of the situation. It’s something you could possibly post about and solicit feedback and ideas from others and talk to about. That seems good and it seems good if you can generate that hypothesis plus other hypotheses (caveats being, as habryka stated, you should likely start with non-coordination work).
An alternative version of your plan which might receive better reception from many is purely research. “We are going to research questions X and Y.” You might have the knack for the that, and I could see it getting funded if you had a good agenda/good plan and seemed your questioning wouldn’t disrupt or displace other work + others weren’t worried you were going to pivot it into something counterproductive.
Sorry, I have to add one more piece that really seems worth calling out.
If someone’s position is “very much don’t try to tell other people what really needs to happen for AI-risk”, you cannot claim that you meant the same thing when your position includes “the goal is to give useful and true advice”. You don’t also mean what Habryka means at all (you’re ignoring it) and it strikes me as disingenuous to claim that you mean the same thing thing as him at all.
More formally:
Person B is absolutely not saying the same thing as A.
Apologies, that was a knee-jerk reply. I take it back: we did disagree about something.
We’re going to take some time to let all of this criticism sink in.
Additional thought, just as an example, something in the post that seemed off:
Researchers and people working at the various organizations are clever, capable people. I would venture that they’re necessarily thinking about the big picture as part of their work and how it fits in. Given their connection and involvement (possibly over the course of years), their models are probably difficult to surpass if you’re starting from the outside. I also don’t think it’s as simple as talking to lots of people to get all their models and combining them (communicating and synthesizing models is really hard + you’d need to build a lot of credibility before others trusted you could, assuming anyone could really do this well).
The line quoted above, as written, almost makes seems like there’s this low hanging fruit because everyone currently working was heads-down on their problems and no one thought to work on the big picture. That strikes me as a very bad assumption and makes me worry about the kind of reasoning you would use to advise others. Possibly you meant something different from that and more defensible . . . but then unambiguous and clear communication is going to be very key in any coordination/advisory role.
Anyhow, I dislike being purely critical. Not many people are dedicating themselves to trying to solve important problems, so I want to say that I do approve of efforts trying. I think it’s good that you sought feedback on your first project and then formed a new plan. I’ve written these comments because I hope they help nudge you in the direction of really good plans. If they’re biased critical, it’s because I’m trying to explain more of factors I think might be leading to a negative reception of this plan. Because we need all the good plans and good people working on them we can get.
This seems roughly correct to me.