Looks like I didn’t entirely succeed in explaining our plan.
My recommendation was very concretely “try to build an internal model of what really needs to happen for AI-risk to go well” and very much not “try to tell other people what really needs to happen for AI-risk”, which is almost the exact opposite.
And that’s also what we meant. The goal isn’t to just give advice. The goal is to give useful and true advice, and this necessarily requires a model of what really needs to happen for AI risk.
We’re not just going to spin up some interesting ideas. That’s not the mindset. The mindset is to generate a robust model and take it from there, if we ever get that far.
We might be talking to people in the process, but as long as we are in the dark the emphasis will be on asking questions.
EDIT: this wasn’t a thoughtful reply. I take it back. See Ruby’s comments below
The goal is to give useful and true advice, and this necessarily requires a model of what really needs to happen for AI risk.
My understanding of habryka, and the position I’d agree with, is that the goal should be solely to be forming a model at this time. As currently stated, even before you’ve formed the model, there is a presumption that once you have a model the right thing to do is then instruct others. That is premature.
The mindset is to generate a robust model and take it from there, if we ever get that far.
That kind of seems good, but that’s not what your plan says. The plan as written states that a management layer is required—that seems like possibly a conclusion you reach after you’ve done your model building stage, yet this plan states up front. That seems like a good recipe for confirmation bias to me.
I think there’s an alternative version of this point, and a good underlying procedure, which is that along the way of forming a model of things you generate various hypotheses. That the EA community needs a management layer or someone to connect the dots, etc., is maybe a reasonable hypothesis / proto-model of the situation. It’s something you could possibly post about and solicit feedback and ideas from others and talk to about. That seems good and it seems good if you can generate that hypothesis plus other hypotheses (caveats being, as habryka stated, you should likely start with non-coordination work).
An alternative version of your plan which might receive better reception from many is purely research. “We are going to research questions X and Y.” You might have the knack for the that, and I could see it getting funded if you had a good agenda/good plan and seemed your questioning wouldn’t disrupt or displace other work + others weren’t worried you were going to pivot it into something counterproductive.
Sorry, I have to add one more piece that really seems worth calling out.
Looks like I didn’t entirely succeed in explaining our plan.
>”My recommendation was very concretely “try to build an internal model of what really needs to happen for AI-risk to go well” and very much not “try to tell other people what really needs to happen for AI-risk”, which is almost the exact opposite.”
And that’s also what we meant. The goal isn’t to just give advice. The goal is to give useful and true advice, and this necessarily requires a model of what really needs to happen for AI risk.
If someone’s position is “very much don’t try to tell other people what really needs to happen for AI-risk”, you cannot claim that you meant the same thing when your position includes “the goal is to give useful and true advice”. You don’t also mean what Habryka means at all (you’re ignoring it) and it strikes me as disingenuous to claim that you mean the same thing thing as him at all.
More formally:
Person A: “X, definitely not Y”
Person B: “X and Y”
Person B is absolutely not saying the same thing as A.
Additional thought, just as an example, something in the post that seemed off:
Instead, someone should be thinking about the bigger picture first.
Researchers and people working at the various organizations are clever, capable people. I would venture that they’re necessarily thinking about the big picture as part of their work and how it fits in. Given their connection and involvement (possibly over the course of years), their models are probably difficult to surpass if you’re starting from the outside. I also don’t think it’s as simple as talking to lots of people to get all their models and combining them (communicating and synthesizing models is really hard + you’d need to build a lot of credibility before others trusted you could, assuming anyone could really do this well).
The line quoted above, as written, almost makes seems like there’s this low hanging fruit because everyone currently working was heads-down on their problems and no one thought to work on the big picture. That strikes me as a very bad assumption and makes me worry about the kind of reasoning you would use to advise others. Possibly you meant something different from that and more defensible . . . but then unambiguous and clear communication is going to be very key in any coordination/advisory role.
Anyhow, I dislike being purely critical. Not many people are dedicating themselves to trying to solve important problems, so I want to say that I do approve of efforts trying. I think it’s good that you sought feedback on your first project and then formed a new plan. I’ve written these comments because I hope they help nudge you in the direction of really good plans. If they’re biased critical, it’s because I’m trying to explain more of factors I think might be leading to a negative reception of this plan. Because we need all the good plans and good people working on them we can get.
Looks like I didn’t entirely succeed in explaining our plan.
And that’s also what we meant. The goal isn’t to just give advice. The goal is to give useful and true advice, and this necessarily requires a model of what really needs to happen for AI risk.
We’re not just going to spin up some interesting ideas. That’s not the mindset. The mindset is to generate a robust model and take it from there, if we ever get that far.
We might be talking to people in the process, but as long as we are in the dark the emphasis will be on asking questions.
EDIT: this wasn’t a thoughtful reply. I take it back. See Ruby’s comments below
My understanding of habryka, and the position I’d agree with, is that the goal should be solely to be forming a model at this time. As currently stated, even before you’ve formed the model, there is a presumption that once you have a model the right thing to do is then instruct others. That is premature.
That kind of seems good, but that’s not what your plan says. The plan as written states that a management layer is required—that seems like possibly a conclusion you reach after you’ve done your model building stage, yet this plan states up front. That seems like a good recipe for confirmation bias to me.
I think there’s an alternative version of this point, and a good underlying procedure, which is that along the way of forming a model of things you generate various hypotheses. That the EA community needs a management layer or someone to connect the dots, etc., is maybe a reasonable hypothesis / proto-model of the situation. It’s something you could possibly post about and solicit feedback and ideas from others and talk to about. That seems good and it seems good if you can generate that hypothesis plus other hypotheses (caveats being, as habryka stated, you should likely start with non-coordination work).
An alternative version of your plan which might receive better reception from many is purely research. “We are going to research questions X and Y.” You might have the knack for the that, and I could see it getting funded if you had a good agenda/good plan and seemed your questioning wouldn’t disrupt or displace other work + others weren’t worried you were going to pivot it into something counterproductive.
Sorry, I have to add one more piece that really seems worth calling out.
If someone’s position is “very much don’t try to tell other people what really needs to happen for AI-risk”, you cannot claim that you meant the same thing when your position includes “the goal is to give useful and true advice”. You don’t also mean what Habryka means at all (you’re ignoring it) and it strikes me as disingenuous to claim that you mean the same thing thing as him at all.
More formally:
Person B is absolutely not saying the same thing as A.
Apologies, that was a knee-jerk reply. I take it back: we did disagree about something.
We’re going to take some time to let all of this criticism sink in.
Additional thought, just as an example, something in the post that seemed off:
Researchers and people working at the various organizations are clever, capable people. I would venture that they’re necessarily thinking about the big picture as part of their work and how it fits in. Given their connection and involvement (possibly over the course of years), their models are probably difficult to surpass if you’re starting from the outside. I also don’t think it’s as simple as talking to lots of people to get all their models and combining them (communicating and synthesizing models is really hard + you’d need to build a lot of credibility before others trusted you could, assuming anyone could really do this well).
The line quoted above, as written, almost makes seems like there’s this low hanging fruit because everyone currently working was heads-down on their problems and no one thought to work on the big picture. That strikes me as a very bad assumption and makes me worry about the kind of reasoning you would use to advise others. Possibly you meant something different from that and more defensible . . . but then unambiguous and clear communication is going to be very key in any coordination/advisory role.
Anyhow, I dislike being purely critical. Not many people are dedicating themselves to trying to solve important problems, so I want to say that I do approve of efforts trying. I think it’s good that you sought feedback on your first project and then formed a new plan. I’ve written these comments because I hope they help nudge you in the direction of really good plans. If they’re biased critical, it’s because I’m trying to explain more of factors I think might be leading to a negative reception of this plan. Because we need all the good plans and good people working on them we can get.
This seems roughly correct to me.