I recently founded Apollo Research: https://www.apolloresearch.ai/
I was previously doing a Ph.D. in ML at the International Max-Planck research school in Tübingen, worked part-time with Epoch and did independent AI safety research.
For more see https://www.mariushobbhahn.com/aboutme/
I subscribe to Crocker’s Rules
Copying from EAF
TL;DR: At least in my experience, AISC was pretty positive for most participants I know and it’s incredibly cheap. It also serves a clear niche that other programs are not filling and it feels reasonable to me to continue the program.
I’ve been a participant in the 2021⁄22 edition. Some thoughts that might make it easier to decide for funders/donors.
1. Impact-per-dollar is probably pretty good for the AISC. It’s incredibly cheap compared to most other AI field-building efforts and scalable.
2. I learned a bunch during AISC and I did enjoy it. It influenced my decision to go deeper into AI safety. It was less impactful than e.g. MATS for me but MATS is a full-time in-person program, so that’s not surprising.
3. AISC fills a couple of important niches in the AI safety ecosystem in my opinion. It’s online and part-time which makes it much easier to join for many people, it implies a much lower commitment which is good for people who want to find out whether they’re a good fit for AIS. It’s also much cheaper than flying everyone to the Bay or London. This also makes it more scalable because the only bottleneck is mentoring capacity without physical constraints.
4. I think AISC is especially good for people who want to test their fit but who are not super experienced yet. This seems like an important function. MATS and ARENA, for example, feel like they target people a bit deeper into the funnel with more experience who are already more certain that they are a good fit.
5. Overall, I think AISC is less impactful than e.g. MATS even without normalizing for participants. Nevertheless, AISC is probably about ~50x cheaper than MATS. So when taking cost into account, it feels clearly impactful enough to continue the project. I think the resulting projects are lower quality but the people are also more junior, so it feels more like an early educational program than e.g. MATS.
6. I have a hard time seeing how the program could be net negative unless something drastically changed since my cohort. In the worst case, people realize that they don’t like one particular type of AI safety research. But since you chat with others who are curious about AIS regularly, it will be much easier to start something that might be more meaningful. Also, this can happen in any field-building program, not just AISC.
7. Caveat: I have done no additional research on this. Maybe others know details that I’m unaware of. See this as my personal opinion and not a detailed research analysis.
(cross-commented from EA forum)
I personally have no stake in defending Conjecture (In fact, I have some questions about the CoEm agenda) but I do think there are a couple of points that feel misleading or wrong to me in your critique.
1. Confidence (meta point): I do not understand where the confidence with which you write the post (or at least how I read it) comes from. I’ve never worked at Conjecture (and presumably you didn’t either) but even I can see that some of your critique is outdated or feels like a misrepresentation of their work to me (see below). For example, making recommendations such as “freezing the hiring of all junior people” or “alignment people should not join Conjecture” require an extremely high bar of evidence in my opinion. I think it is totally reasonable for people who believe in the CoEm agenda to join Conjecture and while Connor has a personality that might not be a great fit for everyone, I could totally imagine working with him productively. Furthermore, making a claim about how and when to hire usually requires a lot of context and depends on many factors, most of which an outsider probably can’t judge.
Given that you state early on that you are an experienced member of the alignment community and your post suggests that you did rigorous research to back up these claims, I think people will put a lot of weight on this post and it does not feel like you use your power responsibly here.
I can very well imagine a less experienced person who is currently looking for positions in the alignment space to go away from this post thinking “well, I shouldn’t apply to Conjecture then” and that feels unjustified to me.
2. Output so far: My understanding of Conjecture’s research agenda so far was roughly: “They started with Polytopes as a big project and published it eventually. On reflection, they were unhappy with the speed and quality of their work (as stated in their reflection post) and decided to change their research strategy. Every two weeks or so, they started a new research sprint in search of a really promising agenda. Then, they wrote up their results in a preliminary report and continued with another project if their findings weren’t sufficiently promising.” In most of their public posts, they stated, that these are preliminary findings and should be treated with caution, etc. Therefore, I think it’s unfair to say that most of their posts do not meet the bar of a conference publication because that wasn’t the intended goal.
Furthermore, I think it’s actually really good that the alignment field is willing to break academic norms and publish preliminary findings. Usually, this makes it much easier to engage with and criticize work earlier and thus improves overall output quality.
On a meta-level, I think it’s bad to criticize labs that do hits-based research approaches for their early output (I also think this applies to your critique of Redwood) because the entire point is that you don’t find a lot until you hit. These kinds of critiques make it more likely that people follow small incremental research agendas and alignment just becomes academia 2.0. When you make a critique like that, at least acknowledge that hits-based research might be the right approach.
3. Your statements about the VCs seem unjustified to me. How do you know they are not aligned? How do you know they wouldn’t support Conjecture doing mostly safety work? How do you know what the VCs were promised in their private conversations with the Conjecture leadership team? Have you talked to the VCs or asked them for a statement?
Of course, you’re free to speculate from the outside but my understanding is that Conjecture actually managed to choose fairly aligned investors who do understand the mission of solving catastrophic risks. I haven’t talked to the VCs either, but I’ve at least asked people who work(ed) at Conjecture.
In conclusion:
1. I think writing critiques is good but really hard without insider knowledge and context.
2. I think this piece will actually (partially) misinform a large part of the community. You can see this already in the comments where people without context say this is a good piece and thanking you for “all the insights”.
3. The EA/LW community seems to be very eager to value critiques highly and for good reason. But whenever people use critiques to spread (partially) misleading information, they should be called out.
4. That being said, I think your critique is partially warranted and things could have gone a lot better at Conjecture. It’s just important to distinguish between “could have gone a lot better” and “we recommend not to work for Conjecture” or adding some half-truths to the warranted critiques.
5. I think your post on Redwood was better but suffered from some of the same problems. Especially the fact that you criticize them for having not enough tangible output when following a hits-based agenda just seems counterproductive to me.