One crude way of doing it is saying that a professor is successful if they are a professor at a top 10-ish university.
But why should that be the case? Academia is hypercompetitive, but the way it selects is not solely on the quality of one’s research. Choosing the trendiest fields has a huge impact. Perhaps the professors that are chosen by prestigious universities are the ones that the prestigious universities think are the best at drawing in grant money and getting publications into high-impact journals, such as Nature, or Science.
Specifically I think professors are at least +2σ at “hedgehog-y” and “selling work” compared to similarly intelligent people who are not successful professors, and more like +σ at the other skills.
How does one determine this?
Overall, it seems like your argument is that AI safety researchers should behave more like traditional academia for a bunch reasons that have mostly to do with social prestige. While I don’t discount the role that social prestige has to play in drawing people into a field and legitimizing it, it seems like overall, the pursuit of prestige has been a net negative for science as a whole, leading to, for example, the replication crisis in medicine and biology, or the nonstop pursuit of string theory over alternate hypotheses in physics. Therefore, I’m not convinced that importing these prestige-oriented traits from traditional science would be a net positive for AI safety research.
Furthermore, I would note that traditional academia has been moving away from these practices, to a certain extent. During the early days of the COVID pandemic, quite a lot of information was exchanged not as formal peer-reviewed research papers, but as blog posts, Twitter threads, and preprints. In AI capabilities research, many new advances are announced as blog posts first, even if they might be formalized in a reseach paper later. Looking further back in the history of science, James Gleick, in Chaos relates how the early researchers into chaos and complexity theories did their research by informally exchanging letters and draft papers. They were outside the normal categories that the bureaucracy of academia had established, so no journal would publish them.
It seems to me that the foundational, paradigm-shifting research always takes place this way. It takes place away from the formal rigors of academia, in informal exchanges between self-selected individuals. Only later, once the core paradigms of the new field have been laid down, does the field become incorporated into the bureaucracy of science, becoming legible enough for journals to routinely publish findings from the new field. I think AI safety research is at this early stage of maturity, and therefore it doesn’t make sense for it to import the practices that would help practitioners survive and thrive in the bureaucracy of “Big Science”.
Overall, it seems like your argument is that AI safety researchers should behave more like traditional academia for a bunch reasons that have mostly to do with social prestige.
That is not what I am saying. I am saying that successful professors are highly successful researchers, that they share many qualities (most of which by the way have nothing to do with social prestige), and that AI safety researchers might consider emulating these qualities.
Furthermore, I would note that traditional academia has been moving away from these practices, to a certain extent. During the early days of the COVID pandemic, quite a lot of information was exchanged not as formal peer-reviewed research papers, but as blog posts, Twitter threads, and preprints. In AI capabilities research, many new advances are announced as blog posts first, even if they might be formalized in a reseach paper later. [...]
This is a non sequitur. I’m not saying stop the blog posts. In fact, I am claiming that “selling your work” is a good thing. Therefore I also think blog posts are fine. When I write about the importance of a good abstract/introduction, I mean not just literally in the context of a NeurIPS paper but also more broadly in the context of motivating ones’ work better, so that a broader scientific audience can read your work and want to build off it. (But also separately I do think people should eventually turn good blog posts into papers for wider reach)
I think AI safety research is at this early stage of maturity
I disagree. Non-EA funding for safety is pouring in. Safety is being talked about in mainstream venues. Also more academic papers popping up, as linked in my post. In terms of progress on aligning AI I agree the field is in its early stages, but in terms of the size of the field and institutions built up around it, nothing about AI safety feels early stage to me anymore.
How does one determine this?
I am confused by your repeated focus on empirics, when I have been very up front that this is a qualitative, anecdotal, personal analysis.
I am saying that successful professors are highly successful researchers
Are they? That’s why I’m focusing on empirics. How do you know that these people are highly successful researchers? What impressive research findings have they developed, and how did e.g. networking and selling their work enable them to get to these findings? Similarly, with regards to bureaucracy, how did successfully navigating the bureaucracy of academia enable these researchers to improve their work?
The way it stands right now, what you’re doing is pointing at some traits that correlate with academic success, and are claiming that
Aspiring to the standards of prestigious academic institutions will speed up AI safety research
Researchers at prestigious academic institutions share certain traits
Therefore adopting these traits will lead to better AI safety research
This reasoning is flawed. First, why should AI safety research aspire to the same standards of “publish or perish” and the emphasis on finding positive results that gave us the replication crisis? It seems to me that, to the greatest extent possible, AI safety research should reject these standards, and focus on finding results that are true, rather than results that are publishable.
Secondly, correlation is not causation. The fact that many researchers from an anecdotal sample share certain attributes doesn’t mean that those attributes are causative of those researchers’ success. There are lots of researchers who do all of the things that you describe, managing their time, networking aggressively, and focusing on understanding grantmaking, who do not end up at prestigious institutions. There are lots of researchers who do all of those things who don’t end up with tenure at all.
This is why I’m so skeptical of your post. I’m not sure that the steps your take are actually causative of academic success, rather than merely correlating with academic success, and furthermore, I’m not even sure that the standards of academic success are even something that AI safety research should aspire to.
There are lots of ways a researcher can choose to adopt new productivity habits. They include:
Inside view, reasoning from first principles
Outside view, copying what successful researchers do
The purpose of this post is to, from an outside view perspective, list what a class of researchers (professors) does, which happens to operate very differently from AI safety.
Once again, I am not claiming to have an inside view argument in favor of the adoption of each of these attributes. I do not have empirics. I am not claiming to have an airtight causal model. If you will refer back to the original post, you will notice that I was careful to call this a list of attributes coming from anecdotal evidence, and if you will refer back to the AI safety section, you will notice that I was careful to call my points considerations and not conclusions.
You keep arguing against a claim which I’ve never put forward, which is something like “The bullshit in academia (publish or perish, positive results give better papers) causes better research to happen.” Of course I disagree with this claim. There is no need to waste ink arguing against it.
It seems like the actual crux we disagree on is: “How similar are the goals success in academia with success in doing good (AI safety) research?” If I had to guess the source of our disagreement, I might speculate that we’ve both heard the same stories about the replication crisis, the inefficiencies of grant proposals and peer review, and other bullshit in academia. But, I’ve additionally encountered a great deal of anecdotal evidence indicating: in spite of all this bullshit, the people at the top seem to overwhelmingly not be bogged down by it, and the first-order factor in them getting where they are was in fact research quality. The way to convince you of this fact might be to repeat the methodology used in Childhoods of exceptional people, but this would be incredibly time consuming. (I’ll give you 1/20th of such a blog post for free: here’s Terry Tao on time management.)
This crux clears up our correlation vs causation disagreement: since I think the goals are very similar, correlation is evidence for causation, whereas since you think the goals are very different, it seems like you think many of the attributes I’ve listed are primarily relevant for the ‘navigating academic bullshit’ part of academia.
I’ve addressed your comment in broad terms, but just to conclude I wanted to respond to one point you made which seems especially wrong.
how did e.g. networking [...] enable them to get to these [impressive research] findings?
In the networking section, you will find that I defined “networking” as “knowing many people doing research in and outside your field, so that you can easily reach out to them to request a collaboration”. People are more likely to respond to collaboration requests from acquaintances than from strangers. Thus for this particular attribute you actually do get a causal model: networking causes collaborations which causes better research results. I guess you can dispute the claim “collaborations cause better research results”, but I think this would be an odd hill to die on, considering most interdisciplinary work relies on collaborations.
What I’m questioning is the implicit assumption in your post that AI safety research will inevitably take place in an academic environment, and therefore productivity practices derived from other academic settings will be helpful. Why should this be the case when, over the past few years, most of the AI capabilities research has occurred in corporate research labs?
Some of your suggestions, of course, work equally well in either environment. But not all, and even the ones which do work would require a shift in emphasis. For example, when you say professors should be acquainted with other professors, that’s valid in academia, where roughly everyone who matters either has tenure or is on a tenure track. However, that is not true in a corporate environment, where many people may not even have PhDs. Furthermore, in a corporate environment, limiting one’s networking to just researchers is probably ill advised, given that there are many other people who would have influence upon the research. Knowing a senior executive with influence over product roadmaps could be just as valuable, even if that executive has no academic pedigree at all.
Prioritizing high value research and ignoring everything else is a skill that works in both corporate and academic environments. But 80/20-ing teaching? In a corporate research lab, one has no teaching responsibilities. One would be far better served learning some basic software engineering practices, in order to better interface with product engineers. Similarly, with regards to publishing, for a corporate research lab, having a working product is worth dozens of research papers. Research papers bring prestige, but they don’t pay the bills. Therefore, I would argue that AI safety researchers should be keeping an eye on how their findings can be applied to existing AI systems. This kind of product-focused development is something that academia is notoriously bad at.
I also question your claim that academic bureaucracy doesn’t slow good researchers down very much. That’s very much not in line with what anecdotes I’ve heard. From what I’ve seen, writing grant proposals, dealing with university bureaucracy, and teaching responsibilities are a significant time suck. Maybe with practice and experience, it’s possible for a good researcher to complete these tasks on “autopilot”, and therefore not notice the time that’s being spent. But the tasks are still costing time and mental energy that, ideally, would be devoted to research or writing.
I don’t think it’s inevitable that academia will take over AI safety research, given the trend in AI capabilities research, and I certainly don’t think that academia taking over AI safety research would be a good thing. For this reason I question whether it’s valuable for AI safety researchers to develop skills valuable for academic research, specifically, as opposed to general time management, software engineering and product development skills.
Any researcher, inside or outside of academia, might consider emulating attributes successful professors have in order to boost personal research productivity.
AI safety researchers outside of academia should try harder to make their legible to academics, as a cheap way to get more good researchers thinking about AI safety.
What I’m questioning is the implicit assumption in your post that AI safety research will inevitably take place in an academic environment [...]
This assumption is not implicit, you’re putting together (1) and (2) in a way which I did not intend.
Furthermore, in a corporate environment, limiting one’s networking to just researchers is probably ill advised, given that there are many other people who would have influence upon the research. Knowing a senior executive with influence over product roadmaps could be just as valuable, even if that executive has no academic pedigree at all.
I agree but this is not a counterargument against my post. This is just an incredibly reasonable interpretation of what it means to be “good at networking” for a industry researcher.
But 80/20-ing teaching? In a corporate research lab, one has no teaching responsibilities. One would be far better served learning some basic software engineering practices, in order to better interface with product engineers.
My post is not literally recommending that non-academics 80⁄20 their teaching. I am confused why you think that I would think this. 80/20ing teaching is an example of how professors allocate their time to what’s important. Professors are being used as a case study in the post. When applied to an AI safety researcher who works independently or as part of an industry lab, perhaps “teaching” might be replaced with “responding to cold emails” or “supervising an intern”. I acknowledge that professors spend more time teaching than non-academic researchers spend doing these tasks. But once again, the point of this post is just to list a bunch of things successful professors do, and then non-professors are meant to consider these points and adapt the advice to their own environment.
Similarly, with regards to publishing, for a corporate research lab, having a working product is worth dozens of research papers. Research papers bring prestige, but they don’t pay the bills. Therefore, I would argue that AI safety researchers should be keeping an eye on how their findings can be applied to existing AI systems. This kind of product-focused development is something that academia is notoriously bad at.
This seems like a crux. It seems like I am more optimistic about leveraging academic labor and expertise, and you are more optimistic about deploying AI safety solutions to existing systems.
I also question your claim that academic bureaucracy doesn’t slow good researchers down very much. That’s very much not in line with what anecdotes I’ve heard. [...]
This is another crux. We both have heard different anecdotal evidence and are weighing it differently.
I don’t think it’s inevitable that academia will take over AI safety research, given the trend in AI capabilities research, and I certainly don’t think that academia taking over AI safety research would be a good thing.
I never said that academia would take over AI safety research, and I also never said this would be a good thing. I believe that there is a lot of untapped free skilled labor in academia, and AI safety researchers should put in more of an effort (e.g. by writing papers) to put that labor to use.
For this reason I question whether it’s valuable for AI safety researchers to develop skills valuable for academic research, specifically, as opposed to general time management, software engineering and product development skills.
One of the attributes I list is literally time management. As for the other two, I think it depends on the kind of AI safety researcher we are talking about—going directly back to our “leveraging academia” versus “product development” crux. I agree that if what you’re trying to do is product development, that the skills you list are critical. But also, I think product development is not at all the only way to do AI safety, and other ways to do AI safety more easily plug into academia.
But why should that be the case? Academia is hypercompetitive, but the way it selects is not solely on the quality of one’s research. Choosing the trendiest fields has a huge impact. Perhaps the professors that are chosen by prestigious universities are the ones that the prestigious universities think are the best at drawing in grant money and getting publications into high-impact journals, such as Nature, or Science.
How does one determine this?
Overall, it seems like your argument is that AI safety researchers should behave more like traditional academia for a bunch reasons that have mostly to do with social prestige. While I don’t discount the role that social prestige has to play in drawing people into a field and legitimizing it, it seems like overall, the pursuit of prestige has been a net negative for science as a whole, leading to, for example, the replication crisis in medicine and biology, or the nonstop pursuit of string theory over alternate hypotheses in physics. Therefore, I’m not convinced that importing these prestige-oriented traits from traditional science would be a net positive for AI safety research.
Furthermore, I would note that traditional academia has been moving away from these practices, to a certain extent. During the early days of the COVID pandemic, quite a lot of information was exchanged not as formal peer-reviewed research papers, but as blog posts, Twitter threads, and preprints. In AI capabilities research, many new advances are announced as blog posts first, even if they might be formalized in a reseach paper later. Looking further back in the history of science, James Gleick, in Chaos relates how the early researchers into chaos and complexity theories did their research by informally exchanging letters and draft papers. They were outside the normal categories that the bureaucracy of academia had established, so no journal would publish them.
It seems to me that the foundational, paradigm-shifting research always takes place this way. It takes place away from the formal rigors of academia, in informal exchanges between self-selected individuals. Only later, once the core paradigms of the new field have been laid down, does the field become incorporated into the bureaucracy of science, becoming legible enough for journals to routinely publish findings from the new field. I think AI safety research is at this early stage of maturity, and therefore it doesn’t make sense for it to import the practices that would help practitioners survive and thrive in the bureaucracy of “Big Science”.
That is not what I am saying. I am saying that successful professors are highly successful researchers, that they share many qualities (most of which by the way have nothing to do with social prestige), and that AI safety researchers might consider emulating these qualities.
This is a non sequitur. I’m not saying stop the blog posts. In fact, I am claiming that “selling your work” is a good thing. Therefore I also think blog posts are fine. When I write about the importance of a good abstract/introduction, I mean not just literally in the context of a NeurIPS paper but also more broadly in the context of motivating ones’ work better, so that a broader scientific audience can read your work and want to build off it. (But also separately I do think people should eventually turn good blog posts into papers for wider reach)
I disagree. Non-EA funding for safety is pouring in. Safety is being talked about in mainstream venues. Also more academic papers popping up, as linked in my post. In terms of progress on aligning AI I agree the field is in its early stages, but in terms of the size of the field and institutions built up around it, nothing about AI safety feels early stage to me anymore.
I am confused by your repeated focus on empirics, when I have been very up front that this is a qualitative, anecdotal, personal analysis.
Are they? That’s why I’m focusing on empirics. How do you know that these people are highly successful researchers? What impressive research findings have they developed, and how did e.g. networking and selling their work enable them to get to these findings? Similarly, with regards to bureaucracy, how did successfully navigating the bureaucracy of academia enable these researchers to improve their work?
The way it stands right now, what you’re doing is pointing at some traits that correlate with academic success, and are claiming that
Aspiring to the standards of prestigious academic institutions will speed up AI safety research
Researchers at prestigious academic institutions share certain traits
Therefore adopting these traits will lead to better AI safety research
This reasoning is flawed. First, why should AI safety research aspire to the same standards of “publish or perish” and the emphasis on finding positive results that gave us the replication crisis? It seems to me that, to the greatest extent possible, AI safety research should reject these standards, and focus on finding results that are true, rather than results that are publishable.
Secondly, correlation is not causation. The fact that many researchers from an anecdotal sample share certain attributes doesn’t mean that those attributes are causative of those researchers’ success. There are lots of researchers who do all of the things that you describe, managing their time, networking aggressively, and focusing on understanding grantmaking, who do not end up at prestigious institutions. There are lots of researchers who do all of those things who don’t end up with tenure at all.
This is why I’m so skeptical of your post. I’m not sure that the steps your take are actually causative of academic success, rather than merely correlating with academic success, and furthermore, I’m not even sure that the standards of academic success are even something that AI safety research should aspire to.
There are lots of ways a researcher can choose to adopt new productivity habits. They include:
Inside view, reasoning from first principles
Outside view, copying what successful researchers do
The purpose of this post is to, from an outside view perspective, list what a class of researchers (professors) does, which happens to operate very differently from AI safety.
Once again, I am not claiming to have an inside view argument in favor of the adoption of each of these attributes. I do not have empirics. I am not claiming to have an airtight causal model. If you will refer back to the original post, you will notice that I was careful to call this a list of attributes coming from anecdotal evidence, and if you will refer back to the AI safety section, you will notice that I was careful to call my points considerations and not conclusions.
You keep arguing against a claim which I’ve never put forward, which is something like “The bullshit in academia (publish or perish, positive results give better papers) causes better research to happen.” Of course I disagree with this claim. There is no need to waste ink arguing against it.
It seems like the actual crux we disagree on is: “How similar are the goals success in academia with success in doing good (AI safety) research?” If I had to guess the source of our disagreement, I might speculate that we’ve both heard the same stories about the replication crisis, the inefficiencies of grant proposals and peer review, and other bullshit in academia. But, I’ve additionally encountered a great deal of anecdotal evidence indicating: in spite of all this bullshit, the people at the top seem to overwhelmingly not be bogged down by it, and the first-order factor in them getting where they are was in fact research quality. The way to convince you of this fact might be to repeat the methodology used in Childhoods of exceptional people, but this would be incredibly time consuming. (I’ll give you 1/20th of such a blog post for free: here’s Terry Tao on time management.)
This crux clears up our correlation vs causation disagreement: since I think the goals are very similar, correlation is evidence for causation, whereas since you think the goals are very different, it seems like you think many of the attributes I’ve listed are primarily relevant for the ‘navigating academic bullshit’ part of academia.
I’ve addressed your comment in broad terms, but just to conclude I wanted to respond to one point you made which seems especially wrong.
In the networking section, you will find that I defined “networking” as “knowing many people doing research in and outside your field, so that you can easily reach out to them to request a collaboration”. People are more likely to respond to collaboration requests from acquaintances than from strangers. Thus for this particular attribute you actually do get a causal model: networking causes collaborations which causes better research results. I guess you can dispute the claim “collaborations cause better research results”, but I think this would be an odd hill to die on, considering most interdisciplinary work relies on collaborations.
What I’m questioning is the implicit assumption in your post that AI safety research will inevitably take place in an academic environment, and therefore productivity practices derived from other academic settings will be helpful. Why should this be the case when, over the past few years, most of the AI capabilities research has occurred in corporate research labs?
Some of your suggestions, of course, work equally well in either environment. But not all, and even the ones which do work would require a shift in emphasis. For example, when you say professors should be acquainted with other professors, that’s valid in academia, where roughly everyone who matters either has tenure or is on a tenure track. However, that is not true in a corporate environment, where many people may not even have PhDs. Furthermore, in a corporate environment, limiting one’s networking to just researchers is probably ill advised, given that there are many other people who would have influence upon the research. Knowing a senior executive with influence over product roadmaps could be just as valuable, even if that executive has no academic pedigree at all.
Prioritizing high value research and ignoring everything else is a skill that works in both corporate and academic environments. But 80/20-ing teaching? In a corporate research lab, one has no teaching responsibilities. One would be far better served learning some basic software engineering practices, in order to better interface with product engineers. Similarly, with regards to publishing, for a corporate research lab, having a working product is worth dozens of research papers. Research papers bring prestige, but they don’t pay the bills. Therefore, I would argue that AI safety researchers should be keeping an eye on how their findings can be applied to existing AI systems. This kind of product-focused development is something that academia is notoriously bad at.
I also question your claim that academic bureaucracy doesn’t slow good researchers down very much. That’s very much not in line with what anecdotes I’ve heard. From what I’ve seen, writing grant proposals, dealing with university bureaucracy, and teaching responsibilities are a significant time suck. Maybe with practice and experience, it’s possible for a good researcher to complete these tasks on “autopilot”, and therefore not notice the time that’s being spent. But the tasks are still costing time and mental energy that, ideally, would be devoted to research or writing.
I don’t think it’s inevitable that academia will take over AI safety research, given the trend in AI capabilities research, and I certainly don’t think that academia taking over AI safety research would be a good thing. For this reason I question whether it’s valuable for AI safety researchers to develop skills valuable for academic research, specifically, as opposed to general time management, software engineering and product development skills.
I have 2 separate claims:
Any researcher, inside or outside of academia, might consider emulating attributes successful professors have in order to boost personal research productivity.
AI safety researchers outside of academia should try harder to make their legible to academics, as a cheap way to get more good researchers thinking about AI safety.
This assumption is not implicit, you’re putting together (1) and (2) in a way which I did not intend.
I agree but this is not a counterargument against my post. This is just an incredibly reasonable interpretation of what it means to be “good at networking” for a industry researcher.
My post is not literally recommending that non-academics 80⁄20 their teaching. I am confused why you think that I would think this. 80/20ing teaching is an example of how professors allocate their time to what’s important. Professors are being used as a case study in the post. When applied to an AI safety researcher who works independently or as part of an industry lab, perhaps “teaching” might be replaced with “responding to cold emails” or “supervising an intern”. I acknowledge that professors spend more time teaching than non-academic researchers spend doing these tasks. But once again, the point of this post is just to list a bunch of things successful professors do, and then non-professors are meant to consider these points and adapt the advice to their own environment.
This seems like a crux. It seems like I am more optimistic about leveraging academic labor and expertise, and you are more optimistic about deploying AI safety solutions to existing systems.
This is another crux. We both have heard different anecdotal evidence and are weighing it differently.
I never said that academia would take over AI safety research, and I also never said this would be a good thing. I believe that there is a lot of untapped free skilled labor in academia, and AI safety researchers should put in more of an effort (e.g. by writing papers) to put that labor to use.
One of the attributes I list is literally time management. As for the other two, I think it depends on the kind of AI safety researcher we are talking about—going directly back to our “leveraging academia” versus “product development” crux. I agree that if what you’re trying to do is product development, that the skills you list are critical. But also, I think product development is not at all the only way to do AI safety, and other ways to do AI safety more easily plug into academia.