The operationalization of SAR used in the AI Futures Model is this: “A SAR, if dropped into an AGI project in the present-day, would be as good at research taste as if there were only human researchers, who were each made as competent at research taste as the top researcher. The SAR must also qualify as an automated coder (AC), i.e. be able to fully automate coding in the AGI project.”
In truth, I missed this part, and thought it was operationalized as an AI that could automate all AI R&D, but might do this through generating a huge number of hypotheses and experiments rather than having really good research taste. That still requires that the AI comes up with the ideas unassisted though. This is an important threshold in capabilities and level of R&D uplift, I think, since human competence would no longer be a major bottleneck even in idea generation.
Related point: In general I think it is better practise to just discuss specific capability levels without using abbreviations, including AGI or ASI, to make things less confusing (unless you need to refer to a specific level many times making the abbreviation more useful). I’ve been considering removing the SAR abbreviation. Might do that when I get the time.
Fair point about the abbreviations. I went through end edited my original response in that vein. Thank you for that—like all deep technologist nerds (and I am their king, I do not use the term pejoratively), I tend to forget my audience might not be technically inclined. So your suggestion is warmly received.
But let’s discuss the idea of research taste. Does that mean coming up with the original research idea? If so, I know several corporate R&D types who would disagree. They often take the original idea from outside their teams. In my world, at least, it’s often a customer who says, “I need X”, and the research team takes it from there, creating one or more hypotheses based on X as a final target product. Alternatively, it might be a field engineer who says “It would be incredibly useful to have Y” and the team takes it from there. So does that mean it’s okay for a Superhuman AI Researcher to take X or Y from a human, and still be considered a true Superhuman AI Researcher? I’m asking because that’s very much what happened in my case. I said, “I need to create a chip that does X”, and I did so knowing I didn’t even know what hypotheses to generate based on that goal. I didn’t honestly know enough to say “H(0) means architecture A is our best performer, and H(1) means architecture B is our best performer”, because I didn’t know anything at that time about chip architecture. Granted, I’ve learned a lot since then, but I quite literally said to my LLM: “Can we create a chip that performs X?” and the LLM took it from there. It came up with a couple of architectural ideas, generated an H(0) and an H(1), tested them and reported back to me. Of course, I asked it to tutor me along the way, because I genuinely wanted to learn, but I did not design the hypotheses, the test processes or the success / failure criteria. I’d have to share the entire record with an expert to determine whether it was exercising genuine “good research taste”, and I may do that at some point. It would be a very interesting paper, and peer review would be valuable on it. I won’t do it now, because the patent process around it is not fully complete. We’ve only filed the provisional, so I don’t want to share the underlying data yet.
In my case, the automated code generation question is a bit less clear, because I explicitly mandated stopping points so that I could learn as we went. But I never told it what kind of code to write, or who the code should be structured. There’s automation and there’s automation. So in my book at least, “automated code generation” is vague, probably too vague to be really useful. My two cents.
But my main question is this: Can we rule out Superhuman AI Researcher (SAR) status in this case? Can we definitively say, given what I can share now, that the LLM was not acting as a true SAR?
Hmm, I was thinking of an operationalization along the lines of “If all staff at a frontier AI company was replaced by an AI, could the AI keep the same pace of development as the human staff with no access to post 2022 AI systems?”
So on the first point about taking suggestions X or Y from a human, that would be allowed as long as those suggestions are not from staff at the company. The AI has to do everything, but it’s also allowed to do anything that the human staff would be allowed to do, as well as use strategies that are more suitable for AIs than humans.
I think current AIs are far from this threshold, but could also reach it fairly soon if development is superexponential.
I would be impressed and quite surprised if it could do novel hardware discoveries when only provided the original idea and no further prompting, and would update towards AI being even more capable than I thought.
Interesting anecdote!
The operationalization of SAR used in the AI Futures Model is this: “A SAR, if dropped into an AGI project in the present-day, would be as good at research taste as if there were only human researchers, who were each made as competent at research taste as the top researcher. The SAR must also qualify as an automated coder (AC), i.e. be able to fully automate coding in the AGI project.”
In truth, I missed this part, and thought it was operationalized as an AI that could automate all AI R&D, but might do this through generating a huge number of hypotheses and experiments rather than having really good research taste. That still requires that the AI comes up with the ideas unassisted though. This is an important threshold in capabilities and level of R&D uplift, I think, since human competence would no longer be a major bottleneck even in idea generation.
Related point: In general I think it is better practise to just discuss specific capability levels without using abbreviations, including AGI or ASI, to make things less confusing (unless you need to refer to a specific level many times making the abbreviation more useful). I’ve been considering removing the SAR abbreviation. Might do that when I get the time.
Fair point about the abbreviations. I went through end edited my original response in that vein. Thank you for that—like all deep technologist nerds (and I am their king, I do not use the term pejoratively), I tend to forget my audience might not be technically inclined. So your suggestion is warmly received.
But let’s discuss the idea of research taste. Does that mean coming up with the original research idea? If so, I know several corporate R&D types who would disagree. They often take the original idea from outside their teams. In my world, at least, it’s often a customer who says, “I need X”, and the research team takes it from there, creating one or more hypotheses based on X as a final target product. Alternatively, it might be a field engineer who says “It would be incredibly useful to have Y” and the team takes it from there. So does that mean it’s okay for a Superhuman AI Researcher to take X or Y from a human, and still be considered a true Superhuman AI Researcher? I’m asking because that’s very much what happened in my case. I said, “I need to create a chip that does X”, and I did so knowing I didn’t even know what hypotheses to generate based on that goal. I didn’t honestly know enough to say “H(0) means architecture A is our best performer, and H(1) means architecture B is our best performer”, because I didn’t know anything at that time about chip architecture. Granted, I’ve learned a lot since then, but I quite literally said to my LLM: “Can we create a chip that performs X?” and the LLM took it from there. It came up with a couple of architectural ideas, generated an H(0) and an H(1), tested them and reported back to me. Of course, I asked it to tutor me along the way, because I genuinely wanted to learn, but I did not design the hypotheses, the test processes or the success / failure criteria. I’d have to share the entire record with an expert to determine whether it was exercising genuine “good research taste”, and I may do that at some point. It would be a very interesting paper, and peer review would be valuable on it. I won’t do it now, because the patent process around it is not fully complete. We’ve only filed the provisional, so I don’t want to share the underlying data yet.
In my case, the automated code generation question is a bit less clear, because I explicitly mandated stopping points so that I could learn as we went. But I never told it what kind of code to write, or who the code should be structured. There’s automation and there’s automation. So in my book at least, “automated code generation” is vague, probably too vague to be really useful. My two cents.
But my main question is this: Can we rule out Superhuman AI Researcher (SAR) status in this case? Can we definitively say, given what I can share now, that the LLM was not acting as a true SAR?
Hmm, I was thinking of an operationalization along the lines of “If all staff at a frontier AI company was replaced by an AI, could the AI keep the same pace of development as the human staff with no access to post 2022 AI systems?”
So on the first point about taking suggestions X or Y from a human, that would be allowed as long as those suggestions are not from staff at the company. The AI has to do everything, but it’s also allowed to do anything that the human staff would be allowed to do, as well as use strategies that are more suitable for AIs than humans.
I think current AIs are far from this threshold, but could also reach it fairly soon if development is superexponential.
I would be impressed and quite surprised if it could do novel hardware discoveries when only provided the original idea and no further prompting, and would update towards AI being even more capable than I thought.