Fair point about the abbreviations. I went through end edited my original response in that vein. Thank you for that—like all deep technologist nerds (and I am their king, I do not use the term pejoratively), I tend to forget my audience might not be technically inclined. So your suggestion is warmly received.
But let’s discuss the idea of research taste. Does that mean coming up with the original research idea? If so, I know several corporate R&D types who would disagree. They often take the original idea from outside their teams. In my world, at least, it’s often a customer who says, “I need X”, and the research team takes it from there, creating one or more hypotheses based on X as a final target product. Alternatively, it might be a field engineer who says “It would be incredibly useful to have Y” and the team takes it from there. So does that mean it’s okay for a Superhuman AI Researcher to take X or Y from a human, and still be considered a true Superhuman AI Researcher? I’m asking because that’s very much what happened in my case. I said, “I need to create a chip that does X”, and I did so knowing I didn’t even know what hypotheses to generate based on that goal. I didn’t honestly know enough to say “H(0) means architecture A is our best performer, and H(1) means architecture B is our best performer”, because I didn’t know anything at that time about chip architecture. Granted, I’ve learned a lot since then, but I quite literally said to my LLM: “Can we create a chip that performs X?” and the LLM took it from there. It came up with a couple of architectural ideas, generated an H(0) and an H(1), tested them and reported back to me. Of course, I asked it to tutor me along the way, because I genuinely wanted to learn, but I did not design the hypotheses, the test processes or the success / failure criteria. I’d have to share the entire record with an expert to determine whether it was exercising genuine “good research taste”, and I may do that at some point. It would be a very interesting paper, and peer review would be valuable on it. I won’t do it now, because the patent process around it is not fully complete. We’ve only filed the provisional, so I don’t want to share the underlying data yet.
In my case, the automated code generation question is a bit less clear, because I explicitly mandated stopping points so that I could learn as we went. But I never told it what kind of code to write, or who the code should be structured. There’s automation and there’s automation. So in my book at least, “automated code generation” is vague, probably too vague to be really useful. My two cents.
But my main question is this: Can we rule out Superhuman AI Researcher (SAR) status in this case? Can we definitively say, given what I can share now, that the LLM was not acting as a true SAR?
Hmm, I was thinking of an operationalization along the lines of “If all staff at a frontier AI company was replaced by an AI, could the AI keep the same pace of development as the human staff with no access to post 2022 AI systems?”
So on the first point about taking suggestions X or Y from a human, that would be allowed as long as those suggestions are not from staff at the company. The AI has to do everything, but it’s also allowed to do anything that the human staff would be allowed to do, as well as use strategies that are more suitable for AIs than humans.
I think current AIs are far from this threshold, but could also reach it fairly soon if development is superexponential.
I would be impressed and quite surprised if it could do novel hardware discoveries when only provided the original idea and no further prompting, and would update towards AI being even more capable than I thought.
Fair point about the abbreviations. I went through end edited my original response in that vein. Thank you for that—like all deep technologist nerds (and I am their king, I do not use the term pejoratively), I tend to forget my audience might not be technically inclined. So your suggestion is warmly received.
But let’s discuss the idea of research taste. Does that mean coming up with the original research idea? If so, I know several corporate R&D types who would disagree. They often take the original idea from outside their teams. In my world, at least, it’s often a customer who says, “I need X”, and the research team takes it from there, creating one or more hypotheses based on X as a final target product. Alternatively, it might be a field engineer who says “It would be incredibly useful to have Y” and the team takes it from there. So does that mean it’s okay for a Superhuman AI Researcher to take X or Y from a human, and still be considered a true Superhuman AI Researcher? I’m asking because that’s very much what happened in my case. I said, “I need to create a chip that does X”, and I did so knowing I didn’t even know what hypotheses to generate based on that goal. I didn’t honestly know enough to say “H(0) means architecture A is our best performer, and H(1) means architecture B is our best performer”, because I didn’t know anything at that time about chip architecture. Granted, I’ve learned a lot since then, but I quite literally said to my LLM: “Can we create a chip that performs X?” and the LLM took it from there. It came up with a couple of architectural ideas, generated an H(0) and an H(1), tested them and reported back to me. Of course, I asked it to tutor me along the way, because I genuinely wanted to learn, but I did not design the hypotheses, the test processes or the success / failure criteria. I’d have to share the entire record with an expert to determine whether it was exercising genuine “good research taste”, and I may do that at some point. It would be a very interesting paper, and peer review would be valuable on it. I won’t do it now, because the patent process around it is not fully complete. We’ve only filed the provisional, so I don’t want to share the underlying data yet.
In my case, the automated code generation question is a bit less clear, because I explicitly mandated stopping points so that I could learn as we went. But I never told it what kind of code to write, or who the code should be structured. There’s automation and there’s automation. So in my book at least, “automated code generation” is vague, probably too vague to be really useful. My two cents.
But my main question is this: Can we rule out Superhuman AI Researcher (SAR) status in this case? Can we definitively say, given what I can share now, that the LLM was not acting as a true SAR?
Hmm, I was thinking of an operationalization along the lines of “If all staff at a frontier AI company was replaced by an AI, could the AI keep the same pace of development as the human staff with no access to post 2022 AI systems?”
So on the first point about taking suggestions X or Y from a human, that would be allowed as long as those suggestions are not from staff at the company. The AI has to do everything, but it’s also allowed to do anything that the human staff would be allowed to do, as well as use strategies that are more suitable for AIs than humans.
I think current AIs are far from this threshold, but could also reach it fairly soon if development is superexponential.
I would be impressed and quite surprised if it could do novel hardware discoveries when only provided the original idea and no further prompting, and would update towards AI being even more capable than I thought.