Thanks for the input. I agree with most of what you’re saying. That section is trying to strike a balance between several goals:
I’m trying to keep the whole post fairly short so that people who would find it useful will read it. As a result, step 2 in particular is way too short to treat the subject of scientific methodology and practice with anything like the depth it deserves, which I try to at least say out loud in the final paragraph of that step.
For readers who have done valid scientific work, I want to make it easier for them to get their work seen, and so I’m aiming for (as you say) ‘Have you done hard work to make this actually useful to other scientists?’
For readers who haven’t done good scientific work, I want them to realize that as quickly and painlessly as possible. Hopefully step 1 will accomplish that in many cases. If it hasn’t, then step 2 (in terms of my goals as a writer) is mostly about getting people to think about whether their ideas can cash out into a falsifiable hypothesis that can make quantitative advance predictions. In cases like this that I’ve read, that’s often the problem; the person’s ideas just don’t meet those criteria at all because (for example) they’re a set of fuzzy descriptive claims that use terms in imprecise ways that don’t and can’t make concrete claims about the world.
The balance I’ve struck is really imperfect. But I suspect that if I say, ‘Well, you don’t always need a falsifiable hypothesis or an experiment’, readers who have been fooled will just assume that their ideas don’t need those things, and so it’ll do more harm than good.
Ideas on how to avoid discouraging people doing valid work without providing a way-too-tempting escape hatch are extremely welcome, from you or anyone!
Perhaps this is elitist or counter-productive to say but… do these people actually exist?
By which I mean, are there people who are using LLMs to do meaningful novel research, while also lacking the faculties/self-awareness to realize that LLMs can’t produce or verify novel ideas?
My impression has been that LLMs can only be used productively in situations where one of the following holds:
- The task is incredibly easy - Precision is not a requirement - You have enough skill that you could have done the thing on your own anyway.
In the last case in particular, LLMs are only an effort-saver, and you’d still need to verify and check every step it took. Novel research in particular requires enormous skill—I’m not sure that someone who had that skill would get to the point where they developed a whole theory without noticing it was made up.
[Also, as a meta-point, this is a great piece but I was wondering if it’s going to be posted somewhere else besides LessWrong? If the target demographic is only LW, I worry that it’s trying to have too many audience. Someone coming to this for advice would see the comments from people like me who were critiquing the piece itself, and that would certainly make it less effective. In the right place (not sure what that it) I think this could essay could be much more effective.]
I think your view here is too strong. For example, there have been papers showing that LLMs come up with ideas that human judges rate as human-level or above in blind testing. I’ve led a team doing empirical research (described here, results forthcoming) showing that current LLMs can propose and experimentally test hypotheses in novel toy scientific domains.
So while the typical claimed breakthrough isn’t real, I don’t think we can rule out real ones a priori.
If the target demographic is only LW, I worry that it’s trying to have too many audience.
I’m not sure what that means, can you clarify?
Someone coming to this for advice would see the comments from people like me who were critiquing the piece itself, and that would certainly make it less effective.
Maybe? I would guess that people who feel they have a breakthrough are usually already aware that they’re going to encounter a lot of skepticism. That’s just my intuition, though; I could be wrong.
I’m certainly open to posting it elsewhere. I posted a link to it to Reddit (in r/agi), but people who see it there have to come back here to read it. Suggestions are welcome, and I’m fine with you or anyone else posting it elsewhere with attribution (I’d appreciate getting a link to versions posted elsewhere).
Thanks for the input. I agree with most of what you’re saying. That section is trying to strike a balance between several goals:
I’m trying to keep the whole post fairly short so that people who would find it useful will read it. As a result, step 2 in particular is way too short to treat the subject of scientific methodology and practice with anything like the depth it deserves, which I try to at least say out loud in the final paragraph of that step.
For readers who have done valid scientific work, I want to make it easier for them to get their work seen, and so I’m aiming for (as you say) ‘Have you done hard work to make this actually useful to other scientists?’
For readers who haven’t done good scientific work, I want them to realize that as quickly and painlessly as possible. Hopefully step 1 will accomplish that in many cases. If it hasn’t, then step 2 (in terms of my goals as a writer) is mostly about getting people to think about whether their ideas can cash out into a falsifiable hypothesis that can make quantitative advance predictions. In cases like this that I’ve read, that’s often the problem; the person’s ideas just don’t meet those criteria at all because (for example) they’re a set of fuzzy descriptive claims that use terms in imprecise ways that don’t and can’t make concrete claims about the world.
The balance I’ve struck is really imperfect. But I suspect that if I say, ‘Well, you don’t always need a falsifiable hypothesis or an experiment’, readers who have been fooled will just assume that their ideas don’t need those things, and so it’ll do more harm than good.
Ideas on how to avoid discouraging people doing valid work without providing a way-too-tempting escape hatch are extremely welcome, from you or anyone!
Perhaps this is elitist or counter-productive to say but… do these people actually exist?
By which I mean, are there people who are using LLMs to do meaningful novel research, while also lacking the faculties/self-awareness to realize that LLMs can’t produce or verify novel ideas?
My impression has been that LLMs can only be used productively in situations where one of the following holds:
- The task is incredibly easy
- Precision is not a requirement
- You have enough skill that you could have done the thing on your own anyway.
In the last case in particular, LLMs are only an effort-saver, and you’d still need to verify and check every step it took. Novel research in particular requires enormous skill—I’m not sure that someone who had that skill would get to the point where they developed a whole theory without noticing it was made up.
[Also, as a meta-point, this is a great piece but I was wondering if it’s going to be posted somewhere else besides LessWrong? If the target demographic is only LW, I worry that it’s trying to have too many audience. Someone coming to this for advice would see the comments from people like me who were critiquing the piece itself, and that would certainly make it less effective. In the right place (not sure what that it) I think this could essay could be much more effective.]
Thanks for the reply!
I think your view here is too strong. For example, there have been papers showing that LLMs come up with ideas that human judges rate as human-level or above in blind testing. I’ve led a team doing empirical research (described here, results forthcoming) showing that current LLMs can propose and experimentally test hypotheses in novel toy scientific domains.
So while the typical claimed breakthrough isn’t real, I don’t think we can rule out real ones a priori.
I’m not sure what that means, can you clarify?
Maybe? I would guess that people who feel they have a breakthrough are usually already aware that they’re going to encounter a lot of skepticism. That’s just my intuition, though; I could be wrong.
I’m certainly open to posting it elsewhere. I posted a link to it to Reddit (in r/agi), but people who see it there have to come back here to read it. Suggestions are welcome, and I’m fine with you or anyone else posting it elsewhere with attribution (I’d appreciate getting a link to versions posted elsewhere).