Your LLM-assisted scientific breakthrough probably isn’t real
I’ve been encountering an increasing number of cases recently of (often very smart) people becoming convinced that they’ve made an important scientific breakthrough with the help of an LLM. Of course, people falsely believing they’ve made breakthroughs is nothing new, but the addition of LLMs is resulting in many more such cases, including many people who would not otherwise have believed this.
This is related to what’s described in So You Think You’ve Awoken ChatGPT and When AI Seems Conscious, but it’s different enough that many people don’t see themselves in those essays even though they’re falling into a similar trap.
So I’ve spent much of today writing up something to point people to. I would really appreciate feedback! My priority here is to help people accept that they might be wrong, and get them to do some reality checking, without making them feel dumb — if I’m failing in parts of this to come across as respecting the reader, please let me know that! Also I wrote this kind of quickly, so it’s very possible I’m forgetting to say important things, and would love to have those pointed out. My main goal is to be as helpful to such people as possible.
Use the following prompt to evaluate it: Please take a look at the attached project and provide a careful critical analysis of it from a scientific perspective. Start with a 200-word summary of the project...
I had, with Claude-Opus-4.1 and Gemini-2.5-Pro, but only with n=1, using a real-world case where I was contacted by someone who felt they had such a breakthrough. I love your idea of trying it on rejected LW posts!
<tries it on three rejected LW posts, chosen quickly based on rejection tag, title, and length. Names omitted for politeness’s sake, clickthrough available>
GPT-5-Thinking: ‘Scientific validity...Low at present; salvageable as a toy model with substantial work.’
Gemini-2.5-Pro: ‘Scientific Validity...The project is not scientifically valid in its current form because its mathematical foundation is critically flawed.’
GPT-5-Thinking: ‘The doc is not a lone-user “I’ve discovered X” claim; it’s a measured integration plan with explicit eval gates and rollback...Where self-deception could creep in is metric design.’
Claude-Opus-4.1: ‘The individual components are scientifically valid...However, the leap from “combining existing techniques” to “achieving AGI” lacks scientific justification. While the proposal addresses real challenges like catastrophic forgetting and sample efficiency, there’s no evidence or theoretical argument for why this particular combination would produce general intelligence.’
Claude-Opus-4.1: ‘Scientific Validity low to moderate...While the document engages with legitimate questions in consciousness studies, it lacks the rigor expected of scientific work...It’s closer to amateur philosophy of mind than scientific theory.’
Gemini-2.5-Pro: ‘Scientific Validity: To a low extent...not a scientifically valid theory...best classified as philosophy of mind.’
Those seem like fairly reasonable results. Case 2 is jargon-heavy and hard to evaluate, but it passes my ‘not obvious nonsense and not blatantly unscientific’ filter, at least on a quick read, so I think it’s good that it’s not fully rejected by the LLMs.
[EDIT: posted, feedback is welcome there]
Request for feedback on draft post:
Your LLM-assisted scientific breakthrough probably isn’t real
I’ve been encountering an increasing number of cases recently of (often very smart) people becoming convinced that they’ve made an important scientific breakthrough with the help of an LLM. Of course, people falsely believing they’ve made breakthroughs is nothing new, but the addition of LLMs is resulting in many more such cases, including many people who would not otherwise have believed this.
This is related to what’s described in So You Think You’ve Awoken ChatGPT and When AI Seems Conscious, but it’s different enough that many people don’t see themselves in those essays even though they’re falling into a similar trap.
So I’ve spent much of today writing up something to point people to. I would really appreciate feedback! My priority here is to help people accept that they might be wrong, and get them to do some reality checking, without making them feel dumb — if I’m failing in parts of this to come across as respecting the reader, please let me know that! Also I wrote this kind of quickly, so it’s very possible I’m forgetting to say important things, and would love to have those pointed out. My main goal is to be as helpful to such people as possible.
[EDIT: posted, feedback is welcome there]
Have you actually tried doing that, e.g. with the https://www.lesswrong.com/moderation#rejected-posts ?
I had, with Claude-Opus-4.1 and Gemini-2.5-Pro, but only with n=1, using a real-world case where I was contacted by someone who felt they had such a breakthrough. I love your idea of trying it on rejected LW posts!
<tries it on three rejected LW posts, chosen quickly based on rejection tag, title, and length. Names omitted for politeness’s sake, clickthrough available>
Case 1:
GPT-5-Thinking: ‘Scientific validity...Low at present; salvageable as a toy model with substantial work.’
Gemini-2.5-Pro: ‘Scientific Validity...The project is not scientifically valid in its current form because its mathematical foundation is critically flawed.’
Case 2:
GPT-5-Thinking: ‘The doc is not a lone-user “I’ve discovered X” claim; it’s a measured integration plan with explicit eval gates and rollback...Where self-deception could creep in is metric design.’
Claude-Opus-4.1: ‘The individual components are scientifically valid...However, the leap from “combining existing techniques” to “achieving AGI” lacks scientific justification. While the proposal addresses real challenges like catastrophic forgetting and sample efficiency, there’s no evidence or theoretical argument for why this particular combination would produce general intelligence.’
Case 3:
Claude-Opus-4.1: ‘Scientific Validity low to moderate...While the document engages with legitimate questions in consciousness studies, it lacks the rigor expected of scientific work...It’s closer to amateur philosophy of mind than scientific theory.’
Gemini-2.5-Pro: ‘Scientific Validity: To a low extent...not a scientifically valid theory...best classified as philosophy of mind.’
Those seem like fairly reasonable results. Case 2 is jargon-heavy and hard to evaluate, but it passes my ‘not obvious nonsense and not blatantly unscientific’ filter, at least on a quick read, so I think it’s good that it’s not fully rejected by the LLMs.