I for one think that’s a great process and you should definitely include that in such a post to indicate how hard you worked to supply the metacognitive skills, judgment, and skepticism the LLMs tend to lack.
I think explaining that process would get more readers and buy-in; if you just say “some model spit this out” people won’t assume you’ve done that much careful work in prompting them to assemble something valid and meaningful.
I just went through a round of using LLMs for research, and my conclusion was that they need to be prompted to find flaws in methodology or interpretation in each paper individually. They’re too credulous in general. FWIW.
I think it would be ideal if you’d identified the alternate explanation Benquo gave; the lack of bioavailability seems like a glaring problem for your theory. But I also think that sort of problem is pretty likely in a thoroughly human-written post too, and it’s quite useful to write it anyway so that public discussion like that happens!
I’m uncertain whether I think supplementation is warranted (pending researching for possible side effects) given that exchange, so I’d love to see your further thoughts on that.
I did explicitly write about microbiome effects. The bioavailability paper Simec et al 2023 that Benquo mention does argue that their findings point towards systematic regulatory effects.
“The poor bioavailability (~0.2 %) of oral hyaluronan indicates that the mechanism of action is the result of the systematic regulatory function of hyaluronan or its metabolites rather than the direct effects of hyaluronan at distal sites of action (skin, joints). [...] The results of the present study suggest that orally ad-ministered HA is degraded to oligosaccharides by bacteria in the cecum, and oligosaccharide HA migrates to the skin through the blood or lymph. It is expected that the absorbed oligosaccharide HA participates in the various effects of orally administered HA”
Most of what I wrote focuses on systemic regulatory action through CD44 activation. Benquo’s hypothesis of all effects being due to microbiome effects might be true, but it’s a contrarian position that’s not held by the people who actually did the bioavailability research. Simec does not even mention Benque’s hypnothesis as one worth considering an running experiments to see whether it’s true.
I do think Benque’s hypnothesis is valid enough that someone in the field should run an experiment, especially given that microbiome interventions can both help with skin and joint outcomes, but when writing a post like this, I think it’s fine to not be contrarian and go with the views of the people running the bioavailability experiments instead of making contrarian interpretations on them.
But I also think that sort of problem is pretty likely in a thoroughly human-written post too
I think it’s more likely that that I’m going to argue a contrarian position instead of just going with the views that the scientists who are experts in the field hold when I’m written a human-written post, but I don’t think that’s an improvement. When it comes to writing like this, I think LLM’s helping grounding the argument from the perspective of domain experts is an improvement.
Sorry I hadn’t tracked your argument structure better.
I agree that you don’t need to take a contrarian position to make it a very worthwhile LW post. I was just suggesting that getting a good grip on the literature benefits from prompting your LLM assistant to take a contrarian view on each paper individually, to help you identify potential alternative explanations. I think many researchers’ interpretations of their research are pretty suspect, even though the research is going to be reported accurately. There’s a lot of motivated reasoning bias toward interpretations that make the research important.
But that’s a minor quibble. I think your methodology is exemplary for how to use LLMs to speed your research and writing, including prompting them to take a skeptical stance sometimes.
I hope to see more from you and others in this vein. I do think it’s important to say exactly how you used the LLMs; I trust this far more than if someone had just prompted Gemini to write this essay. You are supplying the metacognition, judgment, and executive function that the models currently lack.
I’m sorry this post didn’t wind up with a higher vote total. I think it was really good. I attribute this to suspicion from the community about a post that basically just says “some LLM wrote this” instead of saying how carefully you used combinations of LLMs as research and writing assistants.
I for one think that’s a great process and you should definitely include that in such a post to indicate how hard you worked to supply the metacognitive skills, judgment, and skepticism the LLMs tend to lack.
I think explaining that process would get more readers and buy-in; if you just say “some model spit this out” people won’t assume you’ve done that much careful work in prompting them to assemble something valid and meaningful.
I just went through a round of using LLMs for research, and my conclusion was that they need to be prompted to find flaws in methodology or interpretation in each paper individually. They’re too credulous in general. FWIW.
I think it would be ideal if you’d identified the alternate explanation Benquo gave; the lack of bioavailability seems like a glaring problem for your theory. But I also think that sort of problem is pretty likely in a thoroughly human-written post too, and it’s quite useful to write it anyway so that public discussion like that happens!
I’m uncertain whether I think supplementation is warranted (pending researching for possible side effects) given that exchange, so I’d love to see your further thoughts on that.
I did explicitly write about microbiome effects. The bioavailability paper Simec et al 2023 that Benquo mention does argue that their findings point towards systematic regulatory effects.
Most of what I wrote focuses on systemic regulatory action through CD44 activation. Benquo’s hypothesis of all effects being due to microbiome effects might be true, but it’s a contrarian position that’s not held by the people who actually did the bioavailability research. Simec does not even mention Benque’s hypnothesis as one worth considering an running experiments to see whether it’s true.
I do think Benque’s hypnothesis is valid enough that someone in the field should run an experiment, especially given that microbiome interventions can both help with skin and joint outcomes, but when writing a post like this, I think it’s fine to not be contrarian and go with the views of the people running the bioavailability experiments instead of making contrarian interpretations on them.
I think it’s more likely that that I’m going to argue a contrarian position instead of just going with the views that the scientists who are experts in the field hold when I’m written a human-written post, but I don’t think that’s an improvement. When it comes to writing like this, I think LLM’s helping grounding the argument from the perspective of domain experts is an improvement.
Sorry I hadn’t tracked your argument structure better.
I agree that you don’t need to take a contrarian position to make it a very worthwhile LW post. I was just suggesting that getting a good grip on the literature benefits from prompting your LLM assistant to take a contrarian view on each paper individually, to help you identify potential alternative explanations. I think many researchers’ interpretations of their research are pretty suspect, even though the research is going to be reported accurately. There’s a lot of motivated reasoning bias toward interpretations that make the research important.
But that’s a minor quibble. I think your methodology is exemplary for how to use LLMs to speed your research and writing, including prompting them to take a skeptical stance sometimes.
I hope to see more from you and others in this vein. I do think it’s important to say exactly how you used the LLMs; I trust this far more than if someone had just prompted Gemini to write this essay. You are supplying the metacognition, judgment, and executive function that the models currently lack.
I’m sorry this post didn’t wind up with a higher vote total. I think it was really good. I attribute this to suspicion from the community about a post that basically just says “some LLM wrote this” instead of saying how carefully you used combinations of LLMs as research and writing assistants.