I agree! Better and more AI discourse on LW would be great.
One narrow disagreement:
E.g., it would be better if people on LW applied something more like typical researcher norms to research outputs (e.g., for many types of concerns, email the author with the concerns and see if they fix before posting publicly) and tried to avoid their criticism being unnecessarily rude (though not necessarily less hostile)
I endorse posting publicly because
Speed is important. If you wait a few days, a bunch of people will read the summary of the research artifact and some discussion but won’t read your critique.
Often you don’t really care about the artifact’s details, but it has issues that allow you to convincingly argue that the process that generated the artifact is broken. In such cases you don’t want the author to tweak the paper; you just want to share the critique.
More generally, often authors will make superficial changes in response to your criticism—papering over the issues you point out to them—which just makes it harder to later publicly demonstrate the deep issues with the artifact. This happened to me yesterday: I read a doc and I’m glad I posted my critique in slack up front because the author responded to my concerns in a superficial way, marginally improving the doc but making it harder to see that my deeper critique was true.
I agree! Better and more AI discourse on LW would be great.
One narrow disagreement:
I endorse posting publicly because
Speed is important. If you wait a few days, a bunch of people will read the summary of the research artifact and some discussion but won’t read your critique.
Often you don’t really care about the artifact’s details, but it has issues that allow you to convincingly argue that the process that generated the artifact is broken. In such cases you don’t want the author to tweak the paper; you just want to share the critique.
More generally, often authors will make superficial changes in response to your criticism—papering over the issues you point out to them—which just makes it harder to later publicly demonstrate the deep issues with the artifact. This happened to me yesterday: I read a doc and I’m glad I posted my critique in slack up front because the author responded to my concerns in a superficial way, marginally improving the doc but making it harder to see that my deeper critique was true.