I’ve forked and tried to set up a lot of AI safety repos (this is the default action I take when reading a paper which links to code). I’ve also reached out to authors directly whenever I’ve had trouble with reproducing their results. There aren’t any particular patterns that stand out, but I think that writing a top-level post that describes your contention with a paper’s findings is something that the community would be very welcoming to and indeed is how science advances.
I’ve forked and tried to set up a lot of AI safety repos (this is the default action I take when reading a paper which links to code). I’ve also reached out to authors directly whenever I’ve had trouble with reproducing their results.
Out of curiosity:
How often do you end up feeling like there was at least one misleading claim in the paper?
How do the authors react when you contact them with your issues?
How often do you end up feeling like there was at least one misleading claim in the paper?
I am easily and frequently confused, but this is mostly because I find it difficult to thoroughly understand other people’s work in a lot of detail in a short amount of time.
How do the authors react when you contact them with your issues?
I usually get a response within two weeks. If they have a startup background, then this delay is much lower, by multiple orders of magnitude. Authors are typically glad that I am trying to run follow up experiments on their work and give me one to two sentences of feedback over email. Corresponding authors are sometimes bad at taking correspondence, contact information for committers can be found in commit logs via git blame. If it is a problem that may be relevant to other people, I link to a GH issue.
More junior authors tend to be willing to schedule a multi-hour call going over files line-by-line and will also read and give their thoughts on any related work that you share with them.
In the middle ranks, what tends to happen is that you get invited to help review the new project that they are currently working on, or if they’ve shifted directions then you get pointed to someone who has produced some unpublished results in the same general area.
Very senior authors can be politely flagged down during in-person conferences, or even if they’re not presenting personally, someone from their group almost always attends.
I’ve forked and tried to set up a lot of AI safety repos (this is the default action I take when reading a paper which links to code). I’ve also reached out to authors directly whenever I’ve had trouble with reproducing their results. There aren’t any particular patterns that stand out, but I think that writing a top-level post that describes your contention with a paper’s findings is something that the community would be very welcoming to and indeed is how science advances.
Out of curiosity:
How often do you end up feeling like there was at least one misleading claim in the paper?
How do the authors react when you contact them with your issues?
I am easily and frequently confused, but this is mostly because I find it difficult to thoroughly understand other people’s work in a lot of detail in a short amount of time.
I usually get a response within two weeks. If they have a startup background, then this delay is much lower, by multiple orders of magnitude. Authors are typically glad that I am trying to run follow up experiments on their work and give me one to two sentences of feedback over email. Corresponding authors are sometimes bad at taking correspondence, contact information for committers can be found in commit logs via git blame. If it is a problem that may be relevant to other people, I link to a GH issue.
More junior authors tend to be willing to schedule a multi-hour call going over files line-by-line and will also read and give their thoughts on any related work that you share with them.
In the middle ranks, what tends to happen is that you get invited to help review the new project that they are currently working on, or if they’ve shifted directions then you get pointed to someone who has produced some unpublished results in the same general area.
Very senior authors can be politely flagged down during in-person conferences, or even if they’re not presenting personally, someone from their group almost always attends.