Peer review usually results in papers being accepted with minor or major revisions, and very much can and does effect serious changes in the study design. You can read the peer review back-and-forth in many journals, they are often pretty interesting. Machine learning and computer science are different because they usually publish in conference proceedings. That means there are very tight deadlines, so it’s more common to rebut the reviewer’s comments outside of very minor changes. In my opinion it’s why peer review is seen so poorly in ML, because there’s not much paper-improvement going on as a result of the process.
Peer review usually results in papers being accepted with minor or major revisions, and very much can and does effect serious changes in the study design.
Ok… how does this work exactly? You submit your paper for peer review, they say “you should’ve done this differently from the start”, and you go back and start over…?
Isn’t that… basically what I said? You submit a basically finished product, which might get rejected and you have to start over. But you’re not submitting a paper for peer review midway through the study, right…?
You submit a finished product, yes, and it can be accepted without revisions, but I have never heard of that happening actually and nobody I know has had that happen to them, I believe. Or, it might get rejected (but if so, no, you don’t have to start over. If it was sent for review, you will receive feedback you can use to improve the study, and you may be invited to resubmit after making those changes, or you might submit the same paper to a different journal). Hopefully, it is accepted with major or minor revisions, so you go away and make the requested changes over a few more months, and then the reviewers take another look. And these changes can, but not always, be significant alterations to the study design.
Examples from my recent experience: I submitted a paper recently that developed a new data analysis method and then evaluated it on two different synthetic datasets. I was then asked by the editor for revisions: obtaining and using observational data as well as synthetic data. That’s not changing the original study design, but it is a new chunk of research, a lot of work, and the results have to be interpreted differently. Another paper that I co-authored has been asked for major revisions which, if implemented, would be a massive change in the setup, data used, analysis methodology and narrative of the paper. The lead author is still deciding if they want to do that or instead to withdraw and resubmit somewhere else. On the other hand, often I have only been asked for minor text changes to explain things more clearly.
In Nature, the peer review files are openly available for each article, and they are pretty interesting to read, because papers there often go through quite significant changes before publication. That’s a good way to get an idea of the ways papers and studies can evolve as they go through the peer review process. But, yeah, I assure you, in my experience as an author and reviewer, it is a collaborative process that can really reshape the study design in some cases.
Peer review usually results in papers being accepted with minor or major revisions, and very much can and does effect serious changes in the study design. You can read the peer review back-and-forth in many journals, they are often pretty interesting. Machine learning and computer science are different because they usually publish in conference proceedings. That means there are very tight deadlines, so it’s more common to rebut the reviewer’s comments outside of very minor changes. In my opinion it’s why peer review is seen so poorly in ML, because there’s not much paper-improvement going on as a result of the process.
The study design… of the paper being reviewed?
Yes.
Ok… how does this work exactly? You submit your paper for peer review, they say “you should’ve done this differently from the start”, and you go back and start over…?
Isn’t that… basically what I said? You submit a basically finished product, which might get rejected and you have to start over. But you’re not submitting a paper for peer review midway through the study, right…?
You submit a finished product, yes, and it can be accepted without revisions, but I have never heard of that happening actually and nobody I know has had that happen to them, I believe. Or, it might get rejected (but if so, no, you don’t have to start over. If it was sent for review, you will receive feedback you can use to improve the study, and you may be invited to resubmit after making those changes, or you might submit the same paper to a different journal). Hopefully, it is accepted with major or minor revisions, so you go away and make the requested changes over a few more months, and then the reviewers take another look. And these changes can, but not always, be significant alterations to the study design.
Examples from my recent experience: I submitted a paper recently that developed a new data analysis method and then evaluated it on two different synthetic datasets. I was then asked by the editor for revisions: obtaining and using observational data as well as synthetic data. That’s not changing the original study design, but it is a new chunk of research, a lot of work, and the results have to be interpreted differently. Another paper that I co-authored has been asked for major revisions which, if implemented, would be a massive change in the setup, data used, analysis methodology and narrative of the paper. The lead author is still deciding if they want to do that or instead to withdraw and resubmit somewhere else. On the other hand, often I have only been asked for minor text changes to explain things more clearly.
In Nature, the peer review files are openly available for each article, and they are pretty interesting to read, because papers there often go through quite significant changes before publication. That’s a good way to get an idea of the ways papers and studies can evolve as they go through the peer review process. But, yeah, I assure you, in my experience as an author and reviewer, it is a collaborative process that can really reshape the study design in some cases.