Listing these in part so that I hopefully learn from them, but also because I think some of these are common among junior researchers, so maybe it’s helpful for someone else.
I had an idea I liked and stayed attached to it too heavily.
What I should have done was focus more on simpler baselines, and be more scared when I couldn’t beat those simple baselines.
(By “simpler baselines,” I don’t just mean comparing against what other people are using, I also mean ablations where you remove some parts of the method and see if it still works.)
Notably, several researchers I respect told me I shouldn’t be too attached to that idea and should consider using simpler methods.
Apart from being a time sink, this also had bad effects on my research prioritization. It nudged me toward doing experiments that were easy to do with the library or that would be natural additions to the library when implemented. It made it aversive to do experiments that wouldn’t fit naturally into the library at all.
It also made fast iteration more difficult, because I’d often be tempted to quickly integrate some new method/tool into the library instead of doing a hacky version first.
I do still think clean research code and infrastructure are really valuable, so this is difficult to balance. Consider reversing advice especially on this point.
I worked on purely conceptual/theoretical work without collaborators or good mentorship, and with only ~2.5 years of research experience. I expected beforehand that I’d be unusually good at this type of research (and I still think that’s plausibly true), but even so, I don’t think it was time well spent in expectation.
I’m very sympathetic to this kind of work being important. But I think it’s really brutal (and won’t work) without at least one (and ideally more) of (1) strong collaborators and ideally mentors, (2) good empirical feedback loops, (3) a lot of research experience. Maybe there are a small handful of junior researchers who can do useful things there on their own, but I’ve been growing more and more skeptical.
I looked into related work too late several times, and didn’t think about how my work was going to be different early enough.
Drafting a paper outline was great for making me confront that question (and is also a useful exercise for noticing other mistakes).
I didn’t work on control. I was convinced that control was great pretty quickly, and then … assumed that everyone else would also be convinced and lots of people would switch to working on it, and it would end up being non-neglected. This sounds really silly in hindsight, and I suspect I was also doing motivated reasoning to avoid changing my research.
“Working on control” is a messy concept. Mechanistic anomaly detection (which I was working on) seems at least as applicable to control as to alignment. But what I mean by “working on control” includes an associated cluster of research taste aesthetics, such as pessimistic assumptions about inductive biases, meta-level adversarial evals, and thinking about the best protocols a blue team could use given clearly specified (and realistic) rules. I think I’ve been slowly moving closer toward that cluster, but could have gotten there sooner by making a deliberate sudden switch.
I didn’t explicitly realize how “working on control” has this research taste cluster associated with it until more recently, otherwise the “everyone will work on it” argument would’ve had less force anyway.
Agree. There is that old saying about even fools learning from their own mistakes but wise men learning from the mistakes of others. But if everyone is trying to hide their mistakes, that might limit how much learning the wise can do.
I had not really thought about this before, but after seeing your comment the question struck me if social/cultural norms about social status and “loosing face” don’t impact scientific advancement.
It’s kind of strange that, from my perspective, these mistakes are very similar to the mistakes I think I made, and also see a lot of other people making. Perhaps one “must” spend too long doing abstract slippery stuff to really understand the nature of why it doesn’t really work that well?
Yeah. I think there’s a broader phenomenon where it’s way harder to learn from other people’s mistakes than from your own. E.g. see my first bullet point on being too attached to a cool idea. Obviously, I knew in theory that this was a common failure mode (from the Sequences/LW and from common research advice), and someone even told me I might be making the mistake in this specific instance. But my experience up until that point had been that most of the research ideas I’d been similarly excited about ended up ~working (or at least the ones I put serious time into).
curious if you have takes on the right balance between clean research code / infrastructure and moving fast/being flexible. Maybe its some combination of
get hacky results ASAP
lean more towards functional programming / general tools and away from object-oriented programing / frameworks (especially early in projects where the abstractions/experiments/ research questions are more dynamic), but don’t sacrifice code quality and standard practices
~All code should start as a hacky jupyter notebook (like your first point)
As my codebase grows larger and messier, I usually hit a point where it becomes more aversive to work with because I don’t know where things are, there’s too much duplication, etc. Refactor at that point.
When refactoring, don’t add abstractions just because they might become useful in the future. (You still want to think about future plans somewhat of course, maybe the heuristic is to not write code that’s much more verbose than necessary right now, in the hope that it will pay off in the future.)
These are probably geared toward people like me who tend to over-engineer; someone who’s currently unhappy that their code is always a mess might need different ones.
I don’t know whether functional programming is fundamentally better in this respect than object-oriented.
Research mistakes I made over the last 2 years.
Listing these in part so that I hopefully learn from them, but also because I think some of these are common among junior researchers, so maybe it’s helpful for someone else.
I had an idea I liked and stayed attached to it too heavily.
(The idea is using abstractions of computation for mechanistic anomaly detection. I still think there could be something there, but I wasted a lot of time on it.)
What I should have done was focus more on simpler baselines, and be more scared when I couldn’t beat those simple baselines.
(By “simpler baselines,” I don’t just mean comparing against what other people are using, I also mean ablations where you remove some parts of the method and see if it still works.)
Notably, several researchers I respect told me I shouldn’t be too attached to that idea and should consider using simpler methods.
I was too focused initially on growing the field of empirical mechanistic anomaly detection; I should have just tried to get interesting results first.
Relatedly, I spent too much time making a nice library for mechanistic anomaly detection (though in part this was for my own use, not just because of field-building).
Apart from being a time sink, this also had bad effects on my research prioritization. It nudged me toward doing experiments that were easy to do with the library or that would be natural additions to the library when implemented. It made it aversive to do experiments that wouldn’t fit naturally into the library at all.
It also made fast iteration more difficult, because I’d often be tempted to quickly integrate some new method/tool into the library instead of doing a hacky version first.
I do still think clean research code and infrastructure are really valuable, so this is difficult to balance. Consider reversing advice especially on this point.
I worked on purely conceptual/theoretical work without collaborators or good mentorship, and with only ~2.5 years of research experience. I expected beforehand that I’d be unusually good at this type of research (and I still think that’s plausibly true), but even so, I don’t think it was time well spent in expectation.
I’m very sympathetic to this kind of work being important. But I think it’s really brutal (and won’t work) without at least one (and ideally more) of (1) strong collaborators and ideally mentors, (2) good empirical feedback loops, (3) a lot of research experience. Maybe there are a small handful of junior researchers who can do useful things there on their own, but I’ve been growing more and more skeptical.
I looked into related work too late several times, and didn’t think about how my work was going to be different early enough.
Drafting a paper outline was great for making me confront that question (and is also a useful exercise for noticing other mistakes).
I didn’t work on control. I was convinced that control was great pretty quickly, and then … assumed that everyone else would also be convinced and lots of people would switch to working on it, and it would end up being non-neglected. This sounds really silly in hindsight, and I suspect I was also doing motivated reasoning to avoid changing my research.
“Working on control” is a messy concept. Mechanistic anomaly detection (which I was working on) seems at least as applicable to control as to alignment. But what I mean by “working on control” includes an associated cluster of research taste aesthetics, such as pessimistic assumptions about inductive biases, meta-level adversarial evals, and thinking about the best protocols a blue team could use given clearly specified (and realistic) rules. I think I’ve been slowly moving closer toward that cluster, but could have gotten there sooner by making a deliberate sudden switch.
I didn’t explicitly realize how “working on control” has this research taste cluster associated with it until more recently, otherwise the “everyone will work on it” argument would’ve had less force anyway.
I also have most of these regrets when I think about my work in 2022.
I am a big fan of the humility and the intention to help others by openly reflecting on these lessons, thank you for that.
Agree. There is that old saying about even fools learning from their own mistakes but wise men learning from the mistakes of others. But if everyone is trying to hide their mistakes, that might limit how much learning the wise can do.
I had not really thought about this before, but after seeing your comment the question struck me if social/cultural norms about social status and “loosing face” don’t impact scientific advancement.
It’s kind of strange that, from my perspective, these mistakes are very similar to the mistakes I think I made, and also see a lot of other people making. Perhaps one “must” spend too long doing abstract slippery stuff to really understand the nature of why it doesn’t really work that well?
Yeah. I think there’s a broader phenomenon where it’s way harder to learn from other people’s mistakes than from your own. E.g. see my first bullet point on being too attached to a cool idea. Obviously, I knew in theory that this was a common failure mode (from the Sequences/LW and from common research advice), and someone even told me I might be making the mistake in this specific instance. But my experience up until that point had been that most of the research ideas I’d been similarly excited about ended up ~working (or at least the ones I put serious time into).
curious if you have takes on the right balance between clean research code / infrastructure and moving fast/being flexible. Maybe its some combination of
get hacky results ASAP
lean more towards functional programming / general tools and away from object-oriented programing / frameworks (especially early in projects where the abstractions/experiments/ research questions are more dynamic), but don’t sacrifice code quality and standard practices
Some heuristics (not hard rules):
~All code should start as a hacky jupyter notebook (like your first point)
As my codebase grows larger and messier, I usually hit a point where it becomes more aversive to work with because I don’t know where things are, there’s too much duplication, etc. Refactor at that point.
When refactoring, don’t add abstractions just because they might become useful in the future. (You still want to think about future plans somewhat of course, maybe the heuristic is to not write code that’s much more verbose than necessary right now, in the hope that it will pay off in the future.)
These are probably geared toward people like me who tend to over-engineer; someone who’s currently unhappy that their code is always a mess might need different ones.
I don’t know whether functional programming is fundamentally better in this respect than object-oriented.