My main advice to avoid this failure mode is to leverage your Pareto frontier. Apply whatever knowledge, or combination of knowledge, you have which others in the field don’t.
This makes sense if you already have knowledge which other people don’t, but what about if you don’t? How much should “number of people in the alignment community who already know X thing” factor into what you decide to study, relative to other factors like “how useful is X thing, when you ignore what everyone else is doing?” For instance, there are probably fewer people who know a lot about geology than who know a lot about economics, but I would expect that learning about economics would still be more valuable for doing agent foundations research.
(My guess is that the answer is “don’t worry much at all about the Pareto frontier stuff when deciding what to study,” especially because there aren’t that many alignment researchers anyways, but I’m not actually sure.)
My expectation is that if you do the Alignment Game Tree exercise and maybe a few others like it relatively early, and generally study what seems useful from there, and update along the way as you learn more stuff, you’ll end up reasonably-differentiated from other researchers by default. On the other hand, if you find yourself literally only studying ML, then that would be a clear sign that you should diversify more (and also I would guess that’s an indicator that you haven’t gone very deep into the Game Tree).
This makes sense if you already have knowledge which other people don’t, but what about if you don’t? How much should “number of people in the alignment community who already know X thing” factor into what you decide to study, relative to other factors like “how useful is X thing, when you ignore what everyone else is doing?” For instance, there are probably fewer people who know a lot about geology than who know a lot about economics, but I would expect that learning about economics would still be more valuable for doing agent foundations research.
(My guess is that the answer is “don’t worry much at all about the Pareto frontier stuff when deciding what to study,” especially because there aren’t that many alignment researchers anyways, but I’m not actually sure.)
My expectation is that if you do the Alignment Game Tree exercise and maybe a few others like it relatively early, and generally study what seems useful from there, and update along the way as you learn more stuff, you’ll end up reasonably-differentiated from other researchers by default. On the other hand, if you find yourself literally only studying ML, then that would be a clear sign that you should diversify more (and also I would guess that’s an indicator that you haven’t gone very deep into the Game Tree).