I don’t actually think “It is really hard to know what sorts of AI alignment work are good this far out from transformative AI.” is very helpful.
It is currently fairly hard to tell what is good alignment work. A week from TAI, then either, good alignment work will be easier to recognise because of alignment progress not strongly correlated with capabilities, or good alignment research is just as hard to recognise. (More likely the latter) I can’t think of any safety research that can be done on GPT3 that can’t be done on GPT1.
In my picture, research gets done and theorems proved, researcher population grows as funding increases and talent matures. Toy models get produced. Once you can easily write down a description of a FAI with unbounded compute, that’s when you start to look at algorithms that have good capabilities in practice.
I don’t actually think “It is really hard to know what sorts of AI alignment work are good this far out from transformative AI.” is very helpful.
It is currently fairly hard to tell what is good alignment work. A week from TAI, then either, good alignment work will be easier to recognise because of alignment progress not strongly correlated with capabilities, or good alignment research is just as hard to recognise. (More likely the latter) I can’t think of any safety research that can be done on GPT3 that can’t be done on GPT1.
In my picture, research gets done and theorems proved, researcher population grows as funding increases and talent matures. Toy models get produced. Once you can easily write down a description of a FAI with unbounded compute, that’s when you start to look at algorithms that have good capabilities in practice.