I think people on LW somewhat overestimate winning over persistence, especially for generating new perspectives and ideas on deep issues.
If we look at AI Safety to now as a pre-paradigmatic field then one of the main ways of progress is finding frames and good questions to ask. What determines the degree to which a field is good at asking questions? Partly the independently distributed information it can gain, partly the actionability of that information.
Now, what does that mean for you as a theorist? Well, you should have something that 1. brings new useful information to the table and that 2. interfaces well with existing models so people can make progress.
Since rationalists want to win, I think that they on average underestimate pursuits of 1 in how it affects 2nd order causation in the community. You’re not likely to win nor get good feedback very often when you generate new information due to low hanging fruit being already picked with a high likelihood so if you want to bring in new fruit from the tree you need to disregard local reward signals for a bit.
I would finally like to make the argument as this is more important than ever since it seems that it is the theoretical serial time that will bottleneck future AI Safety progress since well scoped questions are likely to have a higher degree of LLM parallelisation to them.
I think people on LW somewhat overestimate winning over persistence, especially for generating new perspectives and ideas on deep issues.
If we look at AI Safety to now as a pre-paradigmatic field then one of the main ways of progress is finding frames and good questions to ask. What determines the degree to which a field is good at asking questions? Partly the independently distributed information it can gain, partly the actionability of that information.
Now, what does that mean for you as a theorist? Well, you should have something that 1. brings new useful information to the table and that 2. interfaces well with existing models so people can make progress.
Since rationalists want to win, I think that they on average underestimate pursuits of 1 in how it affects 2nd order causation in the community. You’re not likely to win nor get good feedback very often when you generate new information due to low hanging fruit being already picked with a high likelihood so if you want to bring in new fruit from the tree you need to disregard local reward signals for a bit.
I would finally like to make the argument as this is more important than ever since it seems that it is the theoretical serial time that will bottleneck future AI Safety progress since well scoped questions are likely to have a higher degree of LLM parallelisation to them.