I think this lens of incentives and the “flinching away” concept are extremely valuable for understanding the field of alignment (and less importantly, everything else:).
I believe “flinching away” is the psychological tendency that creates bigger and more obvious-on-inspection “ugh fields”. I think this is the same underlying mechanism discussed as valence by Steve Byrnes. Motivated reasoning is the name for the resulting cognitive bias. Motivated reasoning overlaps by experimental definition with confirmation bias, the one bias destroying society in Scott Alexander’s terms. After studying cognitive biases through the lens of neuroscience for years, I th nk motivated reasoning is severely hampering progress in alignment, as it is every other project. I have written about it a little in what is the most important cognitive bias to understand, but I want to address more thoroughly how it impacts alignment research.
This post makes a great start at addressing how that’s happening.
I very much agree with the analysis of incentives given here: they are strongly toward tangible and demonstrable progress in any direction vaguely related to the actual problem at hand.
This is a largely separate topic, but I happen to agree that we probably need more experienced thinkers. I disagree that physicists are obviously the best sort of experienced thinkers. I have been a physicist (as an undergrad) and I have watched physicists get into other fields. Their contributions are valuable but far from the final word and are far better when they inspire or collaborate with others with real knowledge of the target field.
There is much more to say on incentives and the field as a whole, but the remainder deserves more careful thought and separate posts.
This analysis of biases and “flinching away” could be applied to many other approaches than the prosaic alignment you target here. I think you’re correct to notice this about prosaic alignment, but it applies to many agent foundations approaches as well.
A relentless focus on the problem at hand, including its most difficult aspects, is absolutely crucial. Those difficult aspects include the theoretical concerns you link to up front, which prosaic alignment largely fails to address. But the difficult spots also include the inconvenient fact that the world is rushing toward building LLM-based or at least deep net based AGI very rapidly, and there are no good ideas about how to make them stop while we go look in a distant but more promising spot to find some keys. Most agent foundations work seems to flinch away from this aspect. Both broad schools largely flinch away from the social, political, and economic aspects of the problem.
We are a lens that can see its flaws, but we need to work to see them clearly. This difficult self-critique of locating our flinches and ugh fields is what we all as individuals, and the field as a collective, need to do to see clearly and speed up progress.
I think this lens of incentives and the “flinching away” concept are extremely valuable for understanding the field of alignment (and less importantly, everything else:).
I believe “flinching away” is the psychological tendency that creates bigger and more obvious-on-inspection “ugh fields”. I think this is the same underlying mechanism discussed as valence by Steve Byrnes. Motivated reasoning is the name for the resulting cognitive bias. Motivated reasoning overlaps by experimental definition with confirmation bias, the one bias destroying society in Scott Alexander’s terms. After studying cognitive biases through the lens of neuroscience for years, I th nk motivated reasoning is severely hampering progress in alignment, as it is every other project. I have written about it a little in what is the most important cognitive bias to understand, but I want to address more thoroughly how it impacts alignment research.
This post makes a great start at addressing how that’s happening.
I very much agree with the analysis of incentives given here: they are strongly toward tangible and demonstrable progress in any direction vaguely related to the actual problem at hand.
This is a largely separate topic, but I happen to agree that we probably need more experienced thinkers. I disagree that physicists are obviously the best sort of experienced thinkers. I have been a physicist (as an undergrad) and I have watched physicists get into other fields. Their contributions are valuable but far from the final word and are far better when they inspire or collaborate with others with real knowledge of the target field.
There is much more to say on incentives and the field as a whole, but the remainder deserves more careful thought and separate posts.
This analysis of biases and “flinching away” could be applied to many other approaches than the prosaic alignment you target here. I think you’re correct to notice this about prosaic alignment, but it applies to many agent foundations approaches as well.
A relentless focus on the problem at hand, including its most difficult aspects, is absolutely crucial. Those difficult aspects include the theoretical concerns you link to up front, which prosaic alignment largely fails to address. But the difficult spots also include the inconvenient fact that the world is rushing toward building LLM-based or at least deep net based AGI very rapidly, and there are no good ideas about how to make them stop while we go look in a distant but more promising spot to find some keys. Most agent foundations work seems to flinch away from this aspect. Both broad schools largely flinch away from the social, political, and economic aspects of the problem.
We are a lens that can see its flaws, but we need to work to see them clearly. This difficult self-critique of locating our flinches and ugh fields is what we all as individuals, and the field as a collective, need to do to see clearly and speed up progress.