good points. But a counterpoint: If a community forms around a certain set of virtues and rules at time T, then that set becomes sticky and harder to change and therefore harder to twist into letting you do the power-seeking or cowardly or status-seeking thing later. For people who don’t have any community or ethical identity, yes, they can choose from the menu the one that most helps them get status power etc. But then once they’ve chosen it, there are switching costs. By contrast if you are a consequentialist, you can simply convince yourself that actually you need to JOIN the AGI company, or that actually you need to ACCELERATE chip production, or whatever. The evidence about what’ll have good vs. bad consequences is constantly changing, so if you change your mind for rationalizing reasons it can be disguised (to yourself and others) as a change of mind based on good evidence.
good points. But a counterpoint: If a community forms around a certain set of virtues and rules at time T, then that set becomes sticky and harder to change and therefore harder to twist into letting you do the power-seeking or cowardly or status-seeking thing later. For people who don’t have any community or ethical identity, yes, they can choose from the menu the one that most helps them get status power etc. But then once they’ve chosen it, there are switching costs. By contrast if you are a consequentialist, you can simply convince yourself that actually you need to JOIN the AGI company, or that actually you need to ACCELERATE chip production, or whatever. The evidence about what’ll have good vs. bad consequences is constantly changing, so if you change your mind for rationalizing reasons it can be disguised (to yourself and others) as a change of mind based on good evidence.
simple solution: if you’re consequentialist, you should be dogmatically attached to your beliefs and refuse to change them regardless of the evidence.