I think this post is interesting, although I don’t particularly agree with the conclusions. I think it is helpful to think about the formation of your mind and goals—a tradition which I know goes back to Rousseau and most likely goes back further (I am not very knowledgeable on the topic).
I think a lot of the difficulty goes back to the distinction between ‘real’/‘intrinsic’ and goals and those people proport to believe in. Looking at your example of a Christian sexual prude, nothing about their behaviour implies to me that these virtues of chastity and distance from sex are false at all. Perhaps had they been raised in the absence of society, they would think differently. But if that were the case their views on fashion would probably vastly differ also, and yet we don’t consider their desire to dress a certain way to be falsified.
To be clear, I am not claiming that values cannot be gained societally. More that the issue at play is not the creation and adoption of these values from a childhood of coercive exposure, but that these values are hard to satisfy. Sexual prudishness makes it difficult to fulfil inbuilt sexual desires, and so combining both into a single person leaves them less overall satisfiable.
But this goes far beyond societal impacts. In a world of scarce resources, our many inbuilt physical desires as well as our entirely inbuilt desire to behave altruistically conflict in a similar way. It is difficult to be both feared and loved, as some people seem to want to be. Your concern might be better framed at this kind of conflict dragging down one’s maximum available utility than a conflict between inbuilt and constructed desires.
Nice post! I’d be interested to see what you think about the above.
As a side note, you mention that men are apparently better at aligning their sexual preferences to those of society than women. This could be evidence of men being hardwired to adopt the expectations of sexual prudishness to their core. But it could also be that men shaped those expectations to begin with through generations of societal dominance. Or a myriad of other factors which mean that for a reason entirely distinct from inbuilt preference falsification, men’s desires just line up better to those of society’s than women’s.
I agree that in a takeover scenario where AI capabilities rush wildly ahead of human understanding or control, the ability of the world’s second species to retain exclusive resource access will be limited. This is a plausible future, but it is not the only one. A lot of effort is, and very likely will continue to be, directed towards controlling frontier AI and making it as economically beneficial for its owners as possible.
If this work bears fruit, a world where AI is made by people with capital for people with capital seems very likely.
The French nobility had their capital taken by force by revolutionaries in a violent takeover—the kind which AI may execute. An analogy for the kind of situation I am concerned about would be some portion of the French nobility making the laws so complex, the financial structure of the kingdom so riddled with loopholes, that the other nobles were disenfranchised by due process rather than violence.
Taking this analogy further, the French king had to care about the nobility, and the nobility about the peasantry, in part because of their productive capacity. As described in The Intelligence Curse — LessWrong, when AGI renders your work valueless the incentive for the state to invest in your welfare greatly decreases.