the AGI might self-modify to be more coherent in a way that involves crushing / erasing a subset of its desires, and this subset might include the desires related to human flourishing. This is analogous to how you or I might try to self-modify to erase some of our (unendorsed) desires (to eat junk food, be selfish, be cruel, etc.), if we could
This seems like a big deal to me, because it feels like if I could modify myself, I would probably do so to make myself better at achieving a handful of goals like {having a positive impact, (maybe) obtaining a large amount of power/money, getting really really good at a particular skill} and everything else like {desire to eat good food, be vengeful, etc} would be thrown out. The first set feels different from the second because those desires feel more like maximising something, which is worrying.
This seems like a big deal to me, because it feels like if I could modify myself, I would probably do so to make myself better at achieving a handful of goals like {having a positive impact, (maybe) obtaining a large amount of power/money, getting really really good at a particular skill} and everything else like {desire to eat good food, be vengeful, etc} would be thrown out. The first set feels different from the second because those desires feel more like maximising something, which is worrying.