I also recognize this feeling of “You have not done enough” or worse “This goal was meaningless in hindsight”. It’s probably very instrumental, pushing us and our genes to ever greater heights.
So should we lean in to it? Accepting happiness is forever lost behind some horizon? You will just walk around with this internal nagging feeling.
Or should we fix this bug as you say, but risk stagnation? One way may be to become a full-time meditating monk. Then you may have a chance to turn your wetware into a personal nirvana untill you pop out of existence. But that feels meaningless as well.
I’m trying to find a blend; take the edge off the suffering while moving forward.
I agree, it’s important to create, or at least detect, well-aligned agents. You suggest we need an honesty API.