[Question] If I knew how to make an omohundru optimizer, would I be able to do anything good with that knowledge?

I’d bet we’re go­ing to figure out how to make an omo­hun­dro op­ti­miser—a fit­ness-max­i­miz­ing AGI—be­fore we figure out how to make AGI that can res­cue the util­ity func­tion, pre­serve a goal, or sig­nifi­cantly op­ti­mise any met­ric other than its own sur­vival, such as pa­per­clip pro­duc­tion, or Good.

(Ar­gu­ing for that is a bit be­yond the scope of the ques­tion, but I know this po­si­tion has a lot of sup­port already. I’ve heard Eliezer say, if not this ex­actly, some­thing very similar. Nick Land es­pe­cially be­lieves that only the omo­hun­dro drives could an­i­mate self-im­prov­ing AGI. I don’t think Nick Land un­der­stands how agency needs to in­ter­cede in pre­dic­tion—that it needs to con­sider all of the com­pet­ing self-fulfilling proph­e­sies and only pro­fess the proph­esy it re­ally wants to live in, in­stead of im­me­di­ately sid­ing with the proph­esy that seems the most hellish, and most eas­iest to stum­ble into. The proph­e­sies he tends to choose do seem like the eas­iest proph­e­sies to stum­ble into, so he pro­vides a use­ful ser­vice as a haz­ard alarm, for we who are try­ing to learn not to stum­ble))

What would you ad­vise we do, when one of us finds our­selves in the po­si­tion of know­ing how to build an omo­hun­dro op­ti­miser? Delete the code and for­get it?

If we had a fit­ness-op­ti­mis­ing pro­gram, is there any­thing good it could be used for?