“turn the universe into paperclips” is an in-reference and might not be suitable in a short article meant to introduce “rescuing the utility function” in isolation. At the very least, we should turn that into a link to an article on paper clip maximizers so that unfamiliar readers can know what the heck that sentence is supposed to mean. Alternatively, we could use a different example that doesn’t rely on that background knowledge.
Nebu Pookins
Karma: 0
It may be worth reading https://wiki.lesswrong.com/wiki/Shut_up_and_multiply as well, as the concepts in these two articles overlap somewhat, and it may be a useful exercise to think about whether the two articles contradict each other or not, and why or why not.