Very interesting, thank you!
I think “maximising” still makes sense in one-shot problems. 2>1 and 1000>1, but it’s also the case that 1000>2, even without expected utility. The way I see it, EU is a method of comparing choices based on their average utility, but the “average” turns out to be a less useful metric when you only have one chance.
So for cases when an outcome is not a constant amount of paperclips we need more rules than what the object of attention is. So a paperclip maximiser is actually underspecified.
If this is true, it would imply that in a one-shot problem, a utility function is not enough on its own to determine what is the “optimal” choice when you want to “maximise” (get the highest value you can) on that utility function. This would be a pretty big result, I think.
I think that if there is a part that is underspecified, though, it’s not the paperclip maximiser, but the word “optimal”. What does it mean for a choice to be “optimal” relative to other choices, when it might turn out better or worse depending on luck? I haven’t been able to answer that question.
Thank you for your response!
My intuition is that in a one-shot problem, gambling everything on an extremely low probability event is a bad idea, even when the reward from that low probability event is very high, because you are effectively certain to lose. This is the basis for me not paying up in Pascal’s Mugging and in the casino problem in the post.
I’m trying to keep my reasoning simple, so in my examples I always assume that there are no infinities, no unknown unknowns, every outcome of every choice is statistically independent, and all the assigned probabilities are statistically correct (if there is a 1⁄6 chance of an outcome and you get to repeat the problem, you will get that outcome on average 1⁄6 of the time).
Honestly, I have no idea how to solve the problem. My intuition is hopelessly muddled on this, and every idea I’ve been able to come up with seems flawed, including the one you’ve just asked about.
My first thought is 1/googolplex chance of losing 3^^^^3 paperclips, and the rest of the probability giving as many paperclips as the kolmogorov complexity constraint allows. I could do better by increasing the probability of the loss, for example 1/googol would be a better probability. However, I have no idea where to draw the line, at what point it stops being a good idea to increase the probability.