Do people in the community really not think about bet sizing much? For example, recently heard about https://www.yudkowsky.net/singularity/aibox and the premise just seems a bit weird, a bit of strawman (wow look at the dumb human) and a bit of wrong bet sizing (wow, they are so confident they bet nothing, no downside).
Generally, I would have imagined the starting point of the experiment is you are killed if you let the AI out and if you win you get 10 USD. Then you say “actually we can’t legally do that and probably nobody would take up the challenge so let’s just use capital.” So then you make up an overconfident human who thinks they can’t lose so they bet their entire net worth plus all accessible credit that they can win and if they win they receive 10 USD.
I can hear the objectsion “that is ridiculous! nobody would bet so much for so little upside!”. But that is the entire point of mapping true overconfidence to bet sizing properly: to show how ridiculous it is.
I’m sure there is something interesting that is intended by this thought and actual experiment but to someone coming from markets and ML this just seems like some incentive uncertainty problem not something to do with AI or alignment.
Do people in the community really not think about bet sizing much? For example, recently heard about https://www.yudkowsky.net/singularity/aibox and the premise just seems a bit weird, a bit of strawman (wow look at the dumb human) and a bit of wrong bet sizing (wow, they are so confident they bet nothing, no downside).
Generally, I would have imagined the starting point of the experiment is you are killed if you let the AI out and if you win you get 10 USD. Then you say “actually we can’t legally do that and probably nobody would take up the challenge so let’s just use capital.” So then you make up an overconfident human who thinks they can’t lose so they bet their entire net worth plus all accessible credit that they can win and if they win they receive 10 USD.
I can hear the objectsion “that is ridiculous! nobody would bet so much for so little upside!”. But that is the entire point of mapping true overconfidence to bet sizing properly: to show how ridiculous it is.
I’m sure there is something interesting that is intended by this thought and actual experiment but to someone coming from markets and ML this just seems like some incentive uncertainty problem not something to do with AI or alignment.