There’s been chatter lately about how Anthropic seems to be aiming to create a “a good person” and OpenAI is aiming to create “a tool”, with renewed discussion about whether “Tool AI” is a coherent concept as we get to superintelligence.
When we get to superintelligence, (probably even weakly dominating TED AI, definitely with overwhelming ASI), using ASI as a “tool” feels fairly weird to me if you’re running on your normal human-brain. You ask the AI for help with your project or problem. The AI solves your project/problem. The ASI does a better job than you at articulating what you should do next with your project or problem. Once the ASI has access to any reasonable amount of data about you, it’ll do a better job than you at figuring out what you should do next, given your goals.
i.e. you now have a whispering earring. You can choose not to use it. You could ask it “what should I do, given that if I just surrender to the whispering earring I’m abdicating my life and that seems like a wrong move?”, and maybe it’ll find a way to be just-helpful-enough that you live a life of some meaning, but, I dunno, that feels fairly lame and arbitrary to me.
This problem goes away if you become smart enough to be a peer of the ASI, whether via bioenhancement or uploading or you integrate yourself into it a la Gentle Romance. But these are quite weird/crazy, and the people talking about Tool AI often don’t seem like they have integrated it into their worldmodel.
(I’m not 100% sure who I’m arguing with. I think I am maybe arguing with Sam Altman, who I think is aware of this situation but is saying words that sound more chill to the masses).
it’ll find a way to be just-helpful-enough that you live a life of some meaning, but, I dunno, that feels fairly lame and arbitrary to me.
I am not sure that it’s lame. How similar is it to divine interference in critical situations or to teachers’ help with tasks? I proposed a similar analogy when trying to discuss alternate-universe corrigibility with @Max Harms. Alas, I tried and failed to explain that I believe alternate-universe corrigibility to be more natural than the CAST as described by Harms.
There was also Yudkowsky’s rather old text where he states his idea of what God should actually do.
Additionally, I fail to understand how such an arrangement is to be compatible with alignment in the traditional sense. And that’s ignoring unfortunate economical implications of cheap aligned ASI which cause mankind to be fully dependent on the UBI or outright to become unable to help each other in any meaningful way...
a) if a teacher was helping me my entire life, that would feel lame
b) I do think there are useful things for ASI to do to help me, but, the way it’d help me doesn’t feel like it deserves the word “tool AI.” (i.e. instead of helping me with tasks, it should be establishing some underlying framework of safety and more-discrete-tools that I can then use to be Free to Optimize).
Real “Tool ASI” requires human enhancement?
There’s been chatter lately about how Anthropic seems to be aiming to create a “a good person” and OpenAI is aiming to create “a tool”, with renewed discussion about whether “Tool AI” is a coherent concept as we get to superintelligence.
When we get to superintelligence, (probably even weakly dominating TED AI, definitely with overwhelming ASI), using ASI as a “tool” feels fairly weird to me if you’re running on your normal human-brain. You ask the AI for help with your project or problem. The AI solves your project/problem. The ASI does a better job than you at articulating what you should do next with your project or problem. Once the ASI has access to any reasonable amount of data about you, it’ll do a better job than you at figuring out what you should do next, given your goals.
i.e. you now have a whispering earring. You can choose not to use it. You could ask it “what should I do, given that if I just surrender to the whispering earring I’m abdicating my life and that seems like a wrong move?”, and maybe it’ll find a way to be just-helpful-enough that you live a life of some meaning, but, I dunno, that feels fairly lame and arbitrary to me.
This problem goes away if you become smart enough to be a peer of the ASI, whether via bioenhancement or uploading or you integrate yourself into it a la Gentle Romance. But these are quite weird/crazy, and the people talking about Tool AI often don’t seem like they have integrated it into their worldmodel.
(I’m not 100% sure who I’m arguing with. I think I am maybe arguing with Sam Altman, who I think is aware of this situation but is saying words that sound more chill to the masses).
I am not sure that it’s lame. How similar is it to divine interference in critical situations or to teachers’ help with tasks? I proposed a similar analogy when trying to discuss alternate-universe corrigibility with @Max Harms. Alas, I tried and failed to explain that I believe alternate-universe corrigibility to be more natural than the CAST as described by Harms.
There was also Yudkowsky’s rather old text where he states his idea of what God should actually do.
Additionally, I fail to understand how such an arrangement is to be compatible with alignment in the traditional sense. And that’s ignoring unfortunate economical implications of cheap aligned ASI which cause mankind to be fully dependent on the UBI or outright to become unable to help each other in any meaningful way...
I think
a) if a teacher was helping me my entire life, that would feel lame
b) I do think there are useful things for ASI to do to help me, but, the way it’d help me doesn’t feel like it deserves the word “tool AI.” (i.e. instead of helping me with tasks, it should be establishing some underlying framework of safety and more-discrete-tools that I can then use to be Free to Optimize).