(This thread is getting a bit long, and we might not be convincing each other very much, so hope it’s ok if I only reply with points I consider interesting—not just push-pull.)
With the concert pianist thing I think there’s a bit of type error going on. The important skill for a musician isn’t having fast fingers, it’s having something to say. Same as: “I’d like to be able to write like a professional writer”—does that mean anything? You either have things you want to write in the way that you want to write, or there’s no point being a writer at all, much less asking an AI to make you one. With music or painting it’s the same. There’s some amount of technique required, but you need to have something to say, otherwise there’s no point doing it.
So with that in mind, maybe music isn’t the best example in your case. Let’s take an area where you have something to say, like philosophy. Would you be willing to outsource that?
Let’s take an area where you have something to say, like philosophy. Would you be willing to outsource that?
Outsourcing philosophy is the main thing I’ve been trying to do, or trying to figure out how to safely do, for decades at this point. I’ve written about it in various places, including this post and my pinned tweet on X. Quoting from the latter:
Among my first reactions upon hearing “artificial superintelligence” were “I can finally get answers to my favorite philosophical problems” followed by “How do I make sure the ASI actually answers them correctly?”
Aside from wanting to outsource philosophy to ASI, I’d also love to have more humans who could answer these questions for me. I think about this a fair bit and wrote some things down but don’t have any magic bullets.
(I currently think the best bet to eventually getting what I want is to encourage an AI pause along with genetic enhancements for human intelligence, have the enhanced humans solve metaphilosophy and other aspects of AI safety, then outsource the rest of philosophy to ASI, or have the enhanced humans decide what to do at that point.)
BTW I thought this would be a good test for how competent current AIs are at understanding someone’s perspective so I asked a bunch of them how Wei Dai would answer your question, and all of them got it wrong on the first try, except Claude Sonnet 4.5 which got it right on the first try but wrong on the second try. It seems like having my public content in their training data isn’t enough, and finding relevant info from the web and understanding nuance are still challenging for them. (GPT-5 essentially said I’d answer no because I wouldn’t trust current AIs enough, which is really missing the point despite having this whole thread as context.)
Yeah, I wouldn’t have predicted this response either. Maybe it’s a case of something we talked about long ago—that if a person’s “true values” are partly defined by how the person themselves would choose to extrapolate them, then different people can end up on very diverging trajectories. Like, it seems I’m slightly more attached to some aspects of human experience that you don’t care much about, and that affects the endpoint a lot.
(This thread is getting a bit long, and we might not be convincing each other very much, so hope it’s ok if I only reply with points I consider interesting—not just push-pull.)
With the concert pianist thing I think there’s a bit of type error going on. The important skill for a musician isn’t having fast fingers, it’s having something to say. Same as: “I’d like to be able to write like a professional writer”—does that mean anything? You either have things you want to write in the way that you want to write, or there’s no point being a writer at all, much less asking an AI to make you one. With music or painting it’s the same. There’s some amount of technique required, but you need to have something to say, otherwise there’s no point doing it.
So with that in mind, maybe music isn’t the best example in your case. Let’s take an area where you have something to say, like philosophy. Would you be willing to outsource that?
Outsourcing philosophy is the main thing I’ve been trying to do, or trying to figure out how to safely do, for decades at this point. I’ve written about it in various places, including this post and my pinned tweet on X. Quoting from the latter:
Aside from wanting to outsource philosophy to ASI, I’d also love to have more humans who could answer these questions for me. I think about this a fair bit and wrote some things down but don’t have any magic bullets.
(I currently think the best bet to eventually getting what I want is to encourage an AI pause along with genetic enhancements for human intelligence, have the enhanced humans solve metaphilosophy and other aspects of AI safety, then outsource the rest of philosophy to ASI, or have the enhanced humans decide what to do at that point.)
BTW I thought this would be a good test for how competent current AIs are at understanding someone’s perspective so I asked a bunch of them how Wei Dai would answer your question, and all of them got it wrong on the first try, except Claude Sonnet 4.5 which got it right on the first try but wrong on the second try. It seems like having my public content in their training data isn’t enough, and finding relevant info from the web and understanding nuance are still challenging for them. (GPT-5 essentially said I’d answer no because I wouldn’t trust current AIs enough, which is really missing the point despite having this whole thread as context.)
Yeah, I wouldn’t have predicted this response either. Maybe it’s a case of something we talked about long ago—that if a person’s “true values” are partly defined by how the person themselves would choose to extrapolate them, then different people can end up on very diverging trajectories. Like, it seems I’m slightly more attached to some aspects of human experience that you don’t care much about, and that affects the endpoint a lot.