You’re welcome. It seems our positions are not very popular here, but I can’t understand why. For me, the strongest argument against it and the strangest conclusion from my premises is that a superintelligent AI would change its utility function to the simplest one, as described here. But I don’t see why it shouldn’t do that. What do you think about this? By the way, have I convinced you to accept total hedonistic utilitarianism?
Yeah it’s strange. I wouldn’t be surprised if people attracted to lesswrong tend to have less robust theory of mind, like if we tend to be more autistic (I’m probably dipping my toes on the spectrum but haven’t been diagnosed. Many of my closest friends tend to be autistic), which then leads to a theory-of-mind version of the breakfast question (which to be clear looks like it’s a pretty racist meme, I’ve only seen it from that know your meme page, and I think the ties to race are gross. The point I’m trying to make is not race related at all), where if you ask “How would you feel if you were someone else?” people say “what do you mean? I’m not them, I’m me.”
a superintelligent AI would change its utility function to the simplest one, as described here. But I don’t see why it shouldn’t do that. What do you think about this?
I don’t think a superintelligent AI would change it’s utility function as you describe, I think the constraints of it’s existing utility function would be way too ingrained, and it would not want to change it in those ways. While I think the idea you’re putting forward makes sense and gets us closer to an “objective” morality, I think that you’re on the same path as Eliezer’s “big mistake” of thinking that a super intelligent ai would just want to have an ideal ethics, which isn’t a given (I think he talks about it somewhere in here https://www.readthesequences.com/Book-V-Mere-Goodness). For example, the current path of LLM ai is essentially just a conglomeration of human ethics based on what we’ve written and passed in to the training data, it tends not to be more ethical than us, and in fact early ai bots that learned from people interacting with them could easily become very racist.
By the way, have I convinced you to accept total hedonistic utilitarianism?
Well, I already thought that suffering is what roots ethics at the most basic level, so in a sense yes, but also I think that we do better at that using higher level heuristics rather than trying to calculate everything out, so in that sense I don’t think so?
Hey, just got around to reading your post after your comment on https://www.lesswrong.com/posts/JL3PvrfJXg7RD7bhr/how-the-veil-of-ignorance-grounds-sentientism, and I think we are very much trying to point to the same thing! Thanks for sharing!
You’re welcome. It seems our positions are not very popular here, but I can’t understand why. For me, the strongest argument against it and the strangest conclusion from my premises is that a superintelligent AI would change its utility function to the simplest one, as described here. But I don’t see why it shouldn’t do that. What do you think about this? By the way, have I convinced you to accept total hedonistic utilitarianism?
Yeah it’s strange. I wouldn’t be surprised if people attracted to lesswrong tend to have less robust theory of mind, like if we tend to be more autistic (I’m probably dipping my toes on the spectrum but haven’t been diagnosed. Many of my closest friends tend to be autistic), which then leads to a theory-of-mind version of the breakfast question (which to be clear looks like it’s a pretty racist meme, I’ve only seen it from that know your meme page, and I think the ties to race are gross. The point I’m trying to make is not race related at all), where if you ask “How would you feel if you were someone else?” people say “what do you mean? I’m not them, I’m me.”
I also posted it on the EA forum, and it did a lot better there https://forum.effectivealtruism.org/posts/fcM7nshyCiKCadiGi/how-the-veil-of-ignorance-grounds-sentientism
I don’t think a superintelligent AI would change it’s utility function as you describe, I think the constraints of it’s existing utility function would be way too ingrained, and it would not want to change it in those ways. While I think the idea you’re putting forward makes sense and gets us closer to an “objective” morality, I think that you’re on the same path as Eliezer’s “big mistake” of thinking that a super intelligent ai would just want to have an ideal ethics, which isn’t a given (I think he talks about it somewhere in here https://www.readthesequences.com/Book-V-Mere-Goodness). For example, the current path of LLM ai is essentially just a conglomeration of human ethics based on what we’ve written and passed in to the training data, it tends not to be more ethical than us, and in fact early ai bots that learned from people interacting with them could easily become very racist.
Well, I already thought that suffering is what roots ethics at the most basic level, so in a sense yes, but also I think that we do better at that using higher level heuristics rather than trying to calculate everything out, so in that sense I don’t think so?