Toby Ord, you need to click “Sign Out”, then sign in again, even if it shows “You are currently signed in as Toby Ord.”
The reason I’m skeptical of utilitarianism in this context is that my view on morality is that it is subjectively objective. I experience morality as objective in the sense of not being subject to arbitrary alterations and having questions with determinable yet surprising answers. The one who experiences this is me, and a differently constructed agent could experience a different morality. While I still encounter moral questions with surprising answers, I don’t expect to ever encounter an answer so surprising as to convince me that torturing children should be positively terminally valued. I would be very surprised if there were a valid moral argument leading up to that conclusion. I would be less surprised, but still very much so, if there were a valid moral argument leading up to hedonic utilitarianism—and if you present this moral philosophy today, you must have arguments that justify it today. You can’t just say, “Well, there might be an argument” any more than you can just say, “Well, there might be an argument that you should give all your money to Reverend Moon.”
But clear this up for a non-expert: does stating that ethics cannot come from a single utility function imply that no amount of utility functions (i.e. no computer program) will do the job on their own?
That doesn’t follow from the information given. As far as we know at this point, a complex utility function will do the job. There are reasons why you might not want to use a standard-yet-complex utility function in a Friendly AI, but those reasons are beyond the scope of this particular discussion.
You yourself are a computer. You are a protein computer. What you can do, computers can do.
Toby Ord, you need to click “Sign Out”, then sign in again, even if it shows “You are currently signed in as Toby Ord.”
The reason I’m skeptical of utilitarianism in this context is that my view on morality is that it is subjectively objective. I experience morality as objective in the sense of not being subject to arbitrary alterations and having questions with determinable yet surprising answers. The one who experiences this is me, and a differently constructed agent could experience a different morality. While I still encounter moral questions with surprising answers, I don’t expect to ever encounter an answer so surprising as to convince me that torturing children should be positively terminally valued. I would be very surprised if there were a valid moral argument leading up to that conclusion. I would be less surprised, but still very much so, if there were a valid moral argument leading up to hedonic utilitarianism—and if you present this moral philosophy today, you must have arguments that justify it today. You can’t just say, “Well, there might be an argument” any more than you can just say, “Well, there might be an argument that you should give all your money to Reverend Moon.”
But clear this up for a non-expert: does stating that ethics cannot come from a single utility function imply that no amount of utility functions (i.e. no computer program) will do the job on their own?
That doesn’t follow from the information given. As far as we know at this point, a complex utility function will do the job. There are reasons why you might not want to use a standard-yet-complex utility function in a Friendly AI, but those reasons are beyond the scope of this particular discussion.
You yourself are a computer. You are a protein computer. What you can do, computers can do.