It is generally understood now that ethics is subjective, in the following technical sense: ‘what final goals you have’ is a ~free parameter in powerful-mind-space, such that if you make a powerful mind without specifically having a mechanism for getting it to have only the goals you want, it’ll probably end up with goals you don’t want. What if ethics isn’t the only such free parameter? Indeed, philosophers tell us that in the bayesian framework your priors are subjective in this sense, and also that your decision theory is subjective in this sense maybe. Perhaps, therefore, what we consider “doing good/wise philosophy” is going to involve at least a few subjective elements, where what we want is for our AGIs to do philosophy (with respect to those elements) in the same way that we would want and not in various other ways, and that won’t happen by default, we need to have some mechanism to make it happen.
Here’s another bullet point to add to the list:
It is generally understood now that ethics is subjective, in the following technical sense: ‘what final goals you have’ is a ~free parameter in powerful-mind-space, such that if you make a powerful mind without specifically having a mechanism for getting it to have only the goals you want, it’ll probably end up with goals you don’t want. What if ethics isn’t the only such free parameter? Indeed, philosophers tell us that in the bayesian framework your priors are subjective in this sense, and also that your decision theory is subjective in this sense maybe. Perhaps, therefore, what we consider “doing good/wise philosophy” is going to involve at least a few subjective elements, where what we want is for our AGIs to do philosophy (with respect to those elements) in the same way that we would want and not in various other ways, and that won’t happen by default, we need to have some mechanism to make it happen.