You have two questions: why accurately approximate human value, and why not have it just ask us about ambiguities.
Because the hard part is getting it to do anything coherent at all, and once we are there, it is little extra work to make it do what we really want.
This would work. The hard part is to get it to do that.
I would also accept most people as BDFL, over the incumbent gods of indifferent chaos. Again the hard part is kicking out the incumbent. Past that point the debate is basically what color to paint the walls, by comparison.
I was modelling it as a superintelligence acting on eg GWB’s behalf, including doing his moral philosophy (ie GWB’s Extrapolated Volition). I see I wasn’t exactly obvious with that assumption.
Let’s put it this way, conditional on the BDFL doing well by their own standards (so, not the usual human fail), I would probably find that world superior to this.
The only wrench to be thrown in this is human corruption by power, but then it’s debatable whether the BDFL is doing well by their own (previous) standards.
I broadly agree with this. GWB probably thinks of eg minimising gay sex as a terminal value, but I would have thought that a superintelligence extrapolating GWBEV would figure out that value was conditional on their being a God, which there isn’t, and discard it.
You have two questions: why accurately approximate human value, and why not have it just ask us about ambiguities.
Because the hard part is getting it to do anything coherent at all, and once we are there, it is little extra work to make it do what we really want.
This would work. The hard part is to get it to do that.
I would also accept most people as BDFL, over the incumbent gods of indifferent chaos. Again the hard part is kicking out the incumbent. Past that point the debate is basically what color to paint the walls, by comparison.
Not sure I would. Azathoth doesn’t fight back if you try to overthrow it and set up Belldandy in Its place. George W. Bush would.
I was modelling it as a superintelligence acting on eg GWB’s behalf, including doing his moral philosophy (ie GWB’s Extrapolated Volition). I see I wasn’t exactly obvious with that assumption.
Let’s put it this way, conditional on the BDFL doing well by their own standards (so, not the usual human fail), I would probably find that world superior to this.
The only wrench to be thrown in this is human corruption by power, but then it’s debatable whether the BDFL is doing well by their own (previous) standards.
I broadly agree with this. GWB probably thinks of eg minimising gay sex as a terminal value, but I would have thought that a superintelligence extrapolating GWBEV would figure out that value was conditional on their being a God, which there isn’t, and discard it.