I suspect that the moral intuitions that you mention are unpopular not just because of people’s ignorance, but because these ideas reflect only a facet of the ground truth (which, in my opinion, should be more derived from first principles, e.g. claiming that the world itself is a big training environment for increasingly-large-scale coordination).
Historically, taxes that the government gathered were used not just to fund meaningless bureaucracies, but for tasks related to the public good like protection against criminals[1] and rival states, construction of buildings (e.g. Roman Empire’s road network, temples or castles), sustaining the culture, and, in more modern settings, public education.
I expect that moral intuitions of mankind do not imply that being ignorant of ways in which one could do better is enough to be called evil. A Russian writer Andrei Platonov explicitly questioned this idea in a short story where the nomads’ lack of technology-related knowledge prevented them from peacefully coexisting with Russians.
Similar arguments in politics imply not that all governments are evil, but that one should carefully think and select the government which is least likely to experience a fiasco[2] or to lock in a suboptimal value system or power distribution sufficiently worse than the fiasco’s results (e.g. if the USA succeeded in overthrowing Maduro and sufficiently improved the quality of lives of Venezuelans, then Venezuelans would likely have been mistaken in supporting Maduro).
As for claims like “maybe we shouldn’t design AGI or ASI to absolutely refuse to seek power”, I think that they confabulate two different issues:
Were the ASI’s values to be suboptimal for human welfare, mankind would either have the option to change the ASI’s values or live under its rule. Therefore, mankind should ensure that the ASI is either corrigible (i.e. fine with being shut down or having its values changed) or aligned to values sufficiently close to optimal ones.
The ASI aligned to the humans would have to seek leverage over dictatorships trying to lock in suboptimal values. For example, the Slowdown Ending of the AI-2027 forecast has Safer-N destroy the CCP. But one could make similar cases against other collectives like Buck’s Christian homeschoolers[3] locking in false beliefs or suboptimal values.
An incomplete list of fiascos: letting a rival state take over without facing as dire consequences as possible, having a wildly mismanaged economy; in more modern settings states could also fail to develop tech (e.g. useful for warfare) and/or educate the workers; nowadays the entire mankind would collectively experience a failure mode if anyone creates a misaligned ASI without an aligned counterpart.
However, we had Tim Hua claim that he would allow such homeschoolers to exist. What I don’t understand is whether he would let such homeschoolers propagate actual falsehoods and not just a different value set.
I suspect that the moral intuitions that you mention are unpopular not just because of people’s ignorance, but because these ideas reflect only a facet of the ground truth (which, in my opinion, should be more derived from first principles, e.g. claiming that the world itself is a big training environment for increasingly-large-scale coordination).
Historically, taxes that the government gathered were used not just to fund meaningless bureaucracies, but for tasks related to the public good like protection against criminals[1] and rival states, construction of buildings (e.g. Roman Empire’s road network, temples or castles), sustaining the culture, and, in more modern settings, public education.
I expect that moral intuitions of mankind do not imply that being ignorant of ways in which one could do better is enough to be called evil. A Russian writer Andrei Platonov explicitly questioned this idea in a short story where the nomads’ lack of technology-related knowledge prevented them from peacefully coexisting with Russians.
Similar arguments in politics imply not that all governments are evil, but that one should carefully think and select the government which is least likely to experience a fiasco[2] or to lock in a suboptimal value system or power distribution sufficiently worse than the fiasco’s results (e.g. if the USA succeeded in overthrowing Maduro and sufficiently improved the quality of lives of Venezuelans, then Venezuelans would likely have been mistaken in supporting Maduro).
As for claims like “maybe we shouldn’t design AGI or ASI to absolutely refuse to seek power”, I think that they confabulate two different issues:
Were the ASI’s values to be suboptimal for human welfare, mankind would either have the option to change the ASI’s values or live under its rule. Therefore, mankind should ensure that the ASI is either corrigible (i.e. fine with being shut down or having its values changed) or aligned to values sufficiently close to optimal ones.
The ASI aligned to the humans would have to seek leverage over dictatorships trying to lock in suboptimal values. For example, the Slowdown Ending of the AI-2027 forecast has Safer-N destroy the CCP. But one could make similar cases against other collectives like Buck’s Christian homeschoolers[3] locking in false beliefs or suboptimal values.
UPD: I also have prepared an interesting dialogue with Claude Opus 4.5.
The stationary bandit theory lets the government outright evolve from said criminals who are smart enough to think of their long-term interests.
An incomplete list of fiascos: letting a rival state take over without facing as dire consequences as possible, having a wildly mismanaged economy; in more modern settings states could also fail to develop tech (e.g. useful for warfare) and/or educate the workers; nowadays the entire mankind would collectively experience a failure mode if anyone creates a misaligned ASI without an aligned counterpart.
However, we had Tim Hua claim that he would allow such homeschoolers to exist. What I don’t understand is whether he would let such homeschoolers propagate actual falsehoods and not just a different value set.