In response to what I understand to be your question (“So what do you do to make the alignment guarantee good outcomes? People are stupid..”), I think one commonly accepted answer here is:
Yes, that is a real problem. Something like CEV offers a solution (with a spherical cow, in a vacuum).
In response to what I understand to be your question (“So what do you do to make the alignment guarantee good outcomes? People are stupid..”), I think one commonly accepted answer here is:
Yes, that is a real problem. Something like CEV offers a solution (with a spherical cow, in a vacuum).
There is also a useful differentiation to be made between Inner Alignment and Outer Alignment.