I don’t think this is particularly compelling. While technically true, the difference between those probabilities is tiny. Any AGI is highly likely to understand democratic laws.
Regarding, “Any AGI is highly likely to understand democratic laws.”
There is likely much additional work to be done to imbue a comprehensive understanding of law in AGI systems—in particular many of our legal standards (versus rules, which are easier and are already legible to AI) and many nuanced processes that are only in human legal expert minds right now. Making those things structured enough for a computational encoding is not easy.
If we solve that, though, there is still the work to be done on (1.) verifying AGI legal understandings (and AI systems along the way to AGI), and (2.) ensuring that law is still made by humans. Setting new legal precedent (which, broadly defined, includes proposing and enacting legislation, promulgating regulatory agency rules, publishing judicial opinions, enforcing law, and more) should be exclusively reserved for the democratic governmental systems expressing uniquely human values. The positive implications of that normative stance are that the resulting law then encapsulates human views and can be used for AGI understanding human values. We need to do significant technical and governance work to ensure law-making is human for that reason.
when discussed in the larger context, should likely be expressed more like,
P(misalignment x-risk | AGI that understands democratic law, and the law is entirely human-driven, and humans have validation of the ways in which AGI understands the law) < P(misalignment x-risk | AGI)
I don’t think this is particularly compelling. While technically true, the difference between those probabilities is tiny. Any AGI is highly likely to understand democratic laws.
There is likely much additional work to be done to imbue a comprehensive understanding of law in AGI systems—in particular many of our legal standards (versus rules, which are easier and are already legible to AI) and many nuanced processes that are only in human legal expert minds right now. Making those things structured enough for a computational encoding is not easy.
If we solve that, though, there is still the work to be done on (1.) verifying AGI legal understandings (and AI systems along the way to AGI), and (2.) ensuring that law is still made by humans. Setting new legal precedent (which, broadly defined, includes proposing and enacting legislation, promulgating regulatory agency rules, publishing judicial opinions, enforcing law, and more) should be exclusively reserved for the democratic governmental systems expressing uniquely human values. The positive implications of that normative stance are that the resulting law then encapsulates human views and can be used for AGI understanding human values. We need to do significant technical and governance work to ensure law-making is human for that reason.
Therefore,
when discussed in the larger context, should likely be expressed more like,