You can’t require people to “be safe” or “be kind” or “avoid suffering” – or, at least, you can’t do that using any of the legal and policy tools common in English-speaking countries.
This is not true, the entire area of Tort law (especially negligence law) is highly developed in English speaking countries, and is a series of elaborations on “take reasonable care to avoid harm to others.”
I think a significant amount of the probability mass within P(no AI takeover) is in various AI fizzle worlds. In those worlds, anyone outside AI safety who is working on making the world better, is working to increase the flourishing associated with those worlds.