Your framing of the illusion of absolute rules as “polite social fictions” is quite brilliant, because I think that’s what Scott Alexander probably wanted to convey. It comes to mind that such social fictions may be required for people to trust in institutions, and strong institutions are generally credited as a core factor for social progress. Take for example the police—it is an extremely useful social fiction that “the police is your friend and helper”, as they say in German, even though they often aren’t, particularly not to marginalized social groups. Upholding this fiction ensures that most people respect the police, report crimes when they occur, and physical attacks by non-criminals against the police are comparatively rare in most countries. At the same time, I think it is incredibly dangerous to prohibit public discussion of police misconduct. Yes, it may destroy the social fiction of the well-meaning police—but shouldn’t people be made aware of instances of police misconduct, so that they can properly adjust their priors? Rationally speaking, doesn’t the police deserve to be treated with a degree of suspicion proportional to their probability of misconduct?
Your point that most people aren’t consequentialists is probably right. But treating them as non-consequentialists, prohibiting discussions and purposefully upholding social fictions inevitably puts you on a slippery slope, where you’re incentivized to keep discussions under wraps because it might “upset the people”—a slope any rationally-minded policymaker should be aware of.
I completely agree that robotics (and any automation technology) must be fundamentally empowering. Right now, robot intelligence is spread on a continuum that ranges from inclusive, assistive, collaborative paradigms to full autonomy. Even within human-robot collaboration, there are technological choices that can be made that make a collaborative application human-empowering or disempowering. As a society, we are making these decisions now, as robot intelligence is rapidly improving and is increasingly widely deployed. Right now, there are still many fundamentally different technological directions competing (e.g. hybrid architectures like the decendants of Code-as-Policies (https://arxiv.org/abs/2209.07753), world-model-based architectures similar to Scientist AI, but also large monolithic GPT-style architectures). We think that there is a lot of leverage right now to influence the direction of technical progress, if we can understand and demonstrate the concrete risks posed by these technical choices, and improve them in a direction that allows people to have true control over what robots are doing and how they are deployed.