“The Alliance” by Gerald Lund has this premise. (Well, it’s not an AI but human beings enforcing the no harm policy, but it makes points about the loss of agency and identity that happens when you’re no longer able to choose).
Zachary Clement
Karma: 5
Work that reduces the odds of a bad outcome by X% is worth doing if the cost to do the work is less than X% times [cost of bad outcome]. That work is worth doing even if you can’t take the risk of a bad outcome to zero.
In practice it’s really hard to figure out what that X% is but I think a lot of things that might look like “rainmaker“ work to someone outside the field are actually positive in expected value.
The key is not to just discount people who can’t fully control outcomes they’re trying to prevent, but to do an honest cost/benefit analysis to determine what kind of outcomes can be prevented with a given level of prevention and choose the right level.
You can probably reduce risk of a cybersecurity event by having employees do trainings on social engineering attacks for a relatively low cost, but you would have to spend a ton to make your system robust to attacks by state level actors. Cost/benefit might make sense for the first level of investment but not for the second.