“never letting a catastrophe happen” would incentivize the agent to spend a lot of resources on foreseeing catastrophes and building capacity to ward them off. This would distract from the agent’s main task. So we have to give the agent some slack. Is this what you’re getting at? The oracle needs to decide whether or not the agent can be held accountable for a catastrophe, but the article doesn’t say anything how it would do this?
The oracle needs to decide whether or not the agent can be held accountable for a catastrophe, but the article doesn’t say anything how it would do this?
Yes, basically. I’m not saying the article should specify how the oracle should do this, I’m saying that it should flag this as a necessary property of the oracle (or argue why it is not a necessary property).
“never letting a catastrophe happen” would incentivize the agent to spend a lot of resources on foreseeing catastrophes and building capacity to ward them off. This would distract from the agent’s main task. So we have to give the agent some slack. Is this what you’re getting at? The oracle needs to decide whether or not the agent can be held accountable for a catastrophe, but the article doesn’t say anything how it would do this?
Yes, basically. I’m not saying the article should specify how the oracle should do this, I’m saying that it should flag this as a necessary property of the oracle (or argue why it is not a necessary property).
I agree.