I mean, people could construct an AI that will acausally trade with you in a human-understandable way. I don’t think this is completely wild, but I’d agree that as the probabilities get smaller, trade becomes less and less profitable/likely—you don’t just have to find it, it has to find you. This is kind of like a quadratic penalty term.
Overall I think the best anti-acausal-trade-worry idea is “You know that decision-making procedure that you’re worried other agents might use to take your lunch money? What would happen if you used it too, to get things to go well for yourself?”
I mean, people could construct an AI that will acausally trade with you in a human-understandable way. I don’t think this is completely wild, but I’d agree that as the probabilities get smaller, trade becomes less and less profitable/likely—you don’t just have to find it, it has to find you. This is kind of like a quadratic penalty term.
Overall I think the best anti-acausal-trade-worry idea is “You know that decision-making procedure that you’re worried other agents might use to take your lunch money? What would happen if you used it too, to get things to go well for yourself?”
There would be almost infinite types of non Superintelligent AIs too right?