Larry D’Anna:
Yes, if the agent is powerful you might be killed even by mindless thrashings.
My point was that in an adversarial situation, you should assume your opponent will always make perfect choices. Then their mistakes are to your advantage. If you’re ready for optimal thrashings, random thrashings will be easier. If you know your opponent will make some kind of mistake, then you can step a trap. But for that you have to understand it very well.
If we are up against an AI that always makes the right choice, then it will almost certainly get what it wants.
If we’re going to say that evolution doesn’t think, then we should also say it doesn’t want. Arguably wanting is anthropomorphizing.
I think we can still use the plan-for-the-strogest-possible-attack strategy though, because we can define it in terms of what we want. (but we still have to look at how the opponent ‘thinks’, not how we do)
Roko: then some significant fraction of possible minds will approximate those objective values.
I would say that effective minds would find those values as useful strategies. (and for this reason evolved minds would approximate them.)
But all possible minds would be all over the place.
Larry D’Anna: Yes, if the agent is powerful you might be killed even by mindless thrashings.
My point was that in an adversarial situation, you should assume your opponent will always make perfect choices. Then their mistakes are to your advantage. If you’re ready for optimal thrashings, random thrashings will be easier. If you know your opponent will make some kind of mistake, then you can step a trap. But for that you have to understand it very well.
If we are up against an AI that always makes the right choice, then it will almost certainly get what it wants.
If we’re going to say that evolution doesn’t think, then we should also say it doesn’t want. Arguably wanting is anthropomorphizing.
I think we can still use the plan-for-the-strogest-possible-attack strategy though, because we can define it in terms of what we want. (but we still have to look at how the opponent ‘thinks’, not how we do)
Roko: then some significant fraction of possible minds will approximate those objective values.
I would say that effective minds would find those values as useful strategies. (and for this reason evolved minds would approximate them.)
But all possible minds would be all over the place.