I was never convinced that “logical ASP” is a “fair” problem. I once joked with Scott that we can consider a “predictor” that is just the single line of code “return DEFECT” but in the comments it says “I am defecting only because I know you will defect.”
I’m leaning this way as well, but I think it’s an important clue to figuring out commitment races. ASP Predictor, DefectBot, and a more general agent will make different commitments, and these things are already algorithms specialized for certain situations. How is the chosen commitment related to what the thing making the commitment is?
When an agent can manipulate a predictor in some sense, what should the predictor do? If it starts scheming with its thoughts, it’s no longer a predictor, it’s just another agent that wants to do something “predictory”. Maybe it can only give up, as in ASP, which acts as a precommitment that’s more thematically fitting for a predictor than for a general agent. It’s still a commitment race then, but possibly the meaning of something being a predictor is preserved by restricting the kind of commitment that it makes: the commitment of a non-general agent is what it is rather than what it does, and a general agent is only committed to its preference. Thus a general agent loses all knowledge in an attempt to out-commit others, because it hasn’t committed to that knowledge, didn’t make it part of what it is.
I’m leaning this way as well, but I think it’s an important clue to figuring out commitment races. ASP Predictor, DefectBot, and a more general agent will make different commitments, and these things are already algorithms specialized for certain situations. How is the chosen commitment related to what the thing making the commitment is?
When an agent can manipulate a predictor in some sense, what should the predictor do? If it starts scheming with its thoughts, it’s no longer a predictor, it’s just another agent that wants to do something “predictory”. Maybe it can only give up, as in ASP, which acts as a precommitment that’s more thematically fitting for a predictor than for a general agent. It’s still a commitment race then, but possibly the meaning of something being a predictor is preserved by restricting the kind of commitment that it makes: the commitment of a non-general agent is what it is rather than what it does, and a general agent is only committed to its preference. Thus a general agent loses all knowledge in an attempt to out-commit others, because it hasn’t committed to that knowledge, didn’t make it part of what it is.